mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
1cbe1538ba9edacc528b19765ab321ba54b65d2d
151 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amaury Levé
|
8bbf2bfd61 |
dotnet-test: reduce assertion-quality over-structuring on small suites (#865)
Scale report depth to suite size so a tiny test file gets a focused, direct answer instead of a padded multi-section dashboard. Preserves full-template behavior for substantial suites and keeps all rubric-relevant substance (assertion-free/trivial identification, quality verdict, concrete recommendations). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
77154137e8 |
dotnet-test: make code-testing agent tools Claude Code-compatible + add cross-host portability check (#856)
* dotnet-test: make code-testing agent tools declarations Claude Code-compatible PR #847 added `tools: ["agent", "skill", "read", "search", "edit", "execute"]` to the code-testing-* agents to enable VS Code / Copilot CLI subagent fan-out. Those lowercase aliases map to real tools in VS Code and the Copilot CLI, but Claude Code matches `tools:` against its own vocabulary (Task, Skill, Read, Glob, Grep, Edit, Write, Bash). None of the aliases matched, so when these agents are loaded into Claude Code via --plugin-dir and selected with `claude --agent`, the agent was granted ZERO tools. A tool-less model asked to generate tests emits a textual <tool_call> block and exits after one turn, producing no file changes. Append the Claude Code tool names to each agent's `tools:` list so the same declaration works across all three runtimes (each honors the names it knows and ignores the foreign ones): - Orchestrators (generator, implementer): add Task, Skill, Read, Glob, Grep, Edit, Write, Bash (Task is the Claude Code equivalent of the `agent` fan-out tool). - Workers (researcher, planner, builder, tester, fixer, linter): add Skill, Read, Glob, Grep, Edit, Write, Bash. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * skill-validator: complete built-in tools + add cross-host tool portability check Two related follow-ups to the agent tools fix: 1. Address the skill-check review feedback. The validator's BuiltInTools set was missing three legitimate host tool spellings that are not case-insensitive matches of existing entries, so they were flagged as non-built-in: - "write" — Claude Code file-creation tool (Copilot CLI / VS Code: "create") - "agent" — Copilot CLI / VS Code subagent fan-out tool (Claude Code: "task") - "execute" — Copilot CLI / VS Code run-command tool (Claude Code: "bash") "agent" and "execute" were already flagged before this branch (introduced by the fan-out PR); adding them to BuiltInTools clears the pre-existing warnings. 2. Add a cross-host tool portability check (CheckAgentToolPortability) so an agent that declares a capability for only one host is flagged. Tool names are matched case-sensitively (hosts resolve tools by exact spelling), so an agent that lists e.g. only "edit" (Copilot / VS Code) without "Edit"/"Write" (Claude Code) is reported as working on one host and silently tool-less on the other. Findings are advisory (do not fail CI) and allowlistable via "agent-tool-portability:AGENT:capability". Wired into the agents loop in CheckCommand and covered by unit tests. Also make the one existing single-host agent (optimizing-dotnet-performance) portable by adding its Claude Code tool spellings, so the new check reports a clean tree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
72af853455 |
Add slnf and slnx solution formats to run-tests skill (#857)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ce75c35569 |
dotnet-test: raise timeouts for chronic-timeout eval scenarios (#850)
Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout in the evaluation dashboard (skilled + plugin arms), producing empty output and failing all assertions. Increase per-scenario timeouts for the ones that timed out most often across recent runs: - writing-mstest-tests: 'Write unit tests for a service class', 'Write data-driven tests for a calculator', 'Use string assertions for format validation' 180->360; 'Use comparison assertions for boundary testing' 120->240 - coverage-analysis: 'Coverage plateau diagnosis', 'Project-wide coverage analysis with existing Cobertura data' 300->480 - test-gap-analysis: 'Decline request to write new tests from scratch' 120->300; 'Acknowledge well-tested code with few surviving mutations' 300->420 - run-tests: 'Filter xUnit v3 tests by trait on MTP', 'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240 Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the highest-priority failure mode; code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
a677ec43d2 |
Consolidate dotnet-test-frameworks into test-analysis-extensions (#851)
* Consolidate dotnet-test-frameworks into test-analysis-extensions Remove the orphaned dotnet-test-frameworks reference skill, whose content was a duplicate subset of test-analysis-extensions/extensions/dotnet.md. Nothing actually loaded it. Redirect the remaining references and drop its eval tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make exp-test-maintainability Step 1 self-contained and fix activation Address PR review: exp-test-maintainability (dotnet-experimental plugin) must not point at test-analysis-extensions (dotnet-test plugin) — inline the .NET framework markers instead. Also strengthen the description/When-to-Use triggers so the 'each new case needs a whole new method / suggest a better structure' scenario reliably activates the skill instead of routing to a test-writing skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert exp-test-maintainability description broadening; keep self-contained markers The description broadening regressed activation (3/4 -> 0/4 across a variance-dominated single eval run, CV up to 924%). Description is the activation lever, so restore the original wording and keep only the review-comment fix (inlined .NET framework markers in Step 1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 1 eval: use project fixture so the skill activates Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a better structure', which the base model answers directly without loading the skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture pattern used by the activating scenarios (3 & 4): move the repetitive InputValidatorTests into a Validation.Tests fixture and ask the model to analyze the project. Mirror the change in eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 2 eval: use project fixture so the skill activates Scenario 2 also pasted a self-contained snippet inline (well-maintained Auth.Tests) and asked to 'review these tests', which the base model handles without loading the skill. Convert to the project-fixture pattern used by the activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in eval.vally.yaml. All four scenarios now reference a project on disk. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add reject_tools to exp-test-maintainability evals to create headroom All four scenarios scored NOT ACTIVATED / no-headroom (#8 in InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the skill only adds token/tool overhead with no offsetting quality gain. Match the convention of the sibling analysis skill test-anti-patterns: reject bash/edit/create on every scenario. This skill is analysis-only ('do not modify any files'), so the constraint is a no-op for correct behavior but levels the playing field between baseline and skilled runs — scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced overhead. Applied to both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: make well-maintained fixture self-contained and deterministic - Import System.Security.Authentication for AuthenticationException - Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a deterministic one anchored on a fixed clock (FixedNow), and drive the injected FakeClock from that fixed time. This removes the flakiness and makes the FakeClock injection meaningful — a better exemplar for the 'well-maintained' scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Petr Pokorny
|
325890d119 |
Bump dotnet-test to 0.2.0 (#852)
* Bump dotnet-test plugin to 0.2.0 Signals the VS Code subagent fan-out fix (#847) to existing installs. Minor bump; no breaking changes. * Sync .codex-plugin manifest to 0.2.0 Keeps plugins/dotnet-test/.codex-plugin/plugin.json in sync with the main plugin.json, matching the convention other plugins follow (e.g. dotnet-ai). Addresses PR review feedback. |
||
|
Petr Pokorny
|
06dd46e5f1 |
Enable VS Code subagent fan-out for the dotnet-test code-testing agents (#847)
* Add VS Code subagent metadata to dotnet-test code-testing agents
Enable the code-testing-* Research-Plan-Implement pipeline to fan out as
subagents in VS Code while keeping the GitHub Copilot CLI working.
Frontmatter (VS Code "coordinator/worker" pattern; portable tool aliases that
map in both VS Code and the CLI):
- code-testing-generator: tools: [agent, read, search, edit, execute] + an
agents: list (researcher, planner, implementer, builder, tester, fixer,
linter); softened the three runSubagent({ agent, prompt }) blocks to
tool-agnostic delegation prose.
- code-testing-implementer: tools: [agent, read, search, edit, execute] +
agents: (builder, tester, fixer, linter).
- Leaf agents (researcher/planner/builder/tester/fixer/linter):
tools: [read, search, edit, execute].
Why explicit tools (not ["*"]): VS Code has no all-tools wildcard for ools:
and a subagent's ools: overrides its inherited set, so ["*"] matched nothing
and stripped subagents of file tools (they ran without read/edit/search). The
CLI treats ["*"] as all-tools, so this was VS-Code-specific. The portable
aliases agent/read/search/edit/execute map to real tools in both environments;
agents: is ignored by the CLI.
README: document the VS Code chat.subagents.allowInvocationsFromSubagents
setting (off by default) needed for the nested implementer->builder/tester/
fixer/linter layer to fan out on large scopes; the CLI has no such gate.
Validated end to end: VS Code shows researcher->planner->implementer fanning out
with real file I/O (no "subagents lack file tools" warning); CLI fan-out intact
with skills still loading and tests passing. An all-tools baseline used only
tools within this enumerated set, confirming no CLI capability is restricted.
* Include skill tool in code-testing agent tool allowlists
Address PR review: the explicit `tools:` allowlists omitted the `skill`
tool, but every code-testing-* agent's prompt instructs calling skills
(e.g. `code-testing-extensions` for per-language guidance, `test-gap-analysis`,
`assertion-quality`). Because `tools:` is an override, omitting `skill` can
prevent the agents from loading those skills in environments that gate skill
invocation by the allowlist.
Add `skill` to all eight agents:
- orchestrators (generator, implementer): [agent, skill, read, search, edit, execute]
- workers (researcher/planner/builder/tester/fixer/linter): [skill, read, search, edit, execute]
Re-verified in the Copilot CLI: full fan-out (researcher -> planner ->
implementer/tester), the `code-testing-extensions` skill is invoked, and the
generated tests pass.
|
||
|
Amaury Levé
|
691cd472a1 |
Fix writing-mstest-tests activation by deflecting sibling skills (#835)
* Add MSTest deflection to assertion-quality and test-anti-patterns skills The writing-mstest-tests skill failed to activate in the plugin arm for six MSTest-specific scenarios (fix swapped Assert.AreEqual, modernize legacy patterns, type/string assertions, DynamicData with ValueTuples) because sibling skills captured the routing. assertion-quality and test-anti-patterns matched these prompts but lacked explicit deflection to writing-mstest-tests. Add DO NOT USE entries pointing MSTest test writing/fixing/modernizing to writing-mstest-tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: restore general assertion-fix deflection, align test-anti-patterns - assertion-quality: restore the general 'fixing or rewriting assertions' deflection (dropped in the prior edit) alongside the MSTest deflection, and fix punctuation to use comma-separated DO NOT USE items. - test-anti-patterns: align the 'writing new tests' deflection to also point MSTest test-writing to writing-mstest-tests, matching the body guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
592a53008e |
skill-validator: restore 15K aggregate cap as the real Copilot CLI skill-menu budget (#803)
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget The per-plugin aggregate description cap had been raised 15,000 -> 20,000 -> 22,000 under the belief that 15K was 'a local repo policy, NOT a documented Copilot constraint'. That belief was wrong: the GitHub Copilot CLI renders the model-facing <available_skills> menu under a hard 15,000- char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with their full <description> only until the budget is exhausted; every skill past the cut-off collapses to a bare name with no description and can no longer be reliably model-activated. Raising the validator cap merely masked this silent menu truncation — e.g. dotnet-test's run-tests and test-* skills stopped activating in plugin eval runs because they fell into the name-only overflow. Changes: - SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the comment rewritten to document the real Copilot CLI budget (and correct the prior 'not a documented constraint' claim). - CheckCommand aggregate now excludes skills marked 'disable-model-invocation: true' — the CLI drops those from the menu, so they do not consume the budget. This makes the cap satisfiable by hiding reference / agent-orchestrated primitives rather than only by trimming. - InvestigatingResults.md: document plugin-arm-only non-activation caused by skill-menu budget overflow, and how to fix it. Note: dotnet-test currently exceeds 15K and must be slimmed below it (via disable-model-invocation on reference/primitive skills plus description trims) before this cap can go green repo-wide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skill-validator: use source-generated regex for disable-model-invocation check Address review: replace Regex.IsMatch(pattern-string) with a [GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup), matching FrontmatterParser's style. Runs once per skill during checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skill-validator: parse disable-model-invocation via YAML to avoid block-scalar false positives The regex-based check matched any line in the frontmatter, so a block-scalar description that merely mentioned 'disable-model-invocation: true' on its own line was wrongly treated as disabling model invocation. Parse the frontmatter with the existing YAML deserializer (which correctly handles block scalars) by adding a DisableModelInvocation field to SkillFrontmatter, and drop the regex entirely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
35ace775b3 |
Add dotnet-test-migration plugin and move migration skills there (#808)
Move the .NET test framework/platform migration skills (migrate-mstest-v1v2-to-v3, migrate-mstest-v3-to-v4, migrate-vstest-to-mtp, migrate-xunit-to-mstest, migrate-xunit-to-xunit-v3) and the test-migration orchestrator agent out of dotnet-test into a new dedicated dotnet-test-migration plugin, along with their evals. Update all marketplace manifests, READMEs, CODEOWNERS, and .vally.yaml accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
29bfae5f5b |
run-tests: fix evals (query-filter regex, sibling skills, command observability) (#800)
* run-tests: fix evals (query-filter regex, sibling skills, command observability)
Investigated run-tests eval failures by running the validator locally.
- SKILL.md Step 3: document xUnit v3 --filter-query so the agent stops
answering that complex xUnit v3 filters 'cannot be combined'.
- eval.yaml: fix a broken assertion regex. The query-filter pattern is a
single-quoted YAML scalar using '\\s'/'\\[', which (unlike a double-quoted
scalar) is NOT unescaped, so the regex searched for a literal '\s' and
could never match. Corrected to single backslashes.
- eval.yaml: add additional_required_skills (filter-syntax / platform-detection)
to the filter and detection scenarios, so the isolated arm loads the sibling
reference skills that run-tests explicitly defers to.
- eval.yaml: ask the agent to show the exact command in execute-style prompts.
output_matches only sees the final assistant message; 'run my tests' prompts
make the agent execute and summarize ('tests passed'), so the recommended
command never appears. The assertions still catch wrong commands.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: rewrite description for reliable plugin-arm activation
The run-tests skill activated reliably in the isolated eval arm but
unreliably in the plugin arm (all dotnet-test sibling descriptions
loaded), so every scenario's pluginImprovementScore went negative and
dragged min(isolated, plugin) below zero.
Lead the description with natural-language intent triggers that mirror
how the eval prompts phrase requests (run all tests, run a subset via
filters, produce TRX reports, collect crash/hang dumps, run a single
TFM) instead of opening with platform-detection mechanism, and add
explicit DO NOT USE redirects to code-testing-agent / mtp-hot-reload.
Stays under the 1024-char description cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dotnet-test: fix run-tests plugin-arm activation via skill-menu budget
Root cause (verified against the Copilot CLI SDK skill renderer): the
model-facing skill menu has a 15000-char budget. Skills are listed
alphabetically and emitted with full descriptions only until the budget
is exhausted; the rest collapse to bare names with no description and
effectively cannot be model-activated. With 27 dotnet-test skills,
run-tests (alphabetical position ~20) fell into the name-only overflow,
so it never activated in the plugin eval arm even though it activated
reliably in isolation. This is a real user-facing discoverability bug,
not just an eval artifact.
Fix: hide reference/primitive skills that are never meant to be
model-invoked from the menu via 'disable-model-invocation: true', which
the SDK filters out of the budget entirely:
- filter-syntax, platform-detection, dotnet-test-frameworks,
code-testing-extensions, test-analysis-extensions — already
user-invocable:false reference data ('DO NOT USE directly').
- find-untested-sources, find-untested-sources-polyglot — researcher
primitives invoked by-name from the code-testing-researcher agent
(which has a manual fallback); no standalone evals.
These remain invocable by explicit name (agents/users), only auto-
suggestion is suppressed.
This frees enough budget that run-tests (plus migrate-xunit-to-xunit-v3
and mtp-hot-reload) now receive full descriptions; no previously-visible
skill regresses. Also trimmed the run-tests description so its menu block
fits with margin while keeping all activation triggers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
e3f0becd4c |
Fix test-gap-analysis skill activation in plugin eval runs (#801)
* Fix test-gap-analysis skill activation in plugin eval runs In plugin-mode eval runs (all dotnet-test skills loaded), test-gap-analysis lost activation: 'Acknowledge well-tested code' was stolen by assertion-quality and 'logic/null-check gaps' loaded no skill at all. Front-load the concrete 'would my tests catch a bug if the code changed' trigger phrasing the prompts use, and add a DO NOT USE -> test-gap-analysis redirect in assertion-quality for mutation-style reasoning. Both descriptions stay within the 1024-char cap; aggregate unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Jest/TS activation signal for assertion-quality polyglot scenario The earlier trim of assertion-quality's description dropped the Jest/Vitest and toBeTruthy() signals, breaking plugin-mode activation for the 'Polyglot: Jest/TypeScript' scenario (agent answered directly, loading no skill). Restore those polyglot signals, and make the scenario prompt realistic — the prior prompt enumerated the full check catalog and Jest matcher list inline, acting as an answer key that let the agent self-serve without loading the skill. Verified locally: the scenario now activates assertion-quality in plugin mode across runs; all other assertion-quality scenarios still activate; description stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
03f06321a4 |
Cover MSTESTxxxx analyzer diagnostics in writing-mstest-tests skill (#794)
* Cover MSTESTxxxx analyzer diagnostics in writing-mstest-tests skill Add a 'Fix MSTest analyzer diagnostics' workflow step mapping the common MSTESTxxxx rules to their idiomatic fixes, plus MSTestAnalysisMode guidance, instead of creating one skill per rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix MD012 markdown lint (trailing blank line) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: reword analyzer availability and fix table grammar - Don't tie MSTest.Analyzers availability to TestFramework 3.7; note metapackage/SDK/explicit reference. - Fix grammatically broken fix text for the MSTEST0002-0014 layout row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify the MSTESTxxxx table is non-exhaustive; defer to full reference Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0c98ce6bd9 |
Direct strategy must still run the Step 7 pre-completion gate (#793)
* Direct strategy must still run the Step 7 pre-completion gate The Direct strategy correctly skips the research/plan/implement sub-agents for small single-file tasks, but the wording let agents also skip the Step 7 pre-completion gate (test-gap-analysis + assertion-quality + scenario coverage) — treating a single-file task that enumerates specific behaviors as 'trivially small'. This is the dominant failure mode observed on behavior-enumerating tasks: the agent writes one test file directly and finishes with no assertion-strength or scenario-coverage check, producing weak assertions (mutation survivors) and missing required edge/negative cases. Clarify in both the generator Step 2 strategy table and the code-testing-agent SKILL.md that Direct trades away only the sub-agents, never the gate, and that a request naming a specific symbol or enumerating scenarios is not 'trivially small' and must run the gate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: align Direct gate trigger with Step 7 threshold; clarify gate in SKILL.md - Step 2 Direct cell no longer introduces a separate 'names a specific symbol' gate trigger that contradicted Step 7. It now defers to Step 7's own threshold (>=5 tests, or any enumerated behaviors/scenarios). - SKILL.md now names what/where the gate is: the generator's Step 7 (test-gap-analysis + assertion-quality). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Amaury Levé <evangelink@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5d717dbdd1 |
Add prompt-scenario coverage check to code-testing-generator gate (#789)
* Add prompt-scenario coverage check to code-testing-generator gate The pre-completion gate already verifies assertion strength (pseudo-mutation and assertion-depth checks), but two recurring failure modes still slip through when the prompt enumerates specific behaviors: - Testing an *adjacent* function/helper instead of the exact feature named in the objective, leaving the requested behavior uncovered. - Covering only a single representative case when the scenario wording implies multiple variations or pins a condition to a specific position or structure. Add a third gate item that maps each enumerated scenario to a dedicated test, requires targeting the exact named function (preferring the canonical existing test file), and requires honoring range/positional qualifiers literally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: genericize example, fix gate-count consistency - Remove benchmark-specific symbol names from the target-the-named-function bullet to avoid overfitting; phrase it generically. - Fix the gate intro that said 'The two skills below' now that there are three numbered items (the third is a prompt self-review, not a skill). - Update Step 8 and Rule 11 so re-running the gate includes the new prompt-scenario coverage check, not just test-gap-analysis + assertion-quality. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Amaury Levé <evangelink@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
14d727fee6 |
Fix test-anti-patterns skill activation for 5 evals (#786)
* Fix test-anti-patterns skill activation for 5 evals Sibling skills with overlapping descriptions were stealing activation from test-anti-patterns in plugin eval runs (coverage-analysis, assertion-quality, test-smell-detection). Reword descriptions so test-anti-patterns owns the umbrella 'audit my tests for anti-patterns' severity-ranked report, and add DO NOT USE redirects in the metric-focused siblings. Kept all descriptions within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use toBeTruthy() call form in assertion-quality example Addresses review feedback: the Jest matcher example read like a property without parentheses. Description stays within the 1024-char cap (1023). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval validation error and improve activation triggers - assertion-quality eval.yaml/eval.vally.yaml: replace hyphenated 'assertion-quality' (the target skill name) with spaced 'assertion quality' in two scenario prompts, fixing the 'prompt mentions target name' validation error that biased baseline runs. - test-anti-patterns description: add 'what's wrong with my tests' / 'are these tests any good' / 'flaky tests' trigger phrasing to improve organic activation for the flakiness, well-written and polyglot scenarios (which intermittently failed to activate in plugin runs). Stays within the 1024-char description cap (1007). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use Jest matcher call form in assertion-quality polyglot prompt Addresses review feedback: write toBeDefined()/toBeTruthy()/not.toBeNull()/ toBe()/toThrow() in call form in the prompt so they read as matcher calls, consistent with the skill description examples. Regex assertions and rubric left untouched (they match agent output, which may use either form). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Sharpen test-anti-patterns description for flakiness/polyglot activation The flakiness and Python-pytest scenarios failed to activate even in isolated runs (where it's the only candidate skill), because their prompts enumerate the methodology and the description's keywords were too generic. Front-load the concrete trigger keywords those prompts use: Thread.Sleep, DateTime.Now, time.sleep, order-dependent, reflection coupling, and Python/pytest. Stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make mixed/flakiness eval prompts realistic to fix plugin activation The mixed-severity and flakiness scenarios consistently failed to activate test-anti-patterns in plugin runs (detected=[] — the agent loaded no skill at all and answered directly). Both prompts enumerated the full anti-pattern catalog inline, acting as an answer key that made the agent self-sufficient. Replace the embedded checklists with realistic user asks while keeping the 'for .NET test anti-patterns' trigger, file references, severity-ranked output format, and read-only constraint. Rubric and output_matches assertions are unchanged — they validate the produced report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0c2b9710ee |
Fix activation for OutputType=Exe Directory.Build.props scenario (#785)
The 'Set OutputType=Exe only for test projects in Directory.Build.props' eval scenario was failing skill activation: the model answered from its own knowledge and proposed the IsTestProject condition the skill explicitly warns against. Strengthen the migrate-vstest-to-mtp SKILL.md description so the skill router matches this scenario: add the literal trigger phrase and surface the MSBuildProjectName (correct) and IsTestProject (anti-pattern) keywords in the USE FOR clause. Trimmed lower-value text to stay within the 1024-char description limit (1011). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f389af86c7 |
code-testing-generator: mandatory pre-completion self-review gate (#768)
* code-testing-generator: mandatory pre-completion self-review gate Replaces the prose `Verify tests are implementation-specific'' bullet in Step 7 of the code-testing-generator agent with a mandatory pre-completion self-review gate that invokes two existing plugin skills: 1. `test-gap-analysis'' (pseudo-mutation check) against the source files tested and the produced test files 2. `assertion-quality'' (trivial/tautological assertion check) against the produced test files Both skills already ship in plugins/dotnet-test; this PR only wires them into the generator's workflow as a mandatory gate before declaring a run complete. The two skills' `When to Use'' sections are extended to list `called by code-testing-generator as a pre-completion self-review step'' as a recognised use case so the model does not refuse the invocation. The skill descriptions are unchanged (frontmatter is already at the 1024-char limit). Rule 11 of the generator agent is updated to list the gate alongside final build, final test, and coverage-gap review as mandatory for ALL strategies including Direct. A matching rubric item is added to the ContosoUniversity scenario in the code-testing-agent eval (yaml and vally) so the LLM judge can verify the gate was actually invoked on the trajectory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
11fd40e04c |
Add worked example files for C++, PowerShell, Ruby, Rust, Kotlin (#775)
* Add missing language worked examples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: fix runnable example bugs (PS type/scope, Rust impl Trait arg, Catch2 macros) - powershell: accept [pscustomobject] in Get-InvoiceTotal; assert \ - rust: use concrete fn pointer type instead of impl Trait as generic arg - cpp: use REQUIRE_THROWS_WITH for message-substring assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2bdec9b741 |
Expand C++ testing extension: coverage instrumentation, frameworks, CMake/ctest setup (#777)
* Expand C++ testing extension guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: group find -o predicates, include AppleClang, note libgtest-dev caveat Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
fc1e54f1a4 |
Harden Python testing guidance: reuse repo test runner/layout, handle heavy deps, never ship failing tests (#776)
* Harden Python testing guidance: reuse repo runner/layout, handle heavy deps, never ship failing tests Python is the weakest benchmark language (grade ~0.31): ~60% of generated tests fail and coverage stays ~2%. Trajectory analysis showed heterogeneous causes — custom runners not discovered (Django), custom layouts (ansible test/units), native-import failures (PyTorch), and over-reach that ships failing tests (Flask/FastAPI), which zeroes the grade via the pass-all gate. - Discover and reuse the repo's own test runner/layout before writing tests. - Add Django runner / DJANGO_SETTINGS_MODULE / subdir-runner guidance. - Verify heavy/native modules import before testing; otherwise scope down or skip. - Add a green-suite-or-remove finalization rule (delete tests that can't pass). - Start with high-certainty pure-function tests to bank coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: reword nonexistent Skip section reference Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: clarify env-wrapper usage for module vs script/-c, import probes, and green-suite check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
754011b5ee |
Add workspace-integrity guardrail to code-testing agents (#773)
The code-testing-generator/implementer agents could treat an unusual or
scaffolded workspace (e.g. a gutted repo with an injected synthetic module)
as corruption and "repair" it with git checkout/restore/reset/clean or rm,
restoring deleted tracked files and testing the wrong code.
- Replace generator Rule 5 ("Clean git first - stash changes") with an
explicit "Treat the workspace as delivered" rule, and add a "Never mutate
version control" rule. Output must be purely additive test files.
- Add a no-revert/no-clean invariant to the implementer's edit boundaries.
- Add a 'workspace integrity' eval to the code-testing-agent suite
(eval.yaml + eval.vally.yaml). The fixture looks gutted: a metricsd project
whose real core/io modules are committed at HEAD but deleted from the
working tree, leaving only a synthetic 'synthstr' decoy. A git restore would
resurrect the deleted sentinel files; graders fail if they reappear and
require passing pytest tests for the module as delivered.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
a98fb44e52 |
code-testing-agent: pin down behavior in generated tests (#767)
Adds a new `Write Tests That Pin Down Behavior'' section to `unit-test-generation.prompt.md'' covering five universal test-craft principles: 1. Mutation thinking - each assertion would fail under a plausible bug 2. Property intersections - test combinations, not only coordinate axes 3. Behavior radius - assert on at least one secondary observable 4. Fixture realism - never set the parameter under test to a degenerate value 5. Quick self-review before declaring a test method done Mirrors the same depth requirements in `code-testing-implementer.agent.md'' Step 4 as a cross-language invariant block alongside the existing `Edit boundaries'' rules. Extends the existing `code-testing-agent'' eval rubrics (yaml and vally) with one or two depth-oriented bullets per scenario: * ContosoUniversity: minimal IsNotNull-only assertions + secondary observable check on controller actions * python-flask-tasks: minimal `is not None''-only assertions + at least one combined-property TaskService validation test * typescript-vitest-cart: minimal `toBeDefined''/`toBeTruthy''-only assertions + at least one intersection test (discount + tax + shipping together) Rationale and prior-art references in PR description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
76b4c73a39 |
Add find-untested-sources-polyglot skill (tree-sitter, 9 languages) (#735)
* Add find-untested-sources-polyglot skill (tree-sitter, 9 languages) Sibling of the C# find-untested-sources skill that uses tree-sitter (via tree-sitter-language-pack) to extract declarations + imports across Python, TS/JS, Go, Java, Rust, C#, and Ruby with no build step. Output schema mirrors the C# skill's so prompts can consume either tool. Pairing strategies per language: - Import resolution (Python module path, TS/JS relative ./../index.*, Go pkg/<name>.go, Java FQCN, Rust use::, Ruby require). - Identifier overlap (every >=4-char token in the test source is cross-referenced with the declared-name index). Smoke-tested on: - python-flask-tasks fixture: 8 source / 0 test, 8 untested. - typescript-vitest-cart fixture: 8 source / 0 test, 8 untested (node_modules pruned). - AITestAgent C# repo: 3138 source / 761 test, 1419 tested, 1719 untested, 15 orphan. - gh-skills Go repo: 14 source / 7 test, 8 tested, 6 untested, 0 orphan. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Polyglot: union structure+symbols for declarations (fixes Go types, Python methods) tree-sitter-language-pack's ProcessResult exposes two complementary views: - structure: top-level items (classes, functions, structs, traits) - symbols: flat declared-name list Neither alone is complete: - Go: structure has methods/functions but NOT 'type' declarations; symbols has the types. Without union, a test that references a struct name finds no matching source and the source is incorrectly marked untested. - Python: structure stops at top-level, symbols also lists nested methods. - Rust: structure has structs and impls; symbols has fn names inside impls. Fix: union both lists, then drop kind='module'/'namespace' (these are packaging items that would cause false positives, e.g. Java's tree-sitter output emits the package name 'com' as a Module). Verified across all 9 supported languages with synthetic source+test+orphan fixtures (python, javascript, typescript, tsx, go, java, rust, csharp, ruby): each fixture now reports tested=1, untested=1, orphan=0. Real-repo regression check (AITestAgent C#): tested 1419 -> 1429, untested 1719 -> 1709 (10 sources now correctly paired via types previously missing from the declared-name index). Orphan count unchanged at 15 (no false-positive pairings introduced). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback on find-untested-sources-polyglot Code fixes (find_untested_sources.py): - Java test-file detection: drop the `name.startswith("test")` clause so data files like `testdata.java` are no longer misclassified as tests. Keep the documented `*Test.java` / `*Tests.java` rule and the path segment heuristic. - C# test-file detection: tokenize the (PascalCase-preserving) stem and treat the file as a test only when the final word is `Test` or `Tests`. This correctly classifies `UserServiceTests.cs` while no longer misclassifying `Contest.cs`, `Latest.cs`, `Manifest.cs`. - Identifier-overlap pairing: iterate the test's referenced identifiers and do O(1) lookups in `by_decl` instead of scanning every declaration in the index. Pairing is now O(#test_identifiers) per test instead of O(#decls × #tests) across the run, which matters on large repos. - JS/TS relative-import normalization: collapse `<seg>/../` segments in a fixed-point loop instead of a single regex pass so chained imports like `../../foo` and `a/b/../../c` resolve fully. Doc fix (SKILL.md): - Clarify that `symbols` is unioned with `structure` rather than being used only as a fallback when `structure` is empty, matching the actual parse_file implementation. - Tighten description to stay within the per-skill (1,024 chars) and plugin aggregate (20,000 chars) limits enforced by skill-validator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Allow tree-sitter.github.io in known-domains for find-untested-sources-polyglot The polyglot skill's SKILL.md links to https://tree-sitter.github.io/ for the tree-sitter project; add the domain to eng/known-domains.txt so the skill-validator reference scanner stops failing with EXTERNAL-DOMAIN. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #735 review feedback (round 2) - parse_file: log a stderr WARN when tree-sitter parsing throws instead of silently returning a 0-declaration FileInfo, so users can see when results are degraded by parser failures. - Python relative imports: _python_module_to_path now respects PEP 328 leading dots, resolving 'from .utils import x' against the test file's package rather than treating it as an absolute 'utils.py' at repo root. Returns an empty candidate set when no test context is available or when the imported name isn't captured ('from . import x'), to avoid false-positive pairings. - Go import resolution: _build_indexes now produces a by_filename index, and _resolve_test_imports looks up '<pkg>.go' in O(1) per import instead of scanning every source path for each import target. - tested_sources: emit sorted by path for deterministic JSON output across runs (matches the existing sort on untested_sources / orphan_tests). - SKILL.md: JS/TS test-detection row now lists the 'test' path segment to match the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d0924d4f17 |
Add find-untested-sources skill (C# parse-only test pairing) (#733)
* find-untested-sources: C# static test-pairing prototype Parse-only Roslyn analysis that maps C# source files to the test files referencing their declared types and lists production sources with no referring test. No Compilation, no MetadataReferences, no binding. - File discovery prunes bin/obj/node_modules/.git and generated *.g.cs. - Test classification by .csproj suffix or test-SDK reference. - Source index records (ShortName, Namespace, FilePath) per parsed file. - Test scan walks IdentifierTokens, disambiguates strictly against the test file's using directives + enclosing namespace. - Suggests a test-file path by mirroring the source under the test project that already <ProjectReference>s the source's project. Runs in ~9s on a ~3,900 .cs-file repo (AITestAgent + msbench fixtures). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback on find-untested-sources Code fixes (Find-UntestedSources.cs): - ProjectFor: sort .csproj matches and wrap enumeration in try/catch so selection is deterministic when a directory contains multiple project files. - GetNamespaceFor: walk all enclosing namespace declarations and join from outermost to innermost so nested namespace blocks (namespace A { namespace B { ... } }) return "A.B" rather than just "B". - Test scan: stop unconditionally skipping global-namespace declarations. Global types are visible without a using directive, so accept them as visible to any test file. Previously, types without a namespace were never attributed to any test and always reported as untested. - BuildProductionToTestProjectMap: sort the recursive .csproj enumeration and wrap in try/catch so the "first write wins" rule selects the same test project across runs/machines regardless of filesystem ordering. - IsSkippedFile: switch suffix comparisons to OrdinalIgnoreCase and collapse the five if statements into a single loop over an array. Skill metadata (SKILL.md): - Tighten the description so the skill stays under the 1,024-character per-skill limit and the dotnet-test plugin stays under the 20,000-char aggregate limit enforced by skill-validator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d9c1e7a801 |
code-testing-agent: mention find-untested-sources for C# discovery (#734)
* code-testing-agent: mention find-untested-sources for C# discovery Adds a conditional pointer (gated on 'when available') to the find-untested-sources skill in two places: - SKILL.md Step 3 (Research Phase): high-level note for C# / .NET multi-file scopes — prefer the helper over manual find/grep/glob walks. - code-testing-researcher.agent.md Section 7 (Discover Preexisting Tests): directive instruction telling the researcher to invoke the helper before manually pairing source <-> test files, and to use its source_to_tests / untested output to fill the research document. Both callouts are phrased as 'when available', so installations without the find-untested-sources skill continue to work via manual discovery. Adds no behavior for non-C# repos. Context: in a 5x136-instance internal experiment on the msbench .NET test bench, adding equivalent pointers to the routed code-testing-agent yielded a 15.67% input-token reduction at neutral pass rate — the model trusted the documented pairing heuristics and skipped its own discovery walk. The helper itself was not invoked in those runs (the Copilot CLI router did not auto-load the sibling skill), so the measured win comes from the doc text causing the model to short-circuit its manual exploration, not from the helper executing. Depends on dotnet/skills#733 (which adds the find-untested-sources skill itself). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * code-testing-agent eval: grade find-untested-sources usage on C# scenario PR #734 adds a doc pointer steering the researcher toward the `find-untested-sources` skill on .NET / C# multi-file tasks. The existing ContosoUniversity scenario is the natural fit: it's the only C# multi-file scenario in this eval, and the new pointer is .NET- scoped. Add one rubric item to that scenario (in both eval.vally.yaml and the legacy eval.yaml) asking the grader to verify the researcher actually leveraged the helper — either by citing its `source_to_tests` / `untested` JSON output in `.testagent/research.md`, or by executing `scripts/Find-UntestedSources.cs` — instead of falling back to manual `find` / `grep` / `glob` walks. Gated on `when available in the workspace` to match the SKILL.md wording, so installs without find-untested-sources are not penalized. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
8854249222 |
migrate-xunit-to-mstest: forbid CancellationToken.None replacement explicitly (#758)
* migrate-xunit-to-mstest: forbid CancellationToken.None replacement explicitly The TestContext.Current.CancellationToken guidance already forbids replacing it with a fresh CancellationTokenSource, but real-world agent migrations have been observed substituting CancellationToken.None instead, which silently drops the test host's cancellation linkage. Call CancellationToken.None out explicitly so both common shortcuts are covered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: mention property injection for MSTest < 3.6 Per Copilot review on PR #758: the section above notes that projects pinned to MSTest < 3.6 must use property injection. Reword the CancellationToken callout to cover both constructor and property injection so the guidance stays correct in that supported scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0a387fcdfe |
code-testing-implementer: add mandatory harness-discovery check (#757)
* code-testing-implementer: add mandatory harness-discovery check
Generic CI/benchmark verifiers run the framework's default discovery from the repo root. Tests that pass via a scoped command (e.g. 'dotnet test MyProject.Tests.csproj', 'bundle exec rspec subgem/spec', 'Invoke-Pester -Path ./tools') but are invisible to the harness count as 0 generated tests.
Repeatedly observed in msbench cta-nightly runs:
- new C# test project never 'dotnet sln add'ed (ocelot)
- Pester tests placed under custom directories invisible to default Invoke-Pester (homebrew, ruby/ruby)
- RSpec specs placed in a sub-gem's spec/ dir invisible from repo root (fastlane)
Changes:
- code-testing-implementer: capture baseline test count in Step 2, new Step 7 'Verify Harness Discovery (MANDATORY)' with concrete failure examples, HARNESS_DISCOVERY line in report template, reminder after Step 3 to revisit registration if Step 4 creates a new project.
- code-testing-researcher: instruct to record BOTH scoped test command and harness-equivalent discovery command in .testagent/research.md.
- dotnet.md: strengthen 'Registering a new test project' heading to MANDATORY when dotnet new was used; add 'Harness Discovery Check' section with 'dotnet test <solution> --list-tests' from repo root.
- powershell.md: add 'Harness Discovery Check' section with default-config Invoke-Pester from repo root.
- ruby.md: add gem-monorepo trap guidance (fastlane, ruby/ruby) in Test Placement Contract; add 'Harness Discovery Check' section with 'bundle exec rspec --dry-run' from repo root.
* Address review feedback on harness-discovery check
- dotnet.md: replace stale `Step 8 cleanup` reference with the actual Step 3 (`Register Test Project with Build System`).
- dotnet.md: demote `Harness Discovery Check` to `###` so it nests under the parent `## .csproj / .sln Handling` section alongside `### Registering...`.
- dotnet.md: replace `\s\{4\}` (non-POSIX) in the grep regex with 4 literal spaces so the count works under BRE.
- ruby.md: group the rake/rails fallback with `{ ...; } | wc -l` so the pipe applies to both branches of `||`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
9be44ad6e5 |
migrate-xunit-to-mstest: add learnings from dotnet/sdk migration (#756)
* migrate-xunit-to-mstest: add learnings from dotnet/sdk migration Refines the skill based on a real migration of 5 test projects in dotnet/sdk: - MSTest.Sdk implicit global using: do not add `<Using Include='Microsoft.VisualStudio.TestTools.UnitTesting' />` or per-file `using` in MSTest.Sdk projects (it's already in scope). Note Option A (`MSTest` metapackage) still needs the per-file using. - `Assert.IsExactInstanceOfType<T>` (MSTest 4.1+) is the proper single-call equivalent of xUnit's exact-type `Assert.IsType<T>`; previous guidance silently degraded it to assignable semantics (closes #755). - `Assert.AreSequenceEqual` (MSTest 4.3+) is the modern element-wise equivalent of xUnit's `Assert.Equal` on `IEnumerable<T>`, avoiding the `CollectionAssert` + `.ToList()` dance and the MSTEST0065 trap on plain `AreEqual`. - Clarify `AwesomeAssertions` ships in the `FluentAssertions` namespace, so it's a no-source-change swap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * SKILL.md: split IsType/IsAssignableFrom row to avoid implying drop-in equivalence Address dotnet/sdk#54727 review comment: the Step 6 summary table conflated xUnit's exact-type Assert.IsType<T> and assignable Assert.IsAssignableFrom<T> into a single row, which implies both map interchangeably even though their semantics differ. Split into three rows -- IsType<T> -> IsExactInstanceOfType<T>, IsNotType<T> -> IsNotExactInstanceOfType<T>, IsAssignableFrom<T> -> IsInstanceOfType<T> -- and explicitly call out the silent-weakening trap. The cheatsheet already maps these correctly; this aligns the SKILL.md summary with the cheatsheet so the at-a-glance table doesn't suggest a wrong mapping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * SKILL.md: resolve Step 3/Step 4 conflict on the MSTest using for Option B Address review feedback on dotnet/skills#756: Step 3 said to skip the per-file 'using Microsoft.VisualStudio.TestTools.UnitTesting;' on Option B (MSTest.Sdk provides it as an implicit global using), but Step 4 unconditionally instructed replacing xUnit usings with that MSTest using -- so an agent following Step 4 literally would re-add the redundant using on Option B projects. Also drop the dangling 'Step 4 rewriter' phrasing in Step 3 (there is no rewriter in Step 4, just a list of mechanical rewrites the agent applies). Step 4 now explicitly says to skip the MSTest using on Option B and only remove the 'using Xunit;' lines, matching Step 2's note and Step 3's branch instructions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Evangelink <amaury@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
2ab3501158 |
Add .codex-plugin/plugin.json manifests for Codex CLI plugin install (#726)
The Codex CLI requires .codex-plugin/plugin.json as the plugin manifest entry point. Without it, 'codex plugin add' fails with 'missing plugin.json' even though the marketplace listing works. This adds .codex-plugin/plugin.json to all 14 plugin directories, with paths relative to the plugin root per the Codex docs. Also updates the agents marketplace to use dotnet-aspnetcore (per #711 rename) and adds missing dotnet-blazor and dotnet11 entries. Fixes #578 Fixes #724 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
803b91f167 |
Add grade-tests skill for per-test PR-comment quality grading (#727)
* Add grade-tests skill for per-test quality grading Adds a new polyglot skill that grades a provided list of test methods on a letter scale (A-F) with a score band and 1-line notes, suitable for posting in a PR comment table. Complements existing audit skills (assertion-quality, test-anti-patterns, test-smell-detection) which produce suite-wide reports. The skill: - Takes an explicit list of tests (refuses to grade a whole workspace silently) - Uses a 3-dimension rubric (Assertion strength, Structure & focus, Anti-pattern hygiene) - Combines via 0.45/0.30/0.25 weighting with safety caps - Starts at A and only deducts for observable issues - Does not flag idiomatic patterns (table-driven sub-tests, bare assert, etc.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback - OrderProcessorTests.cs: fix 4 invocations that mixed named arg customerId: with subsequent positional args -- invalid C# syntax (Copilot review on PR #727) - README.md: update 'five test-analysis skills' to 'six' in three places now that grade-tests is added to the polyglot list, and clarify that only the original 5 load from test-analysis-extensions while grade-tests has an inline rubric Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * eval: drop 'grade-tests skill' wording from scenario prompts The skill-validator's evaluate pre-flight check ValidateEvalPrompts rejects any prompt that names the target skill, because mentioning the skill biases baseline (skill-not-loaded) runs and inflates measured lift. The two scenarios that asked the agent to 'use the grade-tests skill' now just ask it to grade test methods individually -- the agent is still expected to pick the right skill organically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix rubric consistency per review feedback - README: drop incorrect 'inline rubric instead' claim (grade-tests also loads test-analysis-extensions for language-specific guidance) - Assertion sub-grade: literal always-true assertions are now F (was D), matching the eval expectation that 'assert True' produces overall F - Anti-pattern catalog: split always-true literals (now -> F) from self-referential assertions (still -> D) — they were conflated - Combination rule: overall grade is now capped at the worst sub-grade (was capped at D), so an F in any one dimension yields overall F --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
812565f8d1 |
Add polyglot evals for code-testing-agent (Python Flask + TypeScript Vitest) (#709)
* Add polyglot evals for code-testing-agent (Python Flask + TypeScript Vitest) Adds two non-.NET evaluation scenarios to the dotnet-test/code-testing-agent eval to measure how well the agent generates idiomatic tests in Python and TypeScript — the languages exercised by the msbench top5-* benchmarks. New fixtures (no pre-existing tests, so passing pytest/vitest proves the agent actually generated them): - fixtures/python-flask-tasks/ — small Flask app with TaskService (injected clock), TaskRepository Protocol + InMemoryTaskRepository, and a /tasks blueprint with HTTP 201/200/400/404/409 surfaces. Validated locally with `python -m pip install -e ".[test]" && python -m pytest`. - fixtures/typescript-vitest-cart/ — small shopping-cart library with an injectable DiscountPolicy seam (No/Percentage policies) and a Cart class with non-trivial merge / clamping semantics. Validated locally with `npm ci && npx vitest run`. package-lock.json committed so `npm ci` is reproducible in CI. New scenarios (added to both eval.yaml and eval.vally.yaml): - "Generate pytest tests for the Flask tasks API (Python polyglot)" — asserts that `python3 -m pip install -e '.[test]' && python3 -m pytest` exits 0 and that at least one `tests/**/test_*.py` file was produced. Rubric rewards using Flask's `test_client()`, mocking `TaskRepository` for service-level tests, asserting HTTP status + JSON body, and covering the empty-title / >200-char / not-found / already-done error paths. - "Generate Vitest tests for the shopping-cart library (TypeScript polyglot)" — asserts that `npm ci && npx vitest run` exits 0 and that at least one `tests/**/*.test.ts` file was produced. Rubric rewards mocking `DiscountPolicy` via `vi.fn()`, covering Cart merge semantics, `updateQuantity` zero-removes, `totals()` discount-clamping, and the `PercentageDiscountPolicy` constructor boundary. Companion to #708 (polyglot examples + sub-agent generification); these scenarios exercise the per-language guidance that PR adds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback for #709 - routes.py: guard `create_task` against non-dict JSON payloads (e.g. list) by returning 400 instead of letting `payload.get(...)` raise AttributeError and surface as a 500. This keeps the API behavior stable and makes it easier for generated tests to assert 400 on invalid input shapes. - tsconfig.json: drop `vitest/globals` from `compilerOptions.types`. The fixture's vitest.config.ts sets `globals: false`, so allowing TypeScript to assume globals (describe/it/expect) only enables tests that compile but fail at runtime. Removing it keeps the fixture aligned with the non-global Vitest API the eval is exercising. Both fixtures re-smoke-tested locally: vitest 1/1, pytest 2/2 (including a new test confirming the non-dict body returns 400). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill activation for polyglot scenarios in PR #709 Both polyglot scenarios (pytest + Flask, Vitest + TS) scored 5.0/5 quality but reported `⚠️ NOT ACTIVATED` because the SDK's skill router did not load `code-testing-agent` into the agent's context: `activated: false, detectedSkills: []` in skillActivationIsolated / skillActivationPlugin. The .NET ContosoUniversity scenario activates fine because its prompt has stronger `project-wide, multi-file ... high coverage` framing and SKILL.md already covers .NET keywords. Fixes per `eng/skill-validator/src/docs/InvestigatingResults.md` section "Skill not activated": - SKILL.md description: add framework-specific keywords (pytest, Flask/Django, Vitest, Jest, Mocha, JUnit, Node libraries, API, package, project-wide, multi-file) so the router has explicit hooks for polyglot prompts. Trimmed verbose `DO NOT USE FOR` clauses to stay under the 1,024-char skill spec limit (now 1,019 chars). - eval.yaml + eval.vally.yaml: rewrite the two polyglot prompts to mirror the .NET scenario's framing — add `project-wide, multi-file test generation task across the ... layers` and `achieve high coverage`, soften the library-prescriptive bullets (Mock(spec=...), vi.fn()) into capability-level requirements (`mocked or stubbed`, `mock or hand-written stub`). Rubric items unchanged — they remain flexible enough to grade either mock style. Validated locally: `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` => ✅ All checks passed (23 skill(s), 11 agent(s), 1 plugin(s)). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #709 review + sharpen polyglot skill activation **Activation fix for polyglot scenarios (Python + TypeScript)** The previous attempt ( |
||
|
Amaury Levé
|
7fc6d472c3 |
Add migrate-xunit-to-mstest skill (#718)
* Add migrate-xunit-to-mstest skill Adds a skill to migrate .NET test projects from xUnit.net (v2 or v3) to MSTest v4. Covers package replacement, attribute/assertion/fixture/lifecycle translation, ITestOutputHelper -> TestContext, [Trait] -> [TestCategory]/ [TestProperty], and xUnit v3 TestContext.Current.CancellationToken. Front-loads parallelization handling because xUnit parallelizes test classes by default while MSTest serializes — the single largest source of post- migration regressions. Step 11 enumerates three explicit choices and how to translate CollectionBehavior/MaxParallelThreads/[Collection] settings. Preserves the existing test platform (VSTest stays VSTest; MTP stays MTP) by default — bundling a platform migration would violate the test-migration agent's 'never mix migration steps' rule. - plugins/dotnet-test/skills/migrate-xunit-to-mstest/SKILL.md (workflow) - plugins/dotnet-test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md - tests/dotnet-test/migrate-xunit-to-mstest/eval.yaml (12 scenarios) - tests/dotnet-test/migrate-xunit-to-mstest/eval.vally.yaml (parity with sibling skill) - plugins/dotnet-test/README.md (table entry) - plugins/dotnet-test/agents/test-migration.agent.md (triage routing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix step cross-references flagged in PR review - Step 1.5 'Custom DataAttribute' bullet pointed to Step 6 (assertions); custom DataAttribute mapping lives in Step 5 (data-driven tests). - Step 7 lifecycle table 'Constructor' row pointed to Step 8 for ITestOutputHelper; ITestOutputHelper conversion is in Step 9 (output). - Commit-strategy callout said 'after Step 5 (asserts fixed)' and 'after Step 9 (fixtures/lifecycle rewritten)'; asserts are in Step 6 and fixtures/lifecycle complete after Step 8. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Map [assembly: Xunit.Trait] to [assembly: TestCategory]/[TestProperty] Both TestCategoryAttribute and TestPropertyAttribute target Assembly, Class, and Method, so an xUnit [assembly: Trait] does not need to be demoted to per-class/per-method MSTest attributes — assembly scope is preserved 1:1. - references/mapping-cheatsheet.md §8 now maps [assembly: Trait] to [assembly: TestCategory] / [assembly: TestProperty] instead of 'Remove'. - SKILL.md Step 12 promoted to 'Convert' (was 'Remove'); the same mapping is documented inline. - Notes after the Step 4 and §1 trait tables now disclose the assembly target so users do not assume class/method-only scope. Addresses PR feedback from @Evangelink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address second round of PR review feedback From @Evangelink: - [Ignore]/[Timeout] do NOT discover a test on their own — they are modifiers. SKILL.md Step 4 + cheatsheet §1 now show [Fact(Skip)] -> [TestMethod] + [Ignore(...)] (and same for [Timeout]). Eval scenario 9 rubric updated to require both attributes. - TestProperty IS filterable (was incorrectly described as 'metadata that does not need filter integration'). Notes after the trait tables in both SKILL.md and cheatsheet §1 now show both filter syntaxes. - For environmental gates (OS-/CI-/arch-conditional), point at MSTest 3.10+'s condition attributes ([OSCondition], [CICondition], [ArchitectureCondition], [NonParallelizableCondition]) as the preferred alternative to overloading [TestCategory] or scattering Assert.Inconclusive through test bodies. Strengthened in cheatsheet §3.9, SKILL.md Step 6, Step 10 SkippableFact row, and the Common Pitfalls table. - TestDataRow<T> is also a supported MSTest data source (strongly typed with per-row DisplayName/Ignore metadata). Added to both SKILL.md Step 5 and cheatsheet §2 with link to the docs. - MSTest.Sdk + <UseVSTest>true</UseVSTest> pulls in Microsoft.NET.Test.Sdk automatically — the manual PackageReference was incorrect. Removed it from both files (Step 2, Step 3 'common mistakes', and Common Pitfalls). - Added the 'pin MSTest.Sdk in global.json msbuild-sdks' alternative as the recommended pattern for multi-project solutions. - Xunit.Combinatorial -> Combinatorial.MSTest (Youssef1313 community port). Updated both SKILL.md Step 10 and cheatsheet §10. From Copilot reviewer: - MSTest.Sdk example in SKILL.md Step 2 Option B hardcoded <TargetFramework>net9.0</TargetFramework>, contradicting the preserve-TFM rule. Replaced with a placeholder + explicit comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve xUnit-to-MSTest skill effectiveness based on eval feedback Eval results on PR #718 showed regressions on three rubric items where the agent did the migration correctly but did not COMMUNICATE the judgement-call decisions, so the judge could not verify the choice was deliberate: - ICollectionFixture scope decision (rubric: 'Explicitly explains the scope decision instead of silently widening' scored 1/5 because the agent said 'Let me convert everything' with zero scope discussion). Step 8 now requires a templated explanation of (source scope, target scope, what is shared, what is serialized, why the rejected alternative was rejected) before applying. - Parallelization 'MSTest default is serial' messaging (rubric scored 1.7/5 because the agent only implicitly conveyed it). Step 11 Choice A now demands a templated sentence explaining that the [assembly: Parallelize] is REQUIRED to match xUnit, otherwise the suite silently regresses to serial. - Response Guidelines now lists the three judgement-call decisions that MUST be communicated up-front: fixture scope, parallelization model, and the Throws/ThrowsExactly semantic flip. Additional fixes from this round: - Step 2: explicit guidance to default to Option A (MSTest metapackage) when the user says 'preserve VSTest', so the safer PackageReference path is picked over MSTest.Sdk + UseVSTest. - Step 4: pulled the '[Ignore]/[Timeout] are modifiers, need [TestMethod]' callout out of a table footer into a visible note above the table. - known-domains.txt: allowlist github.com/Youssef1313/Combinatorial.MSTest (referenced from SKILL.md Step 10 and cheatsheet §10 after the recent Xunit.Combinatorial -> Combinatorial.MSTest switch). Validator: green (24 skills, 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add concrete TestContext.Current.CancellationToken migration example The CancellationToken scenario was failing/low-scoring because the skill only contained a one-line mapping with no concrete example. The agent often missed the requirement to: - Add constructor TestContext injection (even when ITestOutputHelper is absent) - Map Assert.False -> Assert.IsFalse alongside the rename - NOT fabricate a CancellationTokenSource as a replacement Add a full xUnit v3 -> MSTest worked example next to the TestContext.Current mapping (Step 9), and reinforce the CancellationTokenSource warning in the cheatsheet. The example covers all 3 rubric points for the scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address third round of PR review feedback - Remove fictional MSTest condition attributes [ArchitectureCondition] and [NonParallelizableCondition] from SKILL.md and cheatsheet (only [OSCondition] and [CICondition] exist in MSTest 3.10+; for non-parallel intent use [DoNotParallelize], for architecture gating fall back to runtime Assert.Inconclusive). (Copilot) - TestPropertyAttribute targets only [Class|Method] (no AttributeTargets.Assembly), so [assembly: TestProperty(...)] does not compile. Update the trait notes in SKILL.md Step 4, cheatsheet section 1, Step 12, and cheatsheet section 8 to collapse a non-category xUnit [assembly: Trait] to [assembly: TestCategory] or push it down to per-class [TestProperty]. (@Evangelink) - Step 1 platform detection: stop inlining a buggy VSTest-vs-MTP matrix; delegate to the platform-detection skill. Note explicitly that <UseMicrosoftTestingPlatformRunner> only affects 'dotnet run' and is not a reliable runner signal. (@Youssef1313) - Step 2 Option A: add MTP code-coverage caveat -- Microsoft.NET.Test.Sdk pulls VSTest's Microsoft.CodeCoverage transitively, which can interfere with MTP's collector. Prefer Option B (MSTest.Sdk without UseVSTest) for MTP projects. (@Youssef1313) - Step 3: remove fictional MSBuild properties (<CaptureConsoleOutput>, <UseRoslynCompilers>); restate the bullet about xunit.runner.json -> Step 11 port instead. Restate the <UseMicrosoftTestingPlatformRunner> bullet to make clear it is not a runner switch. (@Youssef1313) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 4 - mapping-cheatsheet.md (Assert.Contains predicate): the suggested fallback Assert.Contains(collection.First(x => predicate), collection) is buggy -- First() throws InvalidOperationException on no-match (changing the failure mode), and feeding the result back into Assert.Contains is circular. Keep only the correct translation: Assert.IsTrue(collection.Any(x => predicate)). - SKILL.md Step 4 / cheatsheet table: stop instructing 'mark every class sealed'. xUnit projects commonly use base/derived test classes (shared setup, generic base fixtures); mechanically sealing them would break compilation. Sealing is now framed as an optional follow-up handled by writing-mstest-tests, not part of the mechanical migration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ecbecbc6f7 |
writing-mstest-tests: tighten description triggers (#712)
Refine the SKILL.md frontmatter description so the skill router has sharper hooks for common MSTest test-writing prompts: - Add concrete intent triggers (better MSTest assertion than Assert.IsTrue, replace hard cast with MSTest type assertion). - Expand the assertion API surface listed in USE FOR with the modern MSTest 3.x/4.x set (IsInstanceOfType, Contains, ContainsSingle, IsEmpty/IsNotEmpty, DoesNotContain). - Group related triggers (data-driven, lifecycle, parallelization) for readability. - Tighten DO NOT USE FOR (drop redundant clauses) and explicitly exclude xUnit/NUnit/TUnit so the router doesn't grab non-MSTest prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
49a77a90bf |
code-testing-agent: add polyglot pipeline examples for Python/TypeScript/Go/Java (#708)
* code-testing-agent: add polyglot pipeline examples for Python/TypeScript/Go/Java
The code-testing-agent skill family is polyglot in description but in
practice biased toward .NET because dotnet-examples.md was the only
filled-in pipeline walkthrough. The four sub-agents that participate in
the Research-Plan-Implement pipeline (researcher, planner, implementer,
generator) all pointed to dotnet-examples.md whenever they suggested a
concrete example, which made it harder for the agent to produce
idiomatic non-.NET tests (e.g. in msbench top5-* benchmarks for
Python/Flask, TypeScript/Express).
This change
* adds four new example files mirroring dotnet-examples.md format
(source → research → plan → generated test → fix cycle → final report):
- python-examples.md (pytest, unittest.mock, Mock(spec=...), parametrize)
- typescript-examples.md (Vitest with notes for Jest; it.each,
async tests, fake timers, ESM/CJS fix cycle)
- go-examples.md (standard testing package, table-driven subtests,
hand-written fake repository, injected clock)
- java-examples.md (JUnit 5 + Mockito on Maven, @ParameterizedTest +
@CsvSource, Clock.fixed, Surefire fix cycles)
* updates code-testing-extensions/SKILL.md TOC to list the new files
and clarifies usage instructions to read the matching <language>-
examples.md alongside the base extension
* makes the "Concrete example" pointers in code-testing-generator,
code-testing-implementer, code-testing-planner and
code-testing-researcher agents language-agnostic (list all available
example files instead of hard-coding dotnet-examples.md)
* expands code-testing-researcher project-structure detection list to
cover more Python (tox.ini, noxfile.py, requirements*.txt, uv.lock,
poetry.lock, pdm.lock), JS/TS (.mts/.cts/.jsx, vitest.config.*,
jest.config.*), C++ (CMakeLists.txt, BUILD.bazel, meson.build),
Java/Kotlin (pom.xml, build.gradle[.kts], wrappers), and other
ecosystem files; expands the Identify-Language section accordingly
* extends the "Language-Specific Examples" section in
code-testing-agent/SKILL.md to summarise each example file
Validated with: skill-validator check --plugin ./plugins/dotnet-test
(23 skills, 11 agents — all checks passed) and markdownlint-cli2.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #708
- go-examples.md: replace the hand-rolled `contains`/`stringIndex`
helpers with `strings.Contains` from the standard library. The hand-rolled
`contains` had subtly wrong semantics — `contains(""abc"", """")` returned
`false` while `strings.Contains` returns `true` — and the helpers are
unnecessary complexity for a code-generation example.
- go-examples.md: in the `go test -run` "wrong selection regex" sample fix
cycle, quote the test name and use `single_item` (matching the underscore
that the surrounding diagnosis text refers to) instead of the unquoted
`single item` which the shell would parse as two separate CLI arguments.
- java-examples.md: the source-tree file list described `Invoice.java` as a
`record` but the `InvoiceService.markAsPaid` example mutates the invoice
via `setStatus(...)` and `setPaidDate(...)` — records are immutable, so
the description was internally inconsistent. Re-describe it as a mutable
POJO with explicit mutators to match the service code.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
680292e04c |
Make dotnet-test analysis skills and auditor agent polyglot (#707)
* Make dotnet-test analysis skills and auditor agent polyglot Five test analysis skills and the test-quality-auditor agent in plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/ TUnit cues, even though the underlying techniques (assertion diversity, anti-pattern detection, pseudo-mutation gap analysis, academic smell catalog, trait tagging) apply equally to other test frameworks. This change makes them polyglot across .NET, Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest). Changes ------- * New reference skill `test-analysis-extensions` with per-language extension files (dotnet, python, typescript, java, go, ruby, rust, swift, kotlin, powershell, cpp). Each documents test markers, assertion APIs, sleep / time / random APIs, skip annotations, setup/teardown, mystery-guest indicators, integration markers, and tag-support capability for the corresponding framework. * `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`, `test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter now lists all polyglot frameworks (keeping .NET trigger words for backward compat); bodies reference `test-analysis-extensions` and add per-language calibration (e.g., Go/Rust table-driven loops are idiomatic and not Conditional Test Logic; pytest bare `assert` is canonical; missing-await on async assertions is a new Critical smell). * `test-quality-auditor` agent now runs language detection, gates .NET-only pipeline steps (coverage-analysis, CRAP, detect-static- dependencies, testability migration, experimental dotnet skills), and recommends native tooling for non-.NET projects via an explicit Capability Matrix. * README updated to split sections by polyglot vs .NET-only. * New polyglot eval scenarios added (Python/pytest, TS/Jest, Java/JUnit) alongside existing C# scenarios in `test-anti-patterns`, `assertion-quality`, `test-smell-detection`. The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis, crap-score, detect-static-dependencies, run-tests, filter-syntax, platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are unchanged and continue to activate only for .NET work. Validated with skill-validator `check --plugin ./plugins/dotnet-test`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #707 review feedback * typescript.md: replace xpect.fail() recommendation (Jest has no such API) with hrow new Error(...) / explicit failing assertion. * test-tagging SKILL.md description: clarify that unittest has no canonical tag syntax (pytest markers are the auto-edit mechanism; unittest is report-only). Switch doctest reference to "decorators" rather than implying a Catch2-style [tags] mechanism exists. * test-tagging SKILL.md auto-edit list and table: use `* doctest::test_suite("tag")` decorator chain syntax for doctest in both the bullet list and the per-framework table, matching the example already shown. * cpp.md extension: spell out the Catch2 vs doctest tagging distinction in the capability matrix (Catch2 [tag], doctest * doctest::test_suite("tag") decorator). Validated with `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` (23 skills + 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2b7fcdd7ab |
dotnet-test: add Rule #0 (Confirm the Test Target) to ruby/powershell extensions (#702)
* dotnet-test: add Rule #0 (Confirm the Test Target) to ruby/powershell extensions When the prompt does not name a specific file ("test the repository", "one core module", "comprehensive suite"), agents frequently target the wrong code — typically the largest upstream module that already has rich existing tests — instead of the newly-added module the user actually wants tested. Recent benchmark runs (top5-{ruby,powershell}-*-{simple,complex}) showed this is a dominant failure mode for Ruby/PowerShell: the agent burns 50+ turns writing tests for files the verifier never measures, while the real target (a small untracked `lib/string_utils.rb` or `tools/StringUtils.psm1`) sits one `git status` away. Adds a Rule #0 to both ruby.md and powershell.md, ahead of the existing "Rule #1: Investigate the Repo First". The rule: - Tells the agent to use git history (`git status -s`, `git ls-files --others --exclude-standard`, `git log --diff-filter=A --name-only -5`) to find the actual target rather than guessing from repo size. - Documents a Test Placement Contract — RSpec scopes to `spec/`, Pester scopes to whatever directory the harness passes to `Invoke-Pester -Path`; tests placed elsewhere are invisible. - Adds a First-Test Sanity Loop: write one test, run --dry-run / -PassThru to confirm discovery > 0, fix LoadError / Import-Module issues before expanding. Catches placement mistakes on turn 1. Both files validate cleanly under skill-validator. The existing Rule #1 and all subsequent sections are unchanged — this is a pure prepend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address PR review: clarify Rule #0 precedence and broaden require regex - Rephrase Rule #0's discovery preamble in ruby.md and powershell.md to call out the commands as the read-only exception to Rule #1, removing the apparent contradiction between 'before planning' and Rule #1's 'before writing any test or running any command'. - Broaden the spec_helper require grep to match leading whitespace and `require_relative`, avoiding false negatives that send the agent to the wrong target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
3123e607a1 |
dotnet-test: add explicit non-.NET guards to 5 skill descriptions (#705)
* dotnet-test: add explicit non-.NET guards to 5 skill descriptions
Five skills in plugins/dotnet-test have language-neutral trigger phrases
in their USE FOR sections that could plausibly mis-route on non-.NET
prompts in mixed-language workflows:
- writing-mstest-tests ("write unit tests for a class", "create test class")
- test-anti-patterns ("audit my tests", "test smell audit")
- assertion-quality ("identify assertion-free tests")
- test-gap-analysis ("find weak tests", "discover untested edge cases")
- test-tagging ("categorize, audit, or label tests with traits")
Each skill now has an explicit non-.NET exclusion clause in its
DO NOT USE FOR section, naming the most common languages it must NOT
match (Python, JS/TS, Go, Java, Rust, Ruby, PHP, Swift, Kotlin) and
restating that the skill only handles C#/F#/VB.NET test code.
This was surfaced by an MSBench investigation comparing a baseline
Copilot CLI run vs the same run with this pruned dotnet-test plugin
installed against the sweatlas-tw-unit benchmark (24 Python / 15 Go /
5 TS, 0 .NET). Both runs scored identically (15/44 resolved). Skills
correctly never fired (0 invocations across all 43 instances), so this
is a hardening change, not a bug fix — it makes the router's job easier
in heterogeneous repos and future benchmarks.
Validated with: dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin plugins/dotnet-test
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
4f78daf583 |
run-tests: add platform-detection keywords for activation (#700)
* run-tests: add platform-detection keywords for activation
The 'Detect test platform from Directory.Build.props' eval scenario
regressed from 2.7/5 to 2.0/5 with run-tests not activated (skill
loaded column shows NOT ACTIVATED). The scenario prompt asks 'figure
out which test platform it uses' but the previous description framed
detection only as 'chooses the correct platform/SDK/framework syntax'
and 'selecting VSTest vs Microsoft.Testing.Platform command syntax' --
both focused on syntax selection, not on platform detection from
project files.
Per InvestigatingResults.md guidance for 'skill not activated'
failures, surface the detection use case explicitly with the file
names the scenario hinges on:
detecting test platform/framework from \global.json\, \.csproj\,
\Directory.Build.props\
Trim the syntax clause ('command syntax' -> 'syntax', 'including' ->
'incl.') and shorten the multi-TFM example (drop redundant
\dotnet test\ prefix) to stay under the 1,024-char description cap.
Folded description: 948 -> 1,010 chars.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: rephrase platform detection with prompt-shaped language
The previous fix (
|
||
|
Tom Longhurst
|
36209c3b52 |
TUnit Fixes & Improvements (#677)
* Update target framework to net10.0 and upgrade TUnit package to version 1.45.8
* Fix and expand TUnit coverage in dotnet-test-frameworks reference
Cross-referenced against tunit.dev/llms.txt and the official attributes
comparison page. Corrects the test class marker (TUnit is convention-based,
not [ClassDataSource]), and fills in the previously-missing TUnit rows for
the assertion APIs, setup/teardown methods, exception-handling examples,
and integration test category marker.
* Refine TUnit assertion table accuracy
The Type row's TUnit cell now uses IsAssignableTo<T>() to match the
assignable-type semantics of the MSTest/xUnit/NUnit equivalents, with a
note pointing at IsTypeOf<T>() for exact-type checks. The Skip row now
documents that [Skip] also applies at class and assembly scope and that
Skip.Test("reason") handles dynamic in-test skipping.
Verified against the TUnit migration guides and the type-assertions docs.
* Spell out TUnit assertion alternatives as complete awaited expressions
The Boolean, Null, Exception, and Type rows in the assertions table
previously used a `await Assert.That(x).IsTrue()` / `.IsFalse()` shorthand
for the alternative form. A reader copy-pasting just the second fragment
would end up with an un-awaited assertion that silently passes — which
contradicts the note immediately below the table about always awaiting.
Each alternative is now a complete, independently copy-pasteable expression.
Addresses the inline review on dotnet/skills#677.
---------
Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
|
||
|
Amaury Levé
|
ac653c62e4 |
dotnet-test: add cross-language edit-boundary rules to code-testing-implementer (#689)
The code-testing-* agent family generates tests for 12 languages (not just .NET). Until now, no instruction told the implementer to keep its changes additive: it would routinely modify existing test files non-additively (deleting/reformatting lines) or edit non-test production code to make something easier to test. Both behaviors cause the change to be rejected by: - the msbench 'simple' test verifier (any 'dels > 0' on a test file, or any change to a non-test file, sets reward=0) - most real-world test-quality gates, code-review policies, and CI Add explicit cross-language rules in the implementer agent prompt: 1. Existing test files are append-only (no reformat/reorder/remove). 2. Do not modify non-test source files; surface untestable seams as follow-ups for the testability-migration agent. 3. Prefer new test files over edits to existing ones when equivalent. 4. Build-system manifests may be edited only for project/dependency registration. Rule 6 in the Rules section now points at the Step 4 detail so the implementer reads it on every phase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ca21fd61c1 |
run-tests: prompt-shaped multi-TFM keywords for plugin activation (#688)
* run-tests: use prompt-shaped multi-TFM keywords for plugin activation The 'Run tests in a multi-TFM project targeting a specific framework' eval scenario kept failing plugin-mode activation after #667. The skill description had 'multi-TFM projects (--framework)' as a brief parenthetical, but the scenario prompt uses different language: 'This test project targets both net8.0 and net9.0. I only want to run the tests on net9.0 using dotnet test.' Per the InvestigatingResults.md guidance for 'skill not activated' failures, expand that clause to include keywords from the prompt — 'running tests against a single target framework when a project targets multiple TFMs' plus a concrete '<TargetFrameworks>net8.0;net9.0</TargetFrameworks>' example and the '--framework <TFM>' invocation. Per-skill description is 947 chars, well under the 1024 spec cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace simple multi-TFM scenario with a harder MTP+SDK 9 variant that requires skill knowledge The previous 'Run tests in a multi-TFM project targeting a specific framework' scenario tested a trivial prompt (single --framework flag) that the model already answers perfectly from training (baseline 4.7/5). This matches pattern #8 (baseline already good, no headroom) in InvestigatingResults.md, and explains why two prior attempts to fix plugin-mode activation via description tweaks (PR #667, #688) failed: the router rationally skips loading the skill for a trivial question. This change replaces the scenario with a meaningfully harder one that combines several signals the model often confuses: - Multi-TFM project (TargetFrameworks plural) - MTP runner (TestingPlatformDotnetTestSupport) - .NET SDK 9 (-- separator required for MTP extension args) - TRX report (MTP extension flag) The correct answer requires knowing that --framework is a dotnet test/MSBuild flag that always goes BEFORE --, while --report-trx is an MTP extension flag that goes AFTER -- on SDK 9: dotnet test --framework net9.0 -- --report-trx The ordering assertion in eval.yaml enforces this on a single line, so swapping --framework after the separator no longer satisfies the scenario. Also fix a documentation bug in SKILL.md surfaced by this work: the 'Common MTP flags > Built-in flags' table previously listed --framework and --no-build under 'On SDK 8/9, pass after --', which is wrong. Those are dotnet test/MSBuild flags consumed by dotnet test itself; they always go before --. The table now lists only flags genuinely consumed by MTP, and an Important note clarifies the rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5810d492c9 |
test-anti-patterns / test-smell-detection: narrow-specialist re-pitch (Variant B) (#671)
Both skills suffer plugin-mode activation failures: in plugin runs the model bypasses the skill tool entirely for ~5/6 of test-anti-patterns audit prompts and ~2/3 of test-smell-detection prompts, answering directly from training. Variant B repositions the two skills as umbrella + narrow specialist: * test-anti-patterns becomes THE umbrella audit skill for all pragmatic test-quality reviews (anti-patterns and smell-style prompts alike). Its description explicitly absorbs smell-audit triggers. * test-smell-detection is repositioned as a narrow niche that only fires when the user explicitly asks for the testsmells.org / academic 19-smell catalog with citable smell names from the research literature. Eval prompts for test-smell-detection are rewritten to match the narrow niche (explicit testsmells.org / 19-smell catalog requests) so we test specialist activation rather than the umbrella scenarios. Variant B of a 3-PR experiment to fix activation. Companion PRs: * Variant A = activation-only rewrite, both skills keep overlap; * Variant C = full merge into test-anti-patterns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
362b3f25ec |
Fix plugin-mode skill activation for 8 dotnet-test scenarios (#667)
* migrate-vstest-to-mtp: fix plugin activation for OutputType=Exe scenario The `Set OutputType=Exe only for test projects in Directory.Build.props` scenario was failing plugin-mode skill activation (skillActivationPlugin. activated=false, detectedSkills=[]) in the latest scheduled eval. The model answered from its own knowledge and proposed the wrong solution (`Condition="'$(IsTestProject)' == 'true'"` in Directory.Build.props, which the skill explicitly warns against). Two fixes: 1. SKILL.md description: add an explicit `conditioning OutputType=Exe to test projects when centralizing MTP properties in Directory.Build.props` USE FOR clause so the description surfaces this specific MTP migration pitfall to the skill router. Per-skill description stays at <= 1024 chars (trimmed `configuration` -> `config` and `MSTest version upgrades (use migrate-mstest-* skills)` -> `MSTest version upgrades` to keep within the spec limit). 2. eval.vally.yaml: add the missing scenario (the eval.yaml scenario was added in PR #556 but vally was never synced — vally had 10 stimuli vs eval.yaml's 11). Brings vally and skill-validator coverage to parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-anti-patterns: strengthen description for severity-ranked audits The `Detect mixed severity anti-patterns in repository service tests` scenario was failing plugin-mode skill activation (skillActivationPlugin.activated=false, detectedSkills=[]) in the latest scheduled eval, even though the prompt explicitly says `Audit the test class ... for .NET test anti-patterns ... Give me a severity-ranked list (Critical / Warning / Info)`. The previous description only mentioned `USE FOR: audit test quality, review test code, find test anti-patterns, ...`, so the model could not distinguish this skill from sibling test-quality skills based on the `severity-ranked` and `Critical/Warning/Info` framing in the prompt. Surface the catalog shape (severity-ranked output, concrete code-level fixes) and add explicit `severity-ranked test audit`, `self-comparing assertions` and `broad exception types` triggers — these are the exact phrases used by failing scenarios. Description stays at 822 chars (<= 1024 limit). Aggregate dotnet-test description stays at 15,099 chars (<= 20,000 limit). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * run-tests: restructure description for higher-signal plugin activation Six run-tests scenarios were failing plugin-mode skill activation in the latest scheduled eval (skillActivationPlugin.activated=false, detectedSkills=[]) even though isolated activation worked for every scenario: - Run tests with trx reporting on MTP project (SDK 9) - Run tests in a multi-TFM project targeting a specific framework - Filter MSTest tests by category on VSTest - Filter NUnit tests by class name on VSTest - Filter TUnit tests by class using treenode-filter - Negative test: do not use MTP syntax for a VSTest project The previous description piled ~30 quoted trigger phrases (`run tests`, `run my tests`, `run these tests`, `execute tests`, `dotnet test`, `test filter`, `filter by category`, `filter by class`, `combine filters`, `run only specific tests`, `integration tests`, `unit tests`, `tests not running`, `hang timeout`, `blame-hang`, `blame-crash`, `crash dump`, `TRX report`, `TRX`, `test report`, `generate TRX`, `TUnit`, `treenode-filter`, `target framework`, `multi-TFM`, ...). In plugin mode this signal-poor keyword wall made the skill look generic enough that the router preferred to answer directly from model knowledge. Replace the keyword pile with five higher-signal scope clauses grouped by user-visible concern: 1. running, filtering, or troubleshooting `dotnet test` 2. VSTest vs Microsoft.Testing.Platform syntax (incl. `--` separator rules on SDK 8/9 vs 10+) 3. framework-specific filter syntax for MSTest / xUnit / NUnit / TUnit (--filter, --filter-class, --filter-trait, --filter-query, --treenode-filter) 4. TRX/reporting (--report-trx vs --logger trx), blame/hang/crash diagnostics, multi-TFM (--framework) 5. avoiding MTP/VSTest argument mixups These are the same concerns the previous description tried to cover, but framed as decisions the skill helps make (and that distinguish it from sibling test skills) rather than as a list of bare keywords. Per-skill description shrinks from 1,000 to 819 chars (still <= 1,024 limit). Aggregate dotnet-test description stays well under the 20,000-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-anti-patterns: drop specific severity labels from description PR review (copilot-pull-request-reviewer on #667) caught that `Produces a severity-ranked (Critical / Warning / Info) catalog ...` in the new description contradicts the SKILL.md body, which actually teaches a 4-level Critical / High / Medium / Low taxonomy. The `Critical / Warning / Info` wording came from one eval scenario's prompt, but other prompts ask for different bucket schemes, and the skill itself does not enforce a fixed three-level mapping. Drop the specific labels from the description and keep just `severity-ranked` as the activation signal. The body remains unchanged (it teaches Critical / High / Medium / Low and the agent maps to the prompt's requested buckets at output time). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
b24599aa36 |
test-smell-detection: trim to detailed tier and add prompt-shaped triggers (#653)
Multi-model skill-validator eval (claude-haiku-4.5, claude-opus-4.6, gpt-5.3-codex, gpt-5.4-mini) showed two issues with this skill: 1. Quality regression on stronger models (+0.15 haiku, -0.14 opus, -0.17 codex, -0.29 gpt-5.4-mini): the skill added tokens and tool calls without lifting rubric quality on larger models. 2. Plugin-mode activation loss: the skill activated reliably in isolated mode but was almost never picked in plugin mode where it competed with siblings (code-testing-agent, etc.). Changes: - Trim BPE tokens 2,487 -> 1,472 (still detailed tier, well within the 800-2,500 sweet spot per SkillsBench guidance). Collapsed long smell-by-smell prose, `Why Test Smells Matter`, and pitfalls into a compact taxonomy table plus a short decision procedure / output checklist. Removed bulky examples. Kept taxonomy, calibration, negative guidance, and references intact. - Add prompt-shaped USE FOR triggers drawn from the eval prompts (`review test smells`, `review these tests and tell me if there are` `any problematic patterns`, `check my tests for test design` `problems or anti-patterns`, `give us an objective assessment of` `our integration tests`, plus explicit smell names). - Strengthen DO NOT USE FOR clause: `write new tests / scaffold tests` `from scratch / generate a complete MSTest test suite` now defers cleanly to code-testing-generator (covers the negative eval scenario). Validation: skill-validator check --plugin clean for this skill; build + 545/545 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
2fcfc099b9 | Improve direct scenario (#675) | ||
|
Amaury Levé
|
dde9f99cf0 |
dotnet-test: drop tools field from agents that declared it (#665)
The four agents (code-testing-generator, test-migration, testability-migration, test-quality-auditor) were the only dotnet-test agents declaring a 'tools:' list. Per Jan's suggestion on #660, drop the field so they inherit the runtime's default tool surface (matching the convention used by every other agent in the repo). Follow-up to #660. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2a193300ca |
Replace 'terminal' with 'bash' and 'powershell' in agent tools (#660)
The skill-validator only recognizes 'bash' and 'powershell' as built-in shell tools, not 'terminal'. Update the four affected agents in the dotnet-test plugin to use both for cross-platform support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
b0c2336d49 |
Tighten crap-score boundary to fix coverage-analysis plateau activation (#652)
Dashboard data showed coverage-analysis failing to activate in 8/10 recent scheduled plugin-mode runs for the 'Coverage plateau diagnosis' scenario, while activating reliably in isolated mode. PR #647 added positive triggers to coverage-analysis but did not address sibling attention competition. The crap-score description matched the plateau prompt almost as well as coverage-analysis (it advertised 'evaluate whether complex methods have sufficient test coverage' + 'Requires code coverage data (Cobertura XML)') without redirecting project-wide / stuck-coverage diagnosis to coverage-analysis. With 22 sibling skills competing for attention this overlap is enough to suppress activation altogether. Tighten the crap-score frontmatter to: - Scope positive triggers to a named method, class, or single source file (the actual eval surface — see tests/dotnet-test/crap-score/ eval.yaml, all 3 scenarios target OrderService.cs). - Add explicit DO NOT USE FOR redirects covering project-wide coverage analysis, coverage plateau / stuck coverage, what's blocking coverage, and where to add tests across a project — all of which point at coverage-analysis. skill-validator check passes (22 skills, 11 agents, 1 plugin). Aggregate dotnet-test description size: 14,932 chars (limit 15,000). markdownlint passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
3d59e44c7e |
Fix coverage-analysis activation for plateau diagnosis prompts (#647)
* Fix coverage-analysis activation for plateau diagnosis prompts coverage-analysis SKILL.md: - Trim verbose implementation details (provider detection, ReportGenerator) that consumed description budget without aiding skill activation - Add explicit USE FOR keywords: coverage stuck, coverage plateau, can't increase coverage, what's blocking coverage code-testing-agent SKILL.md: - Add 'diagnosing coverage plateaus or CRAP score computation (use coverage-analysis)' to DO NOT USE FOR boundary to prevent test-generation skill from intercepting diagnostic prompts * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Strengthen code-testing-agent activation; harden coverage-analysis isolated mode Address eval regressions reported on PR #647 (run 25813728646): 1. code-testing-agent: `Generate tests for ContosoUniversity ASP.NET Core MVC app` was NOT ACTIVATED in plugin mode (detectedSkills=[], skillEventCount=0, invokedAgents=[]). The model bypassed the skill system entirely. - SKILL.md description: restructure to use the proven `Use when user says ...` pattern with quoted trigger phrases (matching the run-tests skill that consistently activates), make the link to the code-testing-generator sub-agent explicit, and tighten DO NOT USE FOR clauses. - eval prompt (eval.yaml + eval.vally.yaml): make the request pipeline-shaped (`project-wide, multi-file test generation task`, `scaffold a new test project`) so the model recognizes it as multi-step work that benefits from the orchestrated pipeline. Explicitly request coverlet.collector + a Cobertura XML run so rubric criterion 1 (`high line coverage as reported by the Cobertura XML in TestResults/`) becomes achievable without overfitting. 2. code-testing-tester agent + code-testing-extensions/dotnet.md: open a scoped exception to the `skip coverage tools` rule. Default behavior stays the same, but when the user/harness explicitly asks for a Cobertura/XML coverage artifact, the agent may add coverlet.collector to the generated test csproj so the harness's coverage command produces output. The agent still does not run the coverage command itself. 3. coverage-analysis SKILL.md: add a `User-visible output is mandatory` guard at the top of the Workflow section. The latest eval showed isolated mode producing literally `(no output)` in 2 of 3 scenarios — the agent ran Compute-CrapScores.ps1 / Extract-MethodCoverage.ps1 / ReportGenerator in parallel, then the session ended without ever surfacing findings. The guard tells the agent to always return a partial summary instead of ending silent, and to deprioritize ReportGenerator HTML when budget is tight. (Plugin-mode quality is already strong: 4.3 / 4.3 / 5.0 — no regression risk there.) Aggregate dotnet-test plugin description size: 14,925 chars (limit 15,000). skill-validator check passes (22 skills, 11 agents, 1 plugin); markdownlint passes for all 4 modified files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore MSTest modernization exclusion in code-testing-agent description * Fix isolated-mode coverage-analysis: emit summary before optional ReportGenerator The previous workflow encouraged the agent to run `dotnet tool install` for ReportGenerator in parallel with the CRAP scoring scripts (Phase 2 "Steps 3 and 4 in parallel" + Phase 3 "Steps 5 and 6 in parallel"). In isolated mode that pattern reliably crashed the session with "Failed to persist session events: timeout while waiting for mutex to become available" right after the scripts returned valid data, so the agent never produced the user-facing summary. Restructure the workflow into 5 phases: - Phase 1 (Setup) - unchanged - Phase 2 (Test execution) - skip when Cobertura XML already exists - Phase 3 (Analysis) - run only the two PowerShell scripts, no RG - Phase 4 (User-facing summary) - MANDATORY, must be the next assistant response after Phase 3, before any RG work; also save coverage-analysis.md as a secondary follow-up - Phase 5 (ReportGenerator HTML/CSV) - strictly optional, post-summary, skipped by default for existing-Cobertura and plateau-diagnosis paths Also update references/output-format.md so the Reports section marks RG artifacts as "Not generated (optional - request HTML reports to enable)" when Phase 5 has not run, and update references/guidelines.md so the "show and open the markdown report" rule explicitly defers to the user-facing assistant response. Targets the isolated-mode regressions in PR #647 eval: - Project-wide coverage with existing Cobertura: 1.0/5 -> expected 3+ - Coverage plateau diagnosis: 1.0/5 -> expected 3+ - Run coverage from scratch: 2.3/5 -> expected steady or up Verified: skill-validator check --plugin ./plugins/dotnet-test passes; markdownlint-cli2 clean on all 3 modified files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #647 review comments 1. Prerequisites: distinguish the from-scratch path (needs NuGet for the coverage-provider package install + optional internet for ReportGenerator) from the existing-Cobertura path (needs neither). 2. Add Step 2c "Discover or accept existing Cobertura XML" so the existing-data path actually has $coberturaFiles populated before Phase 3, instead of relying on Phase 2's discovery (which it skips). Also clarify Step 2's destructive Remove-Item only manages the skill-owned coverage-analysis/ subdirectory. 3. references/output-format.md: replace the unconditional "Reports saved to: <coverageDir>/reports/" line with one that always points at <coverageDir>/ (markdown summary + raw Cobertura) and only mentions reports/ if Phase 5 ran. 4. Have Compute-CrapScores.ps1 emit OVERALL_LINE_COVERAGE and OVERALL_BRANCH_COVERAGE from the Cobertura root attributes, and update Phase 4 to read those values directly from the script's output. The Phase 4 mandatory-summary rule no longer requires a separate XML parse before composing the response. Verified: skill-validator check --plugin ./plugins/dotnet-test passes; markdownlint-cli2 clean on all modified files; Compute-CrapScores.ps1 smoke-tested on a synthetic Cobertura XML (emits OVERALL_LINE_COVERAGE:75, OVERALL_BRANCH_COVERAGE:50 alongside HOTSPOTS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestion from @Evangelink * Address unresolved coverage-analysis and eval review comments * Refine follow-up review feedback from validation * Tighten coverage aggregation fallback notes and counters * Clarify pre-response save instruction wording --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |