* Add TFM-aware buildTransitive->build forwarding guidance to msbuild skills
buildTransitive/*.props should forward through the corresponding build/*.props (ownership chain buildTransitive -> build -> shared) rather than importing buildMultiTargeting/ directly. When build/ is packed per-TFM (build/<tfm>/), the forwarder must include the TFM segment and derive it from the file own folder, not $(TargetFramework) (NuGet nearest-match can serve a different asset folder), otherwise transitive consumers hit MSB4019.
Updates extension-points (new Forwarding chain section), msbuild-antipatterns AP-13, and the msbuild-code-review agent. Lesson learned from microsoft/testfx#9431.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify ambiguous props/targets glob in forwarder review check
Address review feedback on dotnet/skills#836: spell out ".props/.targets forwarders" instead of the ambiguous `buildTransitive/*.props|targets`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten forwarding-chain guidance to reduce skill token overhead
Condense the new extension-points Forwarding chain section (+26 -> +11 lines) and the AP-13 note by dropping the redundant non-TFM example and self-evident derivation explanation, keeping the chain rule, MSB4019 cause, and the TFM derivation expression. Lower token footprint addresses the skill-validator weighted-score token penalty without losing substance.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review nits: AP range and forward-slash path
Update the additional anti-patterns range to AP-16 through AP-22 (the reference doc now includes AP-22), and use a forward-slash build/MyPackage.props in the forwarding-chain prose to match the build/<tfm>/ convention used in the section.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Condense AP-13 forwarding note to a cross-reference to reduce token overhead
The full TFM-forwarding guidance lives in extension-points; AP-13 only needs a concise pointer. Removes duplication and trims the msbuild-antipatterns skill footprint flagged by the skill-validator token penalty.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Address remaining review nits: targets forwarders, single-source wording, quoted MSBuildThisFileDirectory
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Adds eval.vally.yaml for all 9 dotnet-blazor skills, converted from the legacy skill-validator eval.yaml specs, and registers a dotnet-blazor suite in .vally.yaml so the shadow Vally pipeline discovers them.
Conversion preserves prompts, assertions, and rubric verbatim. Legacy setup blocks map to Vally environment config: 'dotnet new' scaffolding commands become environment.commands. Verified end-to-end by running the author-component eval (baseline + skilled + adapt), which passed the shadow threshold.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add MSTest deflection to assertion-quality and test-anti-patterns skills
The writing-mstest-tests skill failed to activate in the plugin arm for
six MSTest-specific scenarios (fix swapped Assert.AreEqual, modernize
legacy patterns, type/string assertions, DynamicData with ValueTuples)
because sibling skills captured the routing. assertion-quality and
test-anti-patterns matched these prompts but lacked explicit deflection
to writing-mstest-tests. Add DO NOT USE entries pointing MSTest test
writing/fixing/modernizing to writing-mstest-tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: restore general assertion-fix deflection, align test-anti-patterns
- assertion-quality: restore the general 'fixing or rewriting assertions'
deflection (dropped in the prior edit) alongside the MSTest deflection, and
fix punctuation to use comma-separated DO NOT USE items.
- test-anti-patterns: align the 'writing new tests' deflection to also point
MSTest test-writing to writing-mstest-tests, matching the body guidance.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Require environment input to pat_pool shared workflow.
- Updates the PAT pool shared workflow to latest version
- Updates the validate-pat-pool workflow to match other repos
- Updates all agentic workflows to use the updated PAT pool import
with the standard secret names used across other repos
- The new version of the import does not require the 'needs' workaround
but the issue-triage workflow's 'roles: all' configuration requires
a different workaround to ensure the pre_activation job exists for
the pat_pool job to be able to depend on it (necessary for ordering).
* Update markdown bullet style
* Apply Copilot PR feedback
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Bump versions to match gh-aw
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget
The per-plugin aggregate description cap had been raised 15,000 -> 20,000
-> 22,000 under the belief that 15K was 'a local repo policy, NOT a
documented Copilot constraint'. That belief was wrong: the GitHub Copilot
CLI renders the model-facing <available_skills> menu under a hard 15,000-
char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in
CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with
their full <description> only until the budget is exhausted; every skill
past the cut-off collapses to a bare name with no description and can no
longer be reliably model-activated. Raising the validator cap merely
masked this silent menu truncation — e.g. dotnet-test's run-tests and
test-* skills stopped activating in plugin eval runs because they fell
into the name-only overflow.
Changes:
- SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the
comment rewritten to document the real Copilot CLI budget (and correct
the prior 'not a documented constraint' claim).
- CheckCommand aggregate now excludes skills marked
'disable-model-invocation: true' — the CLI drops those from the menu, so
they do not consume the budget. This makes the cap satisfiable by hiding
reference / agent-orchestrated primitives rather than only by trimming.
- InvestigatingResults.md: document plugin-arm-only non-activation caused
by skill-menu budget overflow, and how to fix it.
Note: dotnet-test currently exceeds 15K and must be slimmed below it
(via disable-model-invocation on reference/primitive skills plus
description trims) before this cap can go green repo-wide.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: use source-generated regex for disable-model-invocation check
Address review: replace Regex.IsMatch(pattern-string) with a
[GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup),
matching FrontmatterParser's style. Runs once per skill during checks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: parse disable-model-invocation via YAML to avoid block-scalar false positives
The regex-based check matched any line in the frontmatter, so a block-scalar description that merely mentioned 'disable-model-invocation: true' on its own line was wrongly treated as disabling model invocation. Parse the frontmatter with the existing YAML deserializer (which correctly handles block scalars) by adding a DisableModelInvocation field to SkillFrontmatter, and drop the regex entirely.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add eval coverage for dotnet-test/writing-mstest-tests
Enrich the ServiceRegistry collection/null/reference scenario with deterministic file_contains assertions for the modern MSTest assertion APIs Assert.IsNull, Assert.AreSame, Assert.Contains, Assert.DoesNotContain, Assert.IsEmpty, and Assert.IsNotEmpty.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reject legacy CollectionAssert/StringAssert helpers in collection scenario
Address PR review: file_contains is a case-sensitive substring check, so the Assert.Contains and Assert.DoesNotContain value checks would also pass for CollectionAssert.* and StringAssert.* helpers. Add file_not_contains guards to keep the scenario a deterministic check for the modern Assert.* API.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add eval coverage for dotnet-test/dotnet-test-frameworks
Add a deterministic output_matches assertion for Assert.AreEqual in the MSTest try/catch refactor scenario, covering the previously-uncovered MSTest assertion CodePattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten Assert.AreEqual assertion to require message comparison
Address review feedback: the output_matches pattern now requires the exception message comparison (Assert.AreEqual(..., ex.Message)) rather than matching Assert.AreEqual anywhere. The CodePattern remains covered via the rubric evidence.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enrich the NUnit misleading-names scenario with rubric items and an output_matches assertion covering the [Category] code pattern, reading the test body before classifying, and classifying each test method. Add a Go standard-testing scenario (report-only) with a fixture to cover leaving source files unmodified for report-only frameworks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add eval coverage for dotnet-test/migrate-static-to-wrapper
Extend eval.yaml with rubric items and two scenarios covering: required using directives, build-succeeds-after-migration, test-double updates, no-behavioral-changes, not breaking static classes, and incremental migration.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review on migrate-static-to-wrapper eval
Restore expect_activation: false on the decline scenario and add missing using BillingApp; to the test fixture so it starts buildable.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Get-SignificantTerms returned $terms.Keys; when a teaching point contains the word 'keys', PowerShell hashtable member access resolves to that entry's value instead of the key collection, collapsing the keyword set and making points like test-tagging Step 3 uncoverable. Use get_Keys()/get_Values() to bypass key-name shadowing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds outcome-focused rubric items so two previously-uncovered Validation teaching points become covered:
- Metrics are computed correctly (counts add up)
- If the suite has good diversity, the report acknowledges this
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The plugin's rendered skill-menu was 15,428 chars, over the 15,000 limit. Compressed the verbose per-framework parentheticals in the longest polyglot descriptions and dropped a few redundant clauses, preserving all routing keywords. Aggregate name+description length now ~14.5K.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR #658 bumped .github/aw/actions-lock.json to v0.79.8 but its automation could not recompile the .lock.yml files, because the GITHUB_TOKEN it runs under lacks the 'workflow' scope. PR #815 then shipped prompt fixes but likewise did not re-pin the runtime. As a result main is in a split state: actions-lock.json declares v0.79.8 while all 8 compiled workflows still pin and execute the v0.77.5 runtime (setup@v0.77.5).
Run 'gh aw compile' (CLI pinned to v0.79.8) so the deployed workflows actually run the version main already declares, eliminating the declared-vs-deployed drift. This only re-pins the runtime and regenerates compiled output (the 8 .lock.yml files, the generated agentics-maintenance.yml, and the compile-managed dependabot entry); it makes no prompt/behaviour changes, so #815's groom and health-check fixes are preserved verbatim (groom body_hash unchanged at a705abb0).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rebase setup-local-sdk skill
Replay PR #508 onto current main and address the remaining setup-local-sdk review feedback. The install examples now use fail-fast temporary downloads, the team scripts preserve existing global.json content, and eval assertions cover the critical paths/version/workload behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address setup-local-sdk review comments
Add the missing skill license metadata and fix the PowerShell .gitignore regex examples so they correctly detect an existing .dotnet/ entry.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Stabilize setup-local-sdk evals
Reduce overfit-prone assertions, make incompatible host handling explicit, configure base SDK resolution before slow workload installs, and tighten exact-version/team-script validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address setup-local-sdk follow-up review
Include the shared dotnet skill reviewer team in CODEOWNERS, make .gitignore appends newline-safe, and tighten the incompatible-host eval to reward stopping before local install commands.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten exact local SDK eval
Measure project-level SDK resolution and exact roll-forward behavior using global.json artifact assertions so the scenario distinguishes PATH-based local installs from SDK paths setup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address install script review feedback
Use the documented lowercase dotnet-install quality value and call out both bash and PowerShell official install script URLs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Increase team script eval timeout
Give the team install script scenario the same realistic long-running timeout as other scenarios that install preview SDK bits.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Windows local SDK check
Document the PowerShell equivalent for detecting an existing local SDK and give exact SDK installs enough time to complete in eval.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Stabilize team script eval
Focus the team script scenario on script/config generation, create reproducible setup guidance before long downloads, and avoid brittle transcript assertions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make mktemp usage portable
Use BSD/macOS-compatible mktemp templates in setup-local-sdk bash snippets and generated team install script.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(devops-health): stop recurring groom node_id and health-check tool/token failures
Groom (fixes 780/792/797): MCP issue_read(get_comments) sometimes omits node_id, which hide-comment requires. The agent was reporting missing_data and failing the run. Make node_id best-effort: skip hiding comments without a node_id and continue; the weekly cleanup workflow removes them by age.
Health check (fixes 810/791/788): the agent attempted sed and git commands (incl. manual git plumbing on the cache-memory dir) that the bash allowlist denied, looping on the denials and exhausting the 25M effective-token budget. Add sed and git to the allowlist, and add guidelines to not git-manage cache-memory and not retry denied commands.
Recompiled lock files with pinned gh-aw v0.77.5.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(groom): clarify skipped-comment count goes in noop summary only
Address PR review: Step 6 only emits a summary via noop when no other safe-output calls were made, so 'note in the Step 6 summary' was misleading. Track the skipped count internally and include it in the noop message only when that summary is emitted.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The rendered skill-menu for the dotnet-msbuild plugin was 16,679 characters, exceeding the 15,000 limit. Trim two categories of content from the 18 skill descriptions that do not help agents decide when to invoke a skill:
- Remove INVOKES: clauses (which MCP tools / fallback shell commands a skill runs internally) — implementation mechanics, not routing signal.
- Remove the boilerplate 'Only activate in MSBuild/.NET build context.' repeated verbatim in all 18 skills — redundant, since every description is already MSBuild-specific and each DO NOT USE FOR still excludes non-MSBuild build systems.
USE FOR triggers and DO NOT USE FOR cross-references are preserved. Aggregate skill description size drops from 14,551 to 12,536 chars (rendered menu 16,679 -> 14,664).
Co-authored-by: Yuliia Kovalova <ykovalova@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the .NET test framework/platform migration skills (migrate-mstest-v1v2-to-v3, migrate-mstest-v3-to-v4, migrate-vstest-to-mtp, migrate-xunit-to-mstest, migrate-xunit-to-xunit-v3) and the test-migration orchestrator agent out of dotnet-test into a new dedicated dotnet-test-migration plugin, along with their evals. Update all marketplace manifests, READMEs, CODEOWNERS, and .vally.yaml accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric
Workspace-integrity scenario was NOT ACTIVATED in plugin mode: the prompt
("Generate unit tests for its core module") was terse and small-scoped, so
the runtime did not route to the code-testing-agent skill. Reframe it with
high-level test-generation language ("comprehensive pytest test suite",
"scaffold", "thorough unit tests") that matches the skill description,
while preserving the guardrail anchors: it still points at the on-disk module
without naming it and never implies restoring the gutted tree.
ContosoUniversity rubric item #5 (find-untested-sources) was conditional on
that skill being loaded — only true in plugin mode. In isolated runs the
agent cannot satisfy it, so the judge penalized it asymmetrically, injecting
isolated-vs-plugin variance. Make the conditional deterministic: explicitly
N/A when the skill is not loaded, without lowering the bar when it is.
Applied to both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* code-testing-agent: make find-untested-sources rubric decidable from session timeline
Address review feedback: the 'treat as satisfied (N/A) when the skill is
not loaded' clause is not verifiable from the judge's inputs (which show
which tools were called, not which were available). Rewrite the criterion
to be decidable from the session timeline by requiring a source-to-test
pairing map recorded in .testagent/research.md that either cites
find-untested-sources output or documents the equivalent manual approach.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: fix evals (query-filter regex, sibling skills, command observability)
Investigated run-tests eval failures by running the validator locally.
- SKILL.md Step 3: document xUnit v3 --filter-query so the agent stops
answering that complex xUnit v3 filters 'cannot be combined'.
- eval.yaml: fix a broken assertion regex. The query-filter pattern is a
single-quoted YAML scalar using '\\s'/'\\[', which (unlike a double-quoted
scalar) is NOT unescaped, so the regex searched for a literal '\s' and
could never match. Corrected to single backslashes.
- eval.yaml: add additional_required_skills (filter-syntax / platform-detection)
to the filter and detection scenarios, so the isolated arm loads the sibling
reference skills that run-tests explicitly defers to.
- eval.yaml: ask the agent to show the exact command in execute-style prompts.
output_matches only sees the final assistant message; 'run my tests' prompts
make the agent execute and summarize ('tests passed'), so the recommended
command never appears. The assertions still catch wrong commands.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: rewrite description for reliable plugin-arm activation
The run-tests skill activated reliably in the isolated eval arm but
unreliably in the plugin arm (all dotnet-test sibling descriptions
loaded), so every scenario's pluginImprovementScore went negative and
dragged min(isolated, plugin) below zero.
Lead the description with natural-language intent triggers that mirror
how the eval prompts phrase requests (run all tests, run a subset via
filters, produce TRX reports, collect crash/hang dumps, run a single
TFM) instead of opening with platform-detection mechanism, and add
explicit DO NOT USE redirects to code-testing-agent / mtp-hot-reload.
Stays under the 1024-char description cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dotnet-test: fix run-tests plugin-arm activation via skill-menu budget
Root cause (verified against the Copilot CLI SDK skill renderer): the
model-facing skill menu has a 15000-char budget. Skills are listed
alphabetically and emitted with full descriptions only until the budget
is exhausted; the rest collapse to bare names with no description and
effectively cannot be model-activated. With 27 dotnet-test skills,
run-tests (alphabetical position ~20) fell into the name-only overflow,
so it never activated in the plugin eval arm even though it activated
reliably in isolation. This is a real user-facing discoverability bug,
not just an eval artifact.
Fix: hide reference/primitive skills that are never meant to be
model-invoked from the menu via 'disable-model-invocation: true', which
the SDK filters out of the budget entirely:
- filter-syntax, platform-detection, dotnet-test-frameworks,
code-testing-extensions, test-analysis-extensions — already
user-invocable:false reference data ('DO NOT USE directly').
- find-untested-sources, find-untested-sources-polyglot — researcher
primitives invoked by-name from the code-testing-researcher agent
(which has a manual fallback); no standalone evals.
These remain invocable by explicit name (agents/users), only auto-
suggestion is suppressed.
This frees enough budget that run-tests (plus migrate-xunit-to-xunit-v3
and mtp-hot-reload) now receive full descriptions; no previously-visible
skill regresses. Also trimmed the run-tests description so its menu block
fits with margin while keeping all activation triggers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix code-testing-agent activation for the Flask pytest scenario
The 'Generate pytest tests for the Flask tasks API' scenario failed to activate code-testing-agent in BOTH isolated and plugin mode: its prompt enumerated, file by file, exactly what to mock/inject/test (TaskService with repo mocked + clock injected, queries.apply_query over fixed lists, both repositories, the blueprint via test_client), acting as an answer key that let the base agent generate tests directly with edit tools instead of routing to the skill's research-plan-implement pipeline. Rewrite the prompt to a realistic, high-level ask (mirroring the ContosoUniversity scenario that does activate): describe the app at a layer level, keep the 'no tests yet', project-wide multi-file framing and the 80% coverage floor, and drop the per-module test checklist. Assertions, rubric and timeout are unchanged. Verified locally that the skill now activates in both isolated and plugin mode.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use python3 in Flask prompt to match the grader
Review feedback: the prompt told the agent to run \python -m ...\ but the grader (and the vally command) invoke \python3\. On Linux runners that may lack a \python\ shim the agent could hit command-not-found. Align the prompt to \python3\ in both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The build/check pipeline was failing repo-wide with NU1903 (warnings as
errors): Microsoft.Data.Sqlite 10.0.7 transitively pulls
SQLitePCLRaw.lib.e_sqlite3 2.1.11, whose bundled SQLite is older than
3.50.2 and carries a known high-severity vulnerability
(GHSA-2m69-gcr7-jv3q / CVE-2025-6965). A NuGet advisory update started
flagging it, breaking every build.
Pin the native lib to 3.50.3 (patched SQLite), following the existing
transitive-pin pattern in this project. The 3.50.x native package is
ABI-compatible with the 2.1.x managed provider; verified: restore is
clean (no NU1903/NU1605), build succeeds with 0 warnings, and all 586
tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix test-gap-analysis skill activation in plugin eval runs
In plugin-mode eval runs (all dotnet-test skills loaded), test-gap-analysis lost activation: 'Acknowledge well-tested code' was stolen by assertion-quality and 'logic/null-check gaps' loaded no skill at all. Front-load the concrete 'would my tests catch a bug if the code changed' trigger phrasing the prompts use, and add a DO NOT USE -> test-gap-analysis redirect in assertion-quality for mutation-style reasoning. Both descriptions stay within the 1024-char cap; aggregate unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore Jest/TS activation signal for assertion-quality polyglot scenario
The earlier trim of assertion-quality's description dropped the Jest/Vitest and toBeTruthy() signals, breaking plugin-mode activation for the 'Polyglot: Jest/TypeScript' scenario (agent answered directly, loading no skill). Restore those polyglot signals, and make the scenario prompt realistic — the prior prompt enumerated the full check catalog and Jest matcher list inline, acting as an answer key that let the agent self-serve without loading the skill. Verified locally: the scenario now activates assertion-quality in plugin mode across runs; all other assertion-quality scenarios still activate; description stays within the 1024-char cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Cover MSTESTxxxx analyzer diagnostics in writing-mstest-tests skill
Add a 'Fix MSTest analyzer diagnostics' workflow step mapping the common MSTESTxxxx rules to their idiomatic fixes, plus MSTestAnalysisMode guidance, instead of creating one skill per rule.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix MD012 markdown lint (trailing blank line)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback: reword analyzer availability and fix table grammar
- Don't tie MSTest.Analyzers availability to TestFramework 3.7; note metapackage/SDK/explicit reference.
- Fix grammatically broken fix text for the MSTEST0002-0014 layout row.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify the MSTESTxxxx table is non-exhaustive; defer to full reference
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Direct strategy must still run the Step 7 pre-completion gate
The Direct strategy correctly skips the research/plan/implement sub-agents
for small single-file tasks, but the wording let agents also skip the
Step 7 pre-completion gate (test-gap-analysis + assertion-quality +
scenario coverage) — treating a single-file task that enumerates specific
behaviors as 'trivially small'.
This is the dominant failure mode observed on behavior-enumerating tasks:
the agent writes one test file directly and finishes with no
assertion-strength or scenario-coverage check, producing weak assertions
(mutation survivors) and missing required edge/negative cases.
Clarify in both the generator Step 2 strategy table and the
code-testing-agent SKILL.md that Direct trades away only the sub-agents,
never the gate, and that a request naming a specific symbol or enumerating
scenarios is not 'trivially small' and must run the gate.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: align Direct gate trigger with Step 7 threshold; clarify gate in SKILL.md
- Step 2 Direct cell no longer introduces a separate 'names a specific
symbol' gate trigger that contradicted Step 7. It now defers to Step 7's
own threshold (>=5 tests, or any enumerated behaviors/scenarios).
- SKILL.md now names what/where the gate is: the generator's Step 7
(test-gap-analysis + assertion-quality).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Amaury Levé <evangelink@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add csharp-development skill and update CODEOWNERS and README.md
- Introduced a new skill for C# development guidance.
- Updated CODEOWNERS to include reviewers for the new skill.
- Modified README.md to list the new csharp-development skill.
- Added evaluation scenarios for the csharp-development skill.
* Refactor C# evaluation scenarios to enhance production code review and validation guidance
* Refactor production code review scenarios to improve clarity and focus on async error handling
* Update CODEOWNERS to reflect current ownership and streamline reviewer assignments
* Remove csharp-development skill documentation and evaluation tests
* Remove csharp-development skill from README
* Fix formatting in Constants.g.cs by adding missing semicolon
* Revert Constants.g.cs to main
* Remove csharp-development skill from CODEOWNERS
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update CODEOWNERS to include additional reviewers for dotnet-ai skills
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add prompt-scenario coverage check to code-testing-generator gate
The pre-completion gate already verifies assertion strength (pseudo-mutation
and assertion-depth checks), but two recurring failure modes still slip
through when the prompt enumerates specific behaviors:
- Testing an *adjacent* function/helper instead of the exact feature named
in the objective, leaving the requested behavior uncovered.
- Covering only a single representative case when the scenario wording
implies multiple variations or pins a condition to a specific position
or structure.
Add a third gate item that maps each enumerated scenario to a dedicated
test, requires targeting the exact named function (preferring the canonical
existing test file), and requires honoring range/positional qualifiers
literally.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: genericize example, fix gate-count consistency
- Remove benchmark-specific symbol names from the target-the-named-function
bullet to avoid overfitting; phrase it generically.
- Fix the gate intro that said 'The two skills below' now that there are
three numbered items (the third is a prompt self-review, not a skill).
- Update Step 8 and Rule 11 so re-running the gate includes the new
prompt-scenario coverage check, not just test-gap-analysis + assertion-quality.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Amaury Levé <evangelink@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix test-anti-patterns skill activation for 5 evals
Sibling skills with overlapping descriptions were stealing activation
from test-anti-patterns in plugin eval runs (coverage-analysis,
assertion-quality, test-smell-detection). Reword descriptions so
test-anti-patterns owns the umbrella 'audit my tests for anti-patterns'
severity-ranked report, and add DO NOT USE redirects in the
metric-focused siblings. Kept all descriptions within the 1024-char cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use toBeTruthy() call form in assertion-quality example
Addresses review feedback: the Jest matcher example read like a property
without parentheses. Description stays within the 1024-char cap (1023).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix eval validation error and improve activation triggers
- assertion-quality eval.yaml/eval.vally.yaml: replace hyphenated
'assertion-quality' (the target skill name) with spaced 'assertion
quality' in two scenario prompts, fixing the 'prompt mentions target
name' validation error that biased baseline runs.
- test-anti-patterns description: add 'what's wrong with my tests' /
'are these tests any good' / 'flaky tests' trigger phrasing to improve
organic activation for the flakiness, well-written and polyglot
scenarios (which intermittently failed to activate in plugin runs).
Stays within the 1024-char description cap (1007).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use Jest matcher call form in assertion-quality polyglot prompt
Addresses review feedback: write toBeDefined()/toBeTruthy()/not.toBeNull()/
toBe()/toThrow() in call form in the prompt so they read as matcher calls,
consistent with the skill description examples. Regex assertions and rubric
left untouched (they match agent output, which may use either form).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Sharpen test-anti-patterns description for flakiness/polyglot activation
The flakiness and Python-pytest scenarios failed to activate even in
isolated runs (where it's the only candidate skill), because their prompts
enumerate the methodology and the description's keywords were too generic.
Front-load the concrete trigger keywords those prompts use: Thread.Sleep,
DateTime.Now, time.sleep, order-dependent, reflection coupling, and
Python/pytest. Stays within the 1024-char cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make mixed/flakiness eval prompts realistic to fix plugin activation
The mixed-severity and flakiness scenarios consistently failed to activate
test-anti-patterns in plugin runs (detected=[] — the agent loaded no skill
at all and answered directly). Both prompts enumerated the full anti-pattern
catalog inline, acting as an answer key that made the agent self-sufficient.
Replace the embedded checklists with realistic user asks while keeping the
'for .NET test anti-patterns' trigger, file references, severity-ranked
output format, and read-only constraint. Rubric and output_matches
assertions are unchanged — they validate the produced report.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The 'Set OutputType=Exe only for test projects in Directory.Build.props' eval scenario was failing skill activation: the model answered from its own knowledge and proposed the IsTestProject condition the skill explicitly warns against.
Strengthen the migrate-vstest-to-mtp SKILL.md description so the skill router matches this scenario: add the literal trigger phrase and surface the MSBuildProjectName (correct) and IsTestProject (anti-pattern) keywords in the USE FOR clause. Trimmed lower-value text to stay within the 1024-char description limit (1011).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Five migrate-mstest-v3-to-v4 eval scenarios were answered from general model knowledge without loading the skill (NOT ACTIVATED): Migrate custom TestMethodAttribute from Execute to ExecuteAsync, Replace ExpectedExceptionAttribute with Assert.ThrowsExactly, Fix multiple v4 breaking changes, Fix TestMethodAttribute display name constructor, and Fix Assert.IsInstanceOfType out parameter removal.
Reworded each prompt in both eval.yaml and eval.vally.yaml into an explicit 'I upgraded my test project to MSTest v4' fix request embedding the exact SKILL.md trigger phrase 'Help me fix this MSTest v4 breaking change' plus the matching USE FOR entry (Execute to ExecuteAsync, ExpectedExceptionAttribute removal, CallerInfo constructor on TestMethodAttribute, Assert.IsInstanceOfType out parameter removal). Graders and rubrics are unchanged; both YAML files parse cleanly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reword the 'Handle MSTest.Sdk and MTP changes in v4' prompt in both eval.yaml and eval.vally.yaml from a passive 'What changed?' question into an explicit 'fix this MSTest v4 breaking change' request embedding exact SKILL.md trigger phrases, while still requiring the skill's knowledge to answer.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make vanilla series distinguishable on overlapping dashboard charts
Vanilla lines were hard to tell apart from the isolated/plugin lines when their values coincided. Give every vanilla series hollow diamond markers (larger than the round markers of the other series, transparent fill so the underlying line shows through) and a more prominent dash, across the quality and efficiency charts.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Reword the 'Handle dropped target framework during v3 migration' prompt in both eval.yaml and eval.vally.yaml from a passive yes/no question into an explicit migration request. The prior phrasing was answerable from general knowledge without loading the skill (NOT ACTIVATED pattern). The new prompt embeds exact SKILL.md trigger phrases and names the fixture TFMs without revealing which one is dropped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* code-testing-generator: mandatory pre-completion self-review gate
Replaces the prose `Verify tests are implementation-specific'' bullet
in Step 7 of the code-testing-generator agent with a mandatory
pre-completion self-review gate that invokes two existing plugin
skills:
1. `test-gap-analysis'' (pseudo-mutation check) against the source
files tested and the produced test files
2. `assertion-quality'' (trivial/tautological assertion check)
against the produced test files
Both skills already ship in plugins/dotnet-test; this PR only wires
them into the generator's workflow as a mandatory gate before
declaring a run complete.
The two skills' `When to Use'' sections are extended to list
`called by code-testing-generator as a pre-completion self-review
step'' as a recognised use case so the model does not refuse the
invocation. The skill descriptions are unchanged (frontmatter is
already at the 1024-char limit).
Rule 11 of the generator agent is updated to list the gate alongside
final build, final test, and coverage-gap review as mandatory for ALL
strategies including Direct.
A matching rubric item is added to the ContosoUniversity scenario in
the code-testing-agent eval (yaml and vally) so the LLM judge can
verify the gate was actually invoked on the trajectory.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The Microsoft.AITools.BinlogMcp package is now published on nuget.org as
stable 1.0.0, so the binlog MCP server no longer needs the dnceng AzDO
dotnet-tools feed or the --prerelease flag.
Drop --prerelease and --add-source <azdo> from the binlog server args in
both plugin.json and the .codex-plugin/.mcp.json mirror.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The code-testing-generator/implementer agents could treat an unusual or
scaffolded workspace (e.g. a gutted repo with an injected synthetic module)
as corruption and "repair" it with git checkout/restore/reset/clean or rm,
restoring deleted tracked files and testing the wrong code.
- Replace generator Rule 5 ("Clean git first - stash changes") with an
explicit "Treat the workspace as delivered" rule, and add a "Never mutate
version control" rule. Output must be purely additive test files.
- Add a no-revert/no-clean invariant to the implementer's edit boundaries.
- Add a 'workspace integrity' eval to the code-testing-agent suite
(eval.yaml + eval.vally.yaml). The fixture looks gutted: a metricsd project
whose real core/io modules are committed at HEAD but deleted from the
working tree, leaving only a synthetic 'synthstr' decoy. A git restore would
resurrect the deleted sentinel files; graders fail if they reappear and
require passing pytest tests for the module as delivered.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Decouple agent runs from judging in skill-validator
Allow baseline and treatment agent runs to execute in one parallel pool, with
judging deferred to a final step, removing the serial precompute-baseline ->
evaluate barrier.
evaluate --no-judge: runs the requested agent arms and persists sessions.db +
metrics, but skips all judging and requires no baseline file (exits 0 on
successful runs). Forces session persistence; mutually exclusive with
--baseline-out/--baseline-from; requires --results-dir. Only the agent model is
validated (judge model recorded as metadata); warns when zero runs persist.
rejudge --baseline-dir: cross-directory judging that pairs each treatment run
with its baseline run by the BaselineStore key (prompt SHA + target SHA,
persisted in a new sessions.db baseline_key column), runs the same
pairwise/independent judges and pass/fail gates evaluate uses, and writes
verdicts/reports. Baseline and treatment must share --model (rejected if either
side mixes models); effective judge model resolves to --judge-model, else the
treatment DB's, else the baseline DB's, rejecting a persisted mismatch. The
inline per-run judging is extracted into a shared JudgeRunGroup helper reused by
both the same-dir and cross-dir paths (no duplicated judging logic).
Adds tests for baseline_key round-trip + legacy migration, ComputeScenarioKey,
cross-dir pairing, and model/judge-model compatibility. Updates README and
InvestigatingResults docs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Hoist ComputeScenarioKey to scenario level to avoid per-run fixture hashing
Compute the baseline key once per scenario instead of once per run, so
multi-run evaluations (--runs N) don't re-hash fixture files N times.
The key is threaded into ExecuteRun/ExecuteAgentRun as a parameter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Memoize scenario-key fixture hashing for the duration of an evaluation
Route the persisted baseline_key through an evaluation-scoped BaselineStore
cache (ComputeScenarioKeyCached) instead of the static ComputeScenarioKey, so
a scenario's setup fixtures are hashed at most once across every run, arm, and
scenario of an evaluation. When a baseline store already exists (--baseline-from
/--baseline-out) its warm input-hash cache is reused; otherwise a dedicated,
side-effect-free cache (BaselineStore.ForKeyCache) is threaded through.
Addresses the PR review comment about ComputeScenarioKey re-hashing large
fixture directories. Adds a test asserting the cached variant equals the
canonical static computation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump session schema version to 3 for the baseline_key column
The baseline_key column was added without bumping schema_info.version, so
external tools keying off the version couldn't detect the new column. Introduce
a single SchemaVersion constant (= 3), stamp it after migrations run, and drop
the drift-prone version literal from the raw CREATE SQL so SetSchemaInfo is the
sole source of truth. Update the schema-info and legacy-upgrade tests to expect
version 3.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fail fast on judging-dependent options under --no-judge; fix README prefix
Under --no-judge the noise test and overfitting fix cannot run (they require
judging), so reject --noise-skills-dir and --overfitting-fix up front instead
of silently ignoring them, mirroring the existing --baseline-out/--baseline-from
guard. Add EvaluateCommandTests covering these early rejections (they short-
circuit before any model/network call).
Also prefix the decoupled-runs README example with 'skill-validator' for
consistency with the other CLI invocations in the doc.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
YuliiaKovalova