* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget
The per-plugin aggregate description cap had been raised 15,000 -> 20,000
-> 22,000 under the belief that 15K was 'a local repo policy, NOT a
documented Copilot constraint'. That belief was wrong: the GitHub Copilot
CLI renders the model-facing <available_skills> menu under a hard 15,000-
char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in
CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with
their full <description> only until the budget is exhausted; every skill
past the cut-off collapses to a bare name with no description and can no
longer be reliably model-activated. Raising the validator cap merely
masked this silent menu truncation — e.g. dotnet-test's run-tests and
test-* skills stopped activating in plugin eval runs because they fell
into the name-only overflow.
Changes:
- SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the
comment rewritten to document the real Copilot CLI budget (and correct
the prior 'not a documented constraint' claim).
- CheckCommand aggregate now excludes skills marked
'disable-model-invocation: true' — the CLI drops those from the menu, so
they do not consume the budget. This makes the cap satisfiable by hiding
reference / agent-orchestrated primitives rather than only by trimming.
- InvestigatingResults.md: document plugin-arm-only non-activation caused
by skill-menu budget overflow, and how to fix it.
Note: dotnet-test currently exceeds 15K and must be slimmed below it
(via disable-model-invocation on reference/primitive skills plus
description trims) before this cap can go green repo-wide.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: use source-generated regex for disable-model-invocation check
Address review: replace Regex.IsMatch(pattern-string) with a
[GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup),
matching FrontmatterParser's style. Runs once per skill during checks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test-gap-analysis: cover validation safeguards
Add focused evaluation scenarios for proportional scope, complete outcome inventory, mutation restoration, demonstrated recommendations, and canonical verifier usage.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: harden mutation verifier eval
Anchor verifier paths to the script directory and require the eval response to report the supplied canonical verifier result.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: distinguish verifier failures
Require Python 3, preflight the test runner, build mutants explicitly, and treat only the MTP test-failure exit code as a killed mutation.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: tighten eval evidence
Challenge proportional scope with an unrelated fixture and require affirmative canonical-verifier success wording.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: name MTP failure exit code
Document the Microsoft.Testing.Platform test-failure convention used by the mutation verifier.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: require affirmative verifier result
Constrain the output grader to a direct successful result for verify_mutations.sh so negative mutation summaries cannot pass.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* Improve MSTest migration skill activation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316
* test-gap-analysis: cover proportional scope validation
Name the validation outcome directly and prove the focused review excludes an unrelated AccessControl fixture.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: classify killed mutants by test evidence
Capture MTP output and require an xUnit execution summary with failed tests, avoiding runner-specific exit-code assumptions while preserving infrastructure diagnostics.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: preserve mutant build diagnostics
Capture MSBuild output and print it when mutant compilation fails so verifier infrastructure failures remain actionable in CI.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316
Load performance references only after scan depth and topic signals select them, while preserving reduced-coverage reporting for every selected file.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bf1f2bfa-446d-4d44-9de1-38e4c65ef225
* Improve test migration skill outcomes
Address cross-model routing and execution failures found in retained evaluation evidence, and align the xUnit companion-package rubric with feed-compatible versions.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Cover MTP method filter routing
Include the method-filter token in the VSTest-to-MTP skill description so method-specific migration requests activate consistently.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Clarify MSTest SDK v4 discovery guidance
Scope the missing Test SDK reference to MSTest.Sdk v4 and remove a fixed package-version example in favor of feed-resolved and CPM-aware guidance.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Clarify xUnit MTP runner preservation
State that the migration preserves the MTP runner for projects currently using the YTest shim, rather than preserving the shim package itself.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Avoid xUnit package major assumptions
Resolve companion compatibility from package constraints and configured feeds instead of examples that conflate the xUnit product generation with NuGet major versions.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* List all MSTest SDK discovery paths
Align the CI response contract with the three documented choices for transitional VSTest discovery and MTP-native execution.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix xUnit v3 evaluation evidence
Replace a contradictory async-void output grader with a file-state check, strengthen deterministic BeforeAfter signature checks, and require explicit final evidence for both migration paths.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Align MSTest SDK discovery guidance
Correct the behavioral symptom table to describe the v4-only removal and all three documented migration paths consistently.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Make BeforeAfter eval name agnostic
Validate the required xUnit v3 override and base-call shapes without hardcoding local parameter names.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Strengthen Luna migration outcomes
Add complete advisory output contracts, correct focused-fix execution decisions, require concrete MTP package and filter guidance, and make xUnit package and source evidence explicit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Relax BeforeAfter source formatting
Keep deterministic IXunitTest source validation while relying on compilation and test execution for qualified and multiline signature correctness.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix Codex discovery of the dotnet-msbuild binlog MCP server
.codex-plugin/plugin.json declared "mcpServers": "./.mcp.json", but the file
was packaged at .codex-plugin/.mcp.json. Codex resolves manifest resource
paths against the plugin root, so it looked for
plugins/dotnet-msbuild/.mcp.json and never found the binlog server.
Declare the server inline in .codex-plugin/plugin.json, matching the root
plugin.json and .claude-plugin/plugin.json, and drop the unreachable file.
Add a packaging regression check to skill-validator: every companion manifest
must declare the same MCP servers as the root plugin.json, and a manifest
referencing an external .mcp.json must resolve it from the plugin root the way
hosts do. skill-check.yml already runs `skill-validator check` over plugins/*
on every PR, so this now blocks in CI. A test also loads the shipped
dotnet-msbuild manifests and asserts binlog is present in each.
Fixes#1069
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Report non-object JSON roots as validation errors
JsonElement.TryGetProperty throws InvalidOperationException when the root
value is not an object, so a manifest or referenced .mcp.json that is valid
JSON but not an object (null, array, string) crashed skill-validator instead
of producing a validation error.
Check the root kind while reading and surface it as a structured error.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve test smell skill quality
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Use conventional empty class bodies
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Improve test smell calibration
Align workspace discovery and false-positive decisions with the losing eval transcripts, correct contradictory fixtures, and make graders outcome-focused.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Make eval regexes multiline-safe
Allow outcome evidence to match across line breaks in generated review output.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Make notification fixtures observable
Record notification identifiers so post-wait assertions can fail, while preserving fixed sleeps as the intentional smell under evaluation.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Strengthen test smell stop conditions
Require workspace discovery, preserve formal skip and file classifications, prevent clean-suite false positives, and reduce lexical grader coupling.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Use conventional exception class body
Keep the fixture compatible with compilers that do not accept semicolon-only class declarations.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Remove brittle eval gates
Rely on outcome rubrics instead of narrow lexical matches and keep the Sensitive Equality fixture culture-stable.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Add JUnit eval exit check
Fail fast on empty or failed trial output while dropping a redundant severity-word matcher.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Make remaining eval regexes multiline-safe
Allow concise verdict and async-fix patterns to match wrapped model output across line breaks.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
* Preserve non-catalog validity findings
Keep formal smell classification while separately reporting proven test-validity defects that do not belong to the taxonomy.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b
Resolve bundled files from each skill root, use one bounded directory listing when direct reads fail, and make reduced reference coverage visible.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9c391191-97d7-4a4b-bb79-1e279c37d095
kdinev