* fix: validate native Codex plugin support
Separate Codex-compatible plugin components from GitHub Copilot custom agents, validate Codex manifest fields and MCP shapes, and document the native agent installation boundary.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: strengthen Codex plugin verification
Validate Codex MCP per-tool settings and add a pinned native smoke lane that installs the marketplace, discovers skills and MCP, and calls binlog_overview.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: avoid restarting Codex MCP smoke client
Use the real tool call as the app-server initialization so the Linux smoke test does not start and tear down a redundant status client first.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: harden Codex validation and smoke cleanup
Constrain smoke artifacts to a dedicated child directory, validate Codex manifest field shapes and required skill paths, and document the separate Agents SDK handoff model.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: align Codex smoke and hooks parsing
Use the clean Codex installation's default provider for the direct MCP call and reject heterogeneous hooks arrays like the runtime parser.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: verify every Codex plugin skill
Compare native skill discovery with the full repository inventory, match Codex lexical path rejection, and allow null optional MCP tool settings.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: address Codex compatibility review
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Stamp versions from the latest main first-parent authority: dotnet-test 0.2.22 and dotnet-test-migration 0.1.9.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d007947-f964-4809-a31c-2d9f457a2ba3
Preserve current main content and advance dotnet-test-migration to the next valid release checkpoint.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d007947-f964-4809-a31c-2d9f457a2ba3
* Delete msbuild-server skill
msbuild-server is now on by default since .NET 11 Preview 7. We don't want to recommend turning it on, automatically in other scenarios as critical fixes could be missing.
* Remove MSBuild Server recommendations
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Pilot repository overlays for test skills
Add revisioned, repo-specific overlay bindings for selected dotnet-test skills and eval coverage for valid, scoped, and stale bindings.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Allow attributed overlay references
Add the JeremyKuhne source repositories to the reviewed external reference allowlist.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address overlay pilot review feedback
Strengthen the MSTest eval against duplicate/default policy output and make the suite fixture parameter drive its test filter.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Clarify invalid overlay fallback
Make invalid overlays fall back to portable guidance and accept natural revision mismatch explanations in the stale-binding eval.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix overlay precedence wording
Restore the separator between repository and portable precedence rules.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Use a literal overlay project path check
Prevent the scaffold eval grader from interpreting dots in the expected project path as regex wildcards.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Enforce command-only overlay evals
Reject shell, edit, and create tools in both run-tests overlay prompts that prohibit execution.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Keep test agent state out of commits
Move broad test-generation pipeline state to host scratch storage, worktree-specific Git metadata, or OS temp, and enforce the exclusion in evals.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Clarify absolute test agent state path
Use Git's explicit absolute path formatting in both test-generation entry points.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Prune Git metadata from test agent eval guards
Avoid scanning nested repositories and align the remaining TESTAGENT_DIR placeholder with the documented format.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Clarify test agent command handoff
Require callers to provide exact commands, excerpts, or absolute TESTAGENT_DIR document paths to command-running sub-agents.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Reject all repository-local testagent entries
Match .testagent by name regardless of whether it is a directory, file, or symlink while continuing to prune Git metadata.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Verify external test agent artifacts
Restore broad-run artifact checks at the Git metadata path and pass the researched lint command and state directory to the linter agent.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Make testagent eval guards constant time
Check only the forbidden workspace-root path, including broken symlinks, instead of recursively traversing dependency trees.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Broaden comprehensive test generation
Treat explicit requirements as the floor for broad suites and add mutation-relevant equivalence-partition and invariant coverage without test-count padding.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Fix external artifact grader quoting
Run state checks directly in the harness shell so TESTAGENT_DIR expands after assignment, with an isolated command probe covering valid and forbidden states.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Run broad skill evals in Git worktrees
Initialize the seven broad evaluation roots as Git repositories so TESTAGENT_DIR resolves deterministically and external artifacts remain verifiable.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Clarify non-stageable test agent state
Describe the real invariant across the pipeline: state may live under .git metadata but must never be version-controlled workspace content or appear in git status.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Standardize intermediate test state contract
Use one TESTAGENT_DIR placeholder, clearer intermediate-state terminology, and detect stageable research, plan, or status files regardless of directory name.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Use one Git root in workspace integrity eval
Baseline the fixture from the evaluation root so stageable intermediate-state files remain visible to the directory-independent guard.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Prune Vitest dependencies from state scan
Exclude node_modules through per-eval Git metadata so stageable state detection remains fast without modifying fixture content.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Strengthen focused intermediate-state guards
Separate shell execution, reject Git-metadata files on focused runs, include ignored state files, and prune node_modules with a pathspec exclusion.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
---------
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget
The per-plugin aggregate description cap had been raised 15,000 -> 20,000
-> 22,000 under the belief that 15K was 'a local repo policy, NOT a
documented Copilot constraint'. That belief was wrong: the GitHub Copilot
CLI renders the model-facing <available_skills> menu under a hard 15,000-
char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in
CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with
their full <description> only until the budget is exhausted; every skill
past the cut-off collapses to a bare name with no description and can no
longer be reliably model-activated. Raising the validator cap merely
masked this silent menu truncation — e.g. dotnet-test's run-tests and
test-* skills stopped activating in plugin eval runs because they fell
into the name-only overflow.
Changes:
- SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the
comment rewritten to document the real Copilot CLI budget (and correct
the prior 'not a documented constraint' claim).
- CheckCommand aggregate now excludes skills marked
'disable-model-invocation: true' — the CLI drops those from the menu, so
they do not consume the budget. This makes the cap satisfiable by hiding
reference / agent-orchestrated primitives rather than only by trimming.
- InvestigatingResults.md: document plugin-arm-only non-activation caused
by skill-menu budget overflow, and how to fix it.
Note: dotnet-test currently exceeds 15K and must be slimmed below it
(via disable-model-invocation on reference/primitive skills plus
description trims) before this cap can go green repo-wide.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: use source-generated regex for disable-model-invocation check
Address review: replace Regex.IsMatch(pattern-string) with a
[GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup),
matching FrontmatterParser's style. Runs once per skill during checks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test-gap-analysis: cover validation safeguards
Add focused evaluation scenarios for proportional scope, complete outcome inventory, mutation restoration, demonstrated recommendations, and canonical verifier usage.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: harden mutation verifier eval
Anchor verifier paths to the script directory and require the eval response to report the supplied canonical verifier result.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: distinguish verifier failures
Require Python 3, preflight the test runner, build mutants explicitly, and treat only the MTP test-failure exit code as a killed mutation.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: tighten eval evidence
Challenge proportional scope with an unrelated fixture and require affirmative canonical-verifier success wording.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: name MTP failure exit code
Document the Microsoft.Testing.Platform test-failure convention used by the mutation verifier.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: require affirmative verifier result
Constrain the output grader to a direct successful result for verify_mutations.sh so negative mutation summaries cannot pass.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* Improve MSTest migration skill activation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316
* test-gap-analysis: cover proportional scope validation
Name the validation outcome directly and prove the focused review excludes an unrelated AccessControl fixture.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: classify killed mutants by test evidence
Capture MTP output and require an xUnit execution summary with failed tests, avoiding runner-specific exit-code assumptions while preserving infrastructure diagnostics.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: preserve mutant build diagnostics
Capture MSBuild output and print it when mutant compilation fails so verifier infrastructure failures remain actionable in CI.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316
Load performance references only after scan depth and topic signals select them, while preserving reduced-coverage reporting for every selected file.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bf1f2bfa-446d-4d44-9de1-38e4c65ef225
* Improve test migration skill outcomes
Address cross-model routing and execution failures found in retained evaluation evidence, and align the xUnit companion-package rubric with feed-compatible versions.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Cover MTP method filter routing
Include the method-filter token in the VSTest-to-MTP skill description so method-specific migration requests activate consistently.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Clarify MSTest SDK v4 discovery guidance
Scope the missing Test SDK reference to MSTest.Sdk v4 and remove a fixed package-version example in favor of feed-resolved and CPM-aware guidance.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Clarify xUnit MTP runner preservation
State that the migration preserves the MTP runner for projects currently using the YTest shim, rather than preserving the shim package itself.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Avoid xUnit package major assumptions
Resolve companion compatibility from package constraints and configured feeds instead of examples that conflate the xUnit product generation with NuGet major versions.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* List all MSTest SDK discovery paths
Align the CI response contract with the three documented choices for transitional VSTest discovery and MTP-native execution.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix xUnit v3 evaluation evidence
Replace a contradictory async-void output grader with a file-state check, strengthen deterministic BeforeAfter signature checks, and require explicit final evidence for both migration paths.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Align MSTest SDK discovery guidance
Correct the behavioral symptom table to describe the v4-only removal and all three documented migration paths consistently.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Make BeforeAfter eval name agnostic
Validate the required xUnit v3 override and base-call shapes without hardcoding local parameter names.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Strengthen Luna migration outcomes
Add complete advisory output contracts, correct focused-fix execution decisions, require concrete MTP package and filter guidance, and make xUnit package and source evidence explicit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Relax BeforeAfter source formatting
Keep deterministic IXunitTest source validation while relying on compilation and test execution for qualified and multiline signature correctness.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix Codex discovery of the dotnet-msbuild binlog MCP server
.codex-plugin/plugin.json declared "mcpServers": "./.mcp.json", but the file
was packaged at .codex-plugin/.mcp.json. Codex resolves manifest resource
paths against the plugin root, so it looked for
plugins/dotnet-msbuild/.mcp.json and never found the binlog server.
Declare the server inline in .codex-plugin/plugin.json, matching the root
plugin.json and .claude-plugin/plugin.json, and drop the unreachable file.
Add a packaging regression check to skill-validator: every companion manifest
must declare the same MCP servers as the root plugin.json, and a manifest
referencing an external .mcp.json must resolve it from the plugin root the way
hosts do. skill-check.yml already runs `skill-validator check` over plugins/*
on every PR, so this now blocks in CI. A test also loads the shipped
dotnet-msbuild manifests and asserts binlog is present in each.
Fixes#1069
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Report non-object JSON roots as validation errors
JsonElement.TryGetProperty throws InvalidOperationException when the root
value is not an object, so a manifest or referenced .mcp.json that is valid
JSON but not an object (null, array, string) crashed skill-validator instead
of producing a validation error.
Check the root kind while reading and surface it as a structured error.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Amaury Levé