* fix: validate native Codex plugin support
Separate Codex-compatible plugin components from GitHub Copilot custom agents, validate Codex manifest fields and MCP shapes, and document the native agent installation boundary.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: strengthen Codex plugin verification
Validate Codex MCP per-tool settings and add a pinned native smoke lane that installs the marketplace, discovers skills and MCP, and calls binlog_overview.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: avoid restarting Codex MCP smoke client
Use the real tool call as the app-server initialization so the Linux smoke test does not start and tear down a redundant status client first.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: harden Codex validation and smoke cleanup
Constrain smoke artifacts to a dedicated child directory, validate Codex manifest field shapes and required skill paths, and document the separate Agents SDK handoff model.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: align Codex smoke and hooks parsing
Use the clean Codex installation's default provider for the direct MCP call and reject heterogeneous hooks arrays like the runtime parser.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: verify every Codex plugin skill
Compare native skill discovery with the full repository inventory, match Codex lexical path rejection, and allow null optional MCP tool settings.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: address Codex compatibility review
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Surface activation-only evaluation failures
Add post-activation continuation telemetry to Vally results, PR summaries, and dashboard evidence so models that load a skill and stop can be diagnosed separately from routing and content failures. Relates to #1160.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix activation continuation diagnostics
Use ordered trajectory events to distinguish pre-activation tools from true post-activation continuation, prioritize runtime diagnosis over preference-loss advice, and retain plugin-arm failures on target dormancy cases.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Validate the same startup guard against SDK 1.0.11 used by the PR merge checks and SDK 1.0.13 on this branch. Keep unknown SDK versions fail-closed. Mock the runtime path explicitly in transport-only tests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Guard the pinned SDK's concurrent lazy startup and wait for filesystem-provider readiness before create/resume. Use the trusted launcher for evaluation and comparison without changing model routing, trial concurrency, or result gates.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace Opus 4.8 judging with Haiku 4.5 while retaining Opus executor profiles and cross-family Terra judging for Claude and MAI. Disable duplicate secondary judging, update routing regression tests and guidance, and correct the stale Sonnet schedule comment.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use gpt-5.6-sol for four health and triage workflows. Use claude-sonnet-5 alongside unchanged gpt-5.6-luna in default and full evaluation profiles, retain model overrides and judge routing, and cover actual profile selection in workflow tests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace the concrete Sonnet 4.6 fallback in the four scoped workflows while preserving repository model overrides and PAT environment boundaries. Regenerate locks with gh-aw v0.86.2 without changing action or container pins.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Stamp versions from the latest main first-parent authority: dotnet-test 0.2.22 and dotnet-test-migration 0.1.9.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d007947-f964-4809-a31c-2d9f457a2ba3
Preserve current main content and advance dotnet-test-migration to the next valid release checkpoint.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d007947-f964-4809-a31c-2d9f457a2ba3
* Delete msbuild-server skill
msbuild-server is now on by default since .NET 11 Preview 7. We don't want to recommend turning it on, automatically in other scenarios as critical fixes could be missing.
* Remove MSBuild Server recommendations
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Pilot repository overlays for test skills
Add revisioned, repo-specific overlay bindings for selected dotnet-test skills and eval coverage for valid, scoped, and stale bindings.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Allow attributed overlay references
Add the JeremyKuhne source repositories to the reviewed external reference allowlist.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address overlay pilot review feedback
Strengthen the MSTest eval against duplicate/default policy output and make the suite fixture parameter drive its test filter.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Clarify invalid overlay fallback
Make invalid overlays fall back to portable guidance and accept natural revision mismatch explanations in the stale-binding eval.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix overlay precedence wording
Restore the separator between repository and portable precedence rules.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Use a literal overlay project path check
Prevent the scaffold eval grader from interpreting dots in the expected project path as regex wildcards.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Enforce command-only overlay evals
Reject shell, edit, and create tools in both run-tests overlay prompts that prohibit execution.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Keep test agent state out of commits
Move broad test-generation pipeline state to host scratch storage, worktree-specific Git metadata, or OS temp, and enforce the exclusion in evals.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Clarify absolute test agent state path
Use Git's explicit absolute path formatting in both test-generation entry points.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Prune Git metadata from test agent eval guards
Avoid scanning nested repositories and align the remaining TESTAGENT_DIR placeholder with the documented format.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Clarify test agent command handoff
Require callers to provide exact commands, excerpts, or absolute TESTAGENT_DIR document paths to command-running sub-agents.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Reject all repository-local testagent entries
Match .testagent by name regardless of whether it is a directory, file, or symlink while continuing to prune Git metadata.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Verify external test agent artifacts
Restore broad-run artifact checks at the Git metadata path and pass the researched lint command and state directory to the linter agent.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Make testagent eval guards constant time
Check only the forbidden workspace-root path, including broken symlinks, instead of recursively traversing dependency trees.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Broaden comprehensive test generation
Treat explicit requirements as the floor for broad suites and add mutation-relevant equivalence-partition and invariant coverage without test-count padding.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Fix external artifact grader quoting
Run state checks directly in the harness shell so TESTAGENT_DIR expands after assignment, with an isolated command probe covering valid and forbidden states.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Run broad skill evals in Git worktrees
Initialize the seven broad evaluation roots as Git repositories so TESTAGENT_DIR resolves deterministically and external artifacts remain verifiable.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Clarify non-stageable test agent state
Describe the real invariant across the pipeline: state may live under .git metadata but must never be version-controlled workspace content or appear in git status.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Standardize intermediate test state contract
Use one TESTAGENT_DIR placeholder, clearer intermediate-state terminology, and detect stageable research, plan, or status files regardless of directory name.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Use one Git root in workspace integrity eval
Baseline the fixture from the evaluation root so stageable intermediate-state files remain visible to the directory-independent guard.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Prune Vitest dependencies from state scan
Exclude node_modules through per-eval Git metadata so stageable state detection remains fast without modifying fixture content.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* Strengthen focused intermediate-state guards
Separate shell execution, reject Git-metadata files on focused runs, include ignored state files, and prune node_modules with a pathspec exclusion.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
---------
Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget
The per-plugin aggregate description cap had been raised 15,000 -> 20,000
-> 22,000 under the belief that 15K was 'a local repo policy, NOT a
documented Copilot constraint'. That belief was wrong: the GitHub Copilot
CLI renders the model-facing <available_skills> menu under a hard 15,000-
char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in
CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with
their full <description> only until the budget is exhausted; every skill
past the cut-off collapses to a bare name with no description and can no
longer be reliably model-activated. Raising the validator cap merely
masked this silent menu truncation — e.g. dotnet-test's run-tests and
test-* skills stopped activating in plugin eval runs because they fell
into the name-only overflow.
Changes:
- SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the
comment rewritten to document the real Copilot CLI budget (and correct
the prior 'not a documented constraint' claim).
- CheckCommand aggregate now excludes skills marked
'disable-model-invocation: true' — the CLI drops those from the menu, so
they do not consume the budget. This makes the cap satisfiable by hiding
reference / agent-orchestrated primitives rather than only by trimming.
- InvestigatingResults.md: document plugin-arm-only non-activation caused
by skill-menu budget overflow, and how to fix it.
Note: dotnet-test currently exceeds 15K and must be slimmed below it
(via disable-model-invocation on reference/primitive skills plus
description trims) before this cap can go green repo-wide.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: use source-generated regex for disable-model-invocation check
Address review: replace Regex.IsMatch(pattern-string) with a
[GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup),
matching FrontmatterParser's style. Runs once per skill during checks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test-gap-analysis: cover validation safeguards
Add focused evaluation scenarios for proportional scope, complete outcome inventory, mutation restoration, demonstrated recommendations, and canonical verifier usage.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: harden mutation verifier eval
Anchor verifier paths to the script directory and require the eval response to report the supplied canonical verifier result.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: distinguish verifier failures
Require Python 3, preflight the test runner, build mutants explicitly, and treat only the MTP test-failure exit code as a killed mutation.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: tighten eval evidence
Challenge proportional scope with an unrelated fixture and require affirmative canonical-verifier success wording.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: name MTP failure exit code
Document the Microsoft.Testing.Platform test-failure convention used by the mutation verifier.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: require affirmative verifier result
Constrain the output grader to a direct successful result for verify_mutations.sh so negative mutation summaries cannot pass.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* Improve MSTest migration skill activation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316
* test-gap-analysis: cover proportional scope validation
Name the validation outcome directly and prove the focused review excludes an unrelated AccessControl fixture.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: classify killed mutants by test evidence
Capture MTP output and require an xUnit execution summary with failed tests, avoiding runner-specific exit-code assumptions while preserving infrastructure diagnostics.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
* test-gap-analysis: preserve mutant build diagnostics
Capture MSBuild output and print it when mutant compilation fails so verifier infrastructure failures remain actionable in CI.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846
Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316
Load performance references only after scan depth and topic signals select them, while preserving reduced-coverage reporting for every selected file.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bf1f2bfa-446d-4d44-9de1-38e4c65ef225
Keep configured Copilot model overrides while restoring a concrete Claude fallback when no override is set. Regenerate all affected locks with gh-aw v0.86.2 so model discovery failures cannot pass literal auto into AI-credit pricing.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Amaury Levé