5b4d76e8b4
Merge pull request #1173 from dotnet/abhitejjohn-issue-pr-triage-37b
...
Harden DevOps health investigation and reporting
2026-09-16 10:45:24 -07:00
d77d2479b7
Keep health workflow failures dashboard-bound
...
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
2026-09-16 04:20:09 -07:00
29ebd31633
Harden health workflow state handling
...
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
2026-09-15 16:52:31 -07:00
1f73f15936
Fix evaluation workflow expression limit
...
Move target discovery into a tested PowerShell script so GitHub Actions can parse and dispatch the workflow.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 8c45529b-2515-483d-9e51-e6c0b7cb6852
2026-09-15 09:30:14 -07:00
6deb481989
Merge main into infrastructure health fixes
...
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
2026-09-15 08:03:19 -07:00
36222bf32d
Add first-class custom-agent evaluation coverage ( #1165 )
...
* feat(evaluation): add custom agent coverage
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): address agent review feedback
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): reject linked fixture sources
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): preserve agent result invariants
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): fail closed on agent errors
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): preserve completion regressions
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): preserve nested command quotes
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): harden native agent evidence
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): honor declared agent layout
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): resolve declared agent sources
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): secure agent path discovery
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): reject linked dependencies
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): centralize path safety checks
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): diagnose ambiguous dependencies
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): reject linked allowed roots
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): preserve skill agent isolation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): normalize dashboard evidence
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): preserve agent gate semantics
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): fail closed on incomplete evidence
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): preserve completion evidence
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): reject overflowing durations
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): stage verified plugin skills
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): block shell network access
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): reject linked MCP config files
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): trust manual dispatch path safety
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): keep agent plugin activation diagnostic
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): count failed tool completions
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* fix(evaluation): synchronize agent event capture
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
2026-09-15 16:47:35 +02:00
234c399a95
Close workflow validation gaps
...
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
2026-09-15 02:11:01 -07:00
a226ead6ff
fix(ci): serialize evaluation SDK startup before sessions
...
Guard the pinned SDK's concurrent lazy startup and wait for filesystem-provider readiness before create/resume. Use the trusted launcher for evaluation and comparison without changing model routing, trial concurrency, or result gates.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
2026-09-11 15:00:58 -07:00
94ca0ca748
Improve evaluation freshness and scheduled reliability ( #1081 )
...
* Improve evaluation freshness and reliability
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* Address dashboard freshness review
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* Preserve evidence commit fallback
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 5283cdb3-86e3-41d8-95a9-ebf6b7e0ccbc
* Keep watchdog regression assertion current
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 5283cdb3-86e3-41d8-95a9-ebf6b7e0ccbc
* Add headroom for MSTest migration evaluation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 5283cdb3-86e3-41d8-95a9-ebf6b7e0ccbc
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 5283cdb3-86e3-41d8-95a9-ebf6b7e0ccbc
2026-08-27 13:34:24 +00:00
b9aca2d25b
Bump actions/checkout in the github-actions-dependencies group
...
Bumps the github-actions-dependencies group with 1 update: [actions/checkout](https://github.com/actions/checkout ).
Updates `actions/checkout` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v7...3d3c42e5aac5ba805825da76410c181273ba90b1 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-version: 7.0.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: github-actions-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-08-26 05:44:57 +00:00
4dc894a88f
Merge remote-tracking branch 'origin/main' into abhitejjohn-eval-correctness-infrastructure
2026-08-18 18:08:08 -07:00
9faa838da9
Complete eval correctness hardening
...
Add Vally 0.13 all-error compatibility, a practical net-win floor, objective completion criteria, and CI fault-injection coverage.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 1c6716a2-172b-461a-adb5-127b1ba7e96b
2026-08-17 17:46:33 -07:00
dfd38dba3a
Bump actions/checkout in the github-actions-dependencies group
...
Bumps the github-actions-dependencies group with 1 update: [actions/checkout](https://github.com/actions/checkout ).
Updates `actions/checkout` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v7...3d3c42e5aac5ba805825da76410c181273ba90b1 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-version: 7.0.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: github-actions-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-08-17 19:48:29 +00:00
ca093a06ef
Automate Vally dependency updates ( #1010 )
...
* Automate Vally dependency updates
Move the evaluation CLI pins into a trusted npm manifest and configure weekly Dependabot updates for Vally.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: d4443741-2ca2-47b3-a2db-0f91e937dbf9
* Smoke test evaluation tool updates
Run a secretless install and CLI smoke test whenever the evaluation tool manifest changes.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: d4443741-2ca2-47b3-a2db-0f91e937dbf9
* Generate evaluation tools lockfile in CI
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 8779074c-8123-4bfc-98ff-6a9c2da72cf0
* Make evaluation tool installs reproducible
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 8779074c-8123-4bfc-98ff-6a9c2da72cf0
---------
Copilot-Session: d4443741-2ca2-47b3-a2db-0f91e937dbf9
Copilot-Session: 8779074c-8123-4bfc-98ff-6a9c2da72cf0
2026-08-17 21:46:24 +02:00
6e023a32d7
Bump the github-actions-dependencies group across 1 directory with 2 updates
...
Bumps the github-actions-dependencies group with 2 updates in the / directory: [actions/checkout](https://github.com/actions/checkout ) and [actions/setup-python](https://github.com/actions/setup-python ).
Updates `actions/checkout` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v7...3d3c42e5aac5ba805825da76410c181273ba90b1 )
Updates `actions/setup-python` from 5.6.0 to 7.0.0
- [Release notes](https://github.com/actions/setup-python/releases )
- [Commits](https://github.com/actions/setup-python/compare/a26af69be951a213d495a4c3e4e4022e16d87065...5fda3b95a4ea91299a34e894583c3862153e4b97 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-version: 7.0.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: github-actions-dependencies
- dependency-name: actions/setup-python
dependency-version: 7.0.0
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: github-actions-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-08-13 16:29:55 +00:00
8b6b51ba59
Preflight Copilot PAT pool for evaluation rate limits ( #992 )
...
* Retry eval token selection on rate limits
Probe shuffled Copilot PAT candidates before running Vally, fail over only on explicit rate-limit errors, and keep the remaining pool out of untrusted subprocesses.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* Harden eval token failover boundaries
Skip token work for empty legs, probe the resolved agent and judge models, and execute validator and adapter code from isolated trusted workflow sources.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* Clarify failover test setup errors
Report missing PyYAML and workflow-step shape failures with actionable diagnostics.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* Align validator cache restore path
Use the same archive path as skill-check while deleting any PR-provided file before cache restore.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
* Harden trusted evaluation setup
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d
* Fix Vally runtime tool resolution
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d
* Harden fork evaluation trust boundary
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d
* Use trusted NuGet config for validator cache
Keep the trusted cache producer consistent with the fallback validator build and invalidate caches when the evaluation NuGet configuration changes.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d
* Restore validator package sources
Do not apply the evaluation-only dotnet-public mapping to the trusted validator build because it excludes the validator's analyzer package.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d
* Unify Copilot rate-limit failover checks
Use one superset classifier for preflight and post-run failures, and skip candidates whose availability probes time out.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com >
Copilot-Session: 96292bfe-b3fc-4356-be5b-f51ff8c4b42d
2026-08-05 18:58:51 +00:00
Abhitej John