Amaury Levé 36222bf32d Add first-class custom-agent evaluation coverage (#1165) ...

* feat(evaluation): add custom agent coverage

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): address agent review feedback

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject linked fixture sources

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve agent result invariants

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): fail closed on agent errors

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve completion regressions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve nested command quotes

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): harden native agent evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): honor declared agent layout

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): resolve declared agent sources

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): secure agent path discovery

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject linked dependencies

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): centralize path safety checks

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): diagnose ambiguous dependencies

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject linked allowed roots

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve skill agent isolation

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): normalize dashboard evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve agent gate semantics

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): fail closed on incomplete evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve completion evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject overflowing durations

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): stage verified plugin skills

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): block shell network access

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject linked MCP config files

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): trust manual dispatch path safety

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): keep agent plugin activation diagnostic

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): count failed tool completions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): synchronize agent event capture

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

2026-09-15 16:47:35 +02:00

30 KiB

Raw Blame History

View Raw