Add first-class custom-agent evaluation coverage (#1165)

* feat(evaluation): add custom agent coverage

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): address agent review feedback

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject linked fixture sources

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve agent result invariants

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): fail closed on agent errors

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve completion regressions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve nested command quotes

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): harden native agent evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): honor declared agent layout

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): resolve declared agent sources

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): secure agent path discovery

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject linked dependencies

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): centralize path safety checks

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): diagnose ambiguous dependencies

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject linked allowed roots

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve skill agent isolation

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): normalize dashboard evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve agent gate semantics

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): fail closed on incomplete evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): preserve completion evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject overflowing durations

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): stage verified plugin skills

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): block shell network access

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): reject linked MCP config files

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): trust manual dispatch path safety

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): keep agent plugin activation diagnostic

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): count failed tool completions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evaluation): synchronize agent event capture

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

.agents/skills/create-skill-test/SKILL.md

@@ -44,12 +44,12 @@ tests/<plugin>/agent.<agent-name>/eval.yaml    # agents (the agent. prefix disam
 Verify the target exists at `plugins/<plugin>/skills/<skill-name>/SKILL.md` or
 `plugins/<plugin>/agents/<agent-name>.agent.md`, and read it.
 
-**Agent evals sit outside the verdict flow.** The canonical experiment declares
-`evals: tests/*/!(agent.*)/eval.yaml`, so `agent.*` specs are excluded: no verdict is ever computed
-for them, the stimulus floor does not apply, and `./eng/run-skill-evals.sh` drops them even when you
-name one explicitly (its `--eval-filter` is intersected with that glob). The distinct-stimulus
-floor therefore applies to **skill** evals only. Author agent evals for the
-scenario coverage and the deterministic graders, and run them as described in Step 10.
+**Agent evals use the native SDK agent lane.** Vally 0.14 cannot register custom
+agents, so `agent.*` specs do not run through the skill experiment. The
+evaluation workflow discovers them separately, runs the target agent through
+`skill-validator evaluate`, and adapts that evidence into the same
+schema-versioned result and dashboard pipeline. The distinct-stimulus floor
+applies to both skill and agent evals.
 
 **Be careful with a skill that sets `disable-model-invocation: true`.** The model cannot invoke it,
 so the skill is absent from the model-facing skilled arm and any direct eval compares two identical
@@ -147,9 +147,10 @@ environment:
 **Do not set `environment.skills` in a skill eval.** The experiment declares
 `vary: /environment/skills` and supplies the value itself — `[]` for the baseline arm and
 `plugins/<plugin>/skills/<skill>` for the skilled arm — so anything the eval declares is replaced,
-in every arm. It cannot add a skill to one arm only. `environment.skills` is meaningful only in an
-`agent.*` eval, which the experiment does not vary; there it is the set of skills the agent may
-invoke. Copy the shape from an existing agent eval such as
+in every arm. It cannot add a skill to one arm only. `environment.skills` is meaningful in an
+`agent.*` eval; the native agent lane loads those entries only in the isolated
+target run, while the plugin run loads the production plugin's complete skill
+surface. Copy the shape from an existing agent eval such as
 `tests/dotnet-test/agent.test-quality-auditor/eval.yaml` rather than reproducing a remembered form —
 the specs in this repo are not consistent about how they spell those entries.
 
@@ -291,15 +292,18 @@ python eng/eval-quality/check_eval_quality.py
 ./eng/run-skill-evals.sh <plugin> <skill-name>
 ```
 
-For an **agent** eval, the third command is a no-op: `agent.*` is outside the experiment's `evals:`
-glob. Exercise one by pointing the runner at an experiment file whose glob includes it:
+For an **agent** eval, exercise the native lane directly:
 
 ```bash
-# copy dotnet-skills.experiment.yaml, widen its evals: glob to tests/*/agent.*/eval.yaml
-EXPERIMENT_FILE=my-agent.experiment.yaml ./eng/run-skill-evals.sh <plugin>
+dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate \
+  plugins/<plugin>/agents/<agent>.agent.md \
+  --tests-dir tests/<plugin> \
+  --runs 1 \
+  --verdict-warn-only
 ```
 
-Read the trajectories rather than the verdict — there is no sign-test result for an agent eval.
+CI adapts this result through `eng/vally-adapter/adapt-agent-results.mjs`,
+which applies the same distinct-stimulus sign-test policy used by skill results.
 
 `check_eval_quality.py` blocks eleven structural defect classes that can corrupt a result:
 missing or untracked fixtures, self-contradicting coverage fixtures, empty grader configs, dormancy
@@ -314,7 +318,7 @@ For the official run, submit a PR review containing `/evaluate` so it binds to t
 
 - [ ] Directory is `tests/<plugin>/<skill-name>/` or `tests/<plugin>/agent.<agent-name>/`
 - [ ] Spec uses `stimuli:` / `graders:`, and exactly one of `defaults:` or `config:`
-- [ ] For a skill eval, at least 5 preference-eligible distinct stimuli exist; dormancy contracts do not count toward this floor (agent evals are exempt)
+- [ ] At least 5 preference-eligible distinct stimuli exist; dormancy contracts do not count toward this floor
 - [ ] Each stimulus discriminates a different property and has a stable, unique name
 - [ ] Prompts never name the skill, the agent, or its vocabulary
 - [ ] Every referenced fixture exists and is tracked by `git ls-files`
@@ -343,7 +347,7 @@ For the official run, submit a PR review containing `/evaluate` so it binds to t
 | Duplicate YAML key left behind by an edit | It overwrites the next stimulus field by field — delete the stray block |
 | Duplicate stimulus names | Vally uses names as comparison identity — give every stimulus a stable, unique name |
 | Direct eval for a `disable-model-invocation: true` skill | Remove it and cover the reference through consumer outcomes |
-| Agent eval sized for the stimulus floor | `agent.*` evals get no verdict; size them for scenario coverage instead |
-| Agent eval "run" with `./eng/run-skill-evals.sh` | The glob drops it — use a widened `EXPERIMENT_FILE` |
+| Agent eval below the stimulus floor | The native agent adapter uses the same sign-test gate; add independent preference-eligible stimuli |
+| Agent eval "run" with `./eng/run-skill-evals.sh` | That helper remains skill-only; use `skill-validator evaluate` |
 | Agent eval missing `environment.skills` | Declare the skills the agent routes to, or it cannot invoke them |
 | `environment.skills` set in a **skill** eval | The experiment varies that key and replaces it in every arm; the declaration does nothing |

.github/workflows/evaluation-run.yml

@@ -1,9 +1,9 @@
-# Reusable LLM skill evaluation — the sole eval harness (implemented on Vally).
+# Reusable LLM skill and custom-agent evaluation.
 #
-# Runs each skill's tests/<plugin>/<skill>/eval.yaml through
-# `vally experiment run` (baseline / skilled / plugin variants), then converts
-# the output to per-skill results.json via eng/vally-adapter/adapt.mjs (which
-# scores skilled-vs-baseline with `vally compare`). The main evaluation.yml
+# Skills run through `vally experiment run` (baseline / isolated / plugin).
+# Vally 0.14 cannot register custom agents, so tests/<plugin>/agent.*/eval.yaml
+# runs through skill-validator's native Copilot SDK agent path and is normalized
+# by eng/vally-adapter/adapt-agent-results.mjs. The main evaluation.yml
 # workflow calls this via workflow_call after `discover`; downstream jobs
 # (comment-on-pr, report-status, publish-*) consume the vally-results-* artifacts
 # this produces. Can also be triggered manually via workflow_dispatch.
@@ -54,7 +54,7 @@ on:
         required: true
         type: string
       skill:
-        description: 'Skill to evaluate (optional, evaluates whole plugin if empty)'
+        description: 'Skill or agent.<name> to evaluate (optional, evaluates whole plugin if empty)'
         required: false
         type: string
 
@@ -82,12 +82,65 @@ jobs:
     outputs:
       entries: ${{ steps.build.outputs.entries }}
     steps:
+      - name: Checkout evaluation content
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ inputs.head_sha || '' }}
+          persist-credentials: false
+
       - id: build
         shell: pwsh
         env:
           PLUGIN: ${{ inputs.plugin }}
           SKILL: ${{ inputs.skill }}
         run: |
+          # This job runs after checking out the evaluated commit, so helpers
+          # defined by that checkout are untrusted. Keep this security boundary
+          # inline in the workflow, whose source is github.workflow_sha.
+          function Test-PathHasReparsePoint {
+            param(
+              [Parameter(Mandatory = $true)]
+              [string]$AllowedRoot,
+              [Parameter(Mandatory = $true)]
+              [string]$Path
+            )
+
+            $root = [IO.Path]::TrimEndingDirectorySeparator(
+              [IO.Path]::GetFullPath($AllowedRoot))
+            $fullPath = [IO.Path]::GetFullPath($Path)
+            try {
+              if (([IO.File]::GetAttributes($root) -band [IO.FileAttributes]::ReparsePoint) -ne 0) {
+                return $true
+              }
+            } catch {
+              return $true
+            }
+
+            $relative = [IO.Path]::GetRelativePath($root, $fullPath)
+            if ($relative -eq ".") { return $false }
+            if ([IO.Path]::IsPathRooted($relative) -or
+                $relative -eq ".." -or
+                $relative.StartsWith(".." + [IO.Path]::DirectorySeparatorChar) -or
+                $relative.StartsWith(".." + [IO.Path]::AltDirectorySeparatorChar)) {
+              return $true
+            }
+
+            $current = $root
+            foreach ($segment in $relative.Split(
+              @([IO.Path]::DirectorySeparatorChar, [IO.Path]::AltDirectorySeparatorChar),
+              [StringSplitOptions]::RemoveEmptyEntries)) {
+              $current = Join-Path $current $segment
+              try {
+                if (([IO.File]::GetAttributes($current) -band [IO.FileAttributes]::ReparsePoint) -ne 0) {
+                  return $true
+                }
+              } catch {
+                return $true
+              }
+            }
+            return $false
+          }
+
           # Validate dispatch inputs against a strict allowlist, then build the
           # entries JSON. Values arrive via env (not inlined into the script) to
           # avoid expression injection, and \A..\z anchor to the true string
@@ -104,10 +157,129 @@ jobs:
           if ($s -and ($s -notmatch $namePattern -or $s -match '\.\.' -or $s -eq '.')) {
             throw "Invalid skill '$s' (must match $namePattern, not be '.', and not contain '..')"
           }
+
+          function Get-DeclaredAgentFiles {
+            param([string]$plugin)
+
+            $pluginRoot = [IO.Path]::GetFullPath((Join-Path "plugins" $plugin))
+            $manifestPath = Join-Path $pluginRoot "plugin.json"
+            if (-not (Test-Path $manifestPath)) {
+              throw "Plugin '$plugin' has no plugin.json"
+            }
+            $manifest = Get-Content $manifestPath -Raw | ConvertFrom-Json
+            $declaredPaths = @($manifest.agents | Where-Object { -not [string]::IsNullOrWhiteSpace("$_") })
+            if ($declaredPaths.Count -eq 0) { $declaredPaths = @("./agents/") }
+
+            $agentFiles = @()
+            foreach ($declaredPath in $declaredPaths) {
+              $candidate = [IO.Path]::GetFullPath((Join-Path $pluginRoot "$declaredPath"))
+              $pluginPrefix = $pluginRoot.TrimEnd([IO.Path]::DirectorySeparatorChar, [IO.Path]::AltDirectorySeparatorChar) + [IO.Path]::DirectorySeparatorChar
+              if (-not $candidate.StartsWith($pluginPrefix, [StringComparison]::Ordinal)) {
+                throw "Agent path '$declaredPath' escapes plugin '$plugin'"
+              }
+              if (-not (Test-Path $candidate)) { continue }
+              if (Test-PathHasReparsePoint -allowedRoot $pluginRoot -path $candidate) {
+                throw "Agent path '$declaredPath' contains a symbolic link or reparse point"
+              }
+              if (Test-Path $candidate -PathType Container) {
+                $agentFiles += Get-ChildItem -Path $candidate -Filter "*.agent.md" -File -ErrorAction SilentlyContinue
+              } elseif ((Test-Path $candidate -PathType Leaf) -and
+                        $candidate.EndsWith(".agent.md", [StringComparison]::OrdinalIgnoreCase)) {
+                $agentFiles += Get-Item $candidate
+              }
+            }
+
+            $agentFiles = @($agentFiles | Sort-Object FullName -Unique)
+            foreach ($agentFile in $agentFiles) {
+              if (Test-PathHasReparsePoint -allowedRoot $pluginRoot -path $agentFile.FullName) {
+                throw "Agent file '$($agentFile.FullName)' contains a symbolic link or reparse point"
+              }
+            }
+            return $agentFiles
+          }
+
+          function New-AgentEntry {
+            param(
+              [string]$plugin,
+              [IO.FileInfo]$agentFile,
+              [switch]$Required
+            )
+
+            $agent = $agentFile.Name -replace '\.agent\.md$', ''
+            $agentPath = [IO.Path]::GetRelativePath(
+              [IO.Path]::GetFullPath("."),
+              [IO.Path]::GetFullPath($agentFile.FullName)
+            ).Replace('\', '/')
+
+            $testsRoot = Join-Path "tests" $plugin
+            $evalCandidates = @(
+              (Join-Path $testsRoot "agent.$agent" "eval.yaml"),
+              (Join-Path $testsRoot $agent "eval.yaml")
+            )
+            if (Test-Path $testsRoot) {
+              foreach ($subDir in Get-ChildItem -Path $testsRoot -Directory -ErrorAction SilentlyContinue) {
+                $evalCandidates += Join-Path $subDir.FullName "agent.$agent" "eval.yaml"
+                $evalCandidates += Join-Path $subDir.FullName $agent "eval.yaml"
+              }
+            }
+            $evalFile = @($evalCandidates | Where-Object { Test-Path $_ } | Select-Object -First 1)
+            if ($evalFile.Count -ne 1) {
+              if ($Required) {
+                throw "Agent '$agent' in plugin '$plugin' has no resolvable eval.yaml"
+              }
+              return $null
+            }
+            $evalPath = [IO.Path]::GetRelativePath(
+              [IO.Path]::GetFullPath("."),
+              [IO.Path]::GetFullPath($evalFile[0])
+            ).Replace('\', '/')
+
+            return @{
+              name = "$plugin--agent.$agent"
+              plugin = $plugin
+              target_kind = "agent"
+              skills_path = ""
+              agents_path = $agentPath
+              eval_path = $evalPath
+            }
+          }
+
           if ($s) {
-            $json = @(@{ name = "$p--$s"; plugin = $p; skills_path = "plugins/$p/skills/$s" }) | ConvertTo-Json -Compress -AsArray
+            if ($s.StartsWith("agent.")) {
+              $agent = $s.Substring("agent.".Length)
+              if (-not $agent) { throw "Invalid agent target '$s'" }
+              $agentFiles = @(Get-DeclaredAgentFiles -plugin $p | Where-Object { $_.Name -eq "$agent.agent.md" })
+              if ($agentFiles.Count -ne 1) {
+                throw "Agent '$agent' in plugin '$p' resolved to $($agentFiles.Count) declared file(s)"
+              }
+              $json = @((New-AgentEntry -plugin $p -agentFile $agentFiles[0] -Required)) |
+                ConvertTo-Json -Compress -AsArray
+            } else {
+              $json = @(@{
+                name = "$p--$s"
+                plugin = $p
+                target_kind = "skill"
+                skills_path = "plugins/$p/skills/$s"
+                agents_path = ""
+                eval_path = ""
+              }) | ConvertTo-Json -Compress -AsArray
+            }
           } else {
-            $json = @(@{ name = $p; plugin = $p; skills_path = "plugins/$p/skills" }) | ConvertTo-Json -Compress -AsArray
+            $entries = @(@{
+              name = $p
+              plugin = $p
+              target_kind = "skill"
+              skills_path = "plugins/$p/skills"
+              agents_path = ""
+              eval_path = ""
+            })
+            foreach ($agentFile in @(Get-DeclaredAgentFiles -plugin $p)) {
+              $entry = New-AgentEntry -plugin $p -agentFile $agentFile
+              if ($null -ne $entry) {
+                $entries += $entry
+              }
+            }
+            $json = @($entries) | ConvertTo-Json -Compress -AsArray
           }
           "entries=$json" >> $env:GITHUB_OUTPUT
 
@@ -202,7 +374,10 @@ jobs:
         env:
           ENTRY_PLUGIN: ${{ matrix.entry.plugin }}
           ENTRY_NAME: ${{ matrix.entry.name }}
+          ENTRY_TARGET_KIND: ${{ matrix.entry.target_kind }}
           ENTRY_SKILLS_PATH: ${{ matrix.entry.skills_path }}
+          ENTRY_AGENTS_PATH: ${{ matrix.entry.agents_path }}
+          ENTRY_EVAL_PATH: ${{ matrix.entry.eval_path }}
           ENTRY_MODEL: ${{ matrix.entry.model }}
           ENTRY_JUDGE: ${{ matrix.entry.judge }}
           ENTRY_JUDGE2: ${{ matrix.entry.judge2 }}
@@ -215,6 +390,8 @@ jobs:
           export LC_ALL=C
           name_re='^[A-Za-z0-9._-]+$'
           path_re='^plugins/[A-Za-z0-9._-]+/skills(/[A-Za-z0-9._-]+)?$'
+          agent_path_re='^plugins/[A-Za-z0-9._-]+/([A-Za-z0-9._-]+/)*[A-Za-z0-9._-]+\.agent\.md$'
+          eval_path_re='^tests/[A-Za-z0-9._-]+/([A-Za-z0-9._-]+/)+eval\.yaml$'
           # A lone-dot component ('.') matches the charset and holds no '..', but
           # it is a path-normalization token, so reject it for names and for any
           # path component ('.../skills/.' or 'plugins/./skills').
@@ -223,6 +400,9 @@ jobs:
               echo "::error::Invalid matrix value '$val' (must match $name_re, not be '.', and not contain '..')"; exit 1
             fi
           done
+          if [ "$ENTRY_TARGET_KIND" != "skill" ] && [ "$ENTRY_TARGET_KIND" != "agent" ]; then
+            echo "::error::Invalid target_kind '$ENTRY_TARGET_KIND' (must be skill or agent)"; exit 1
+          fi
           # Cross-family executor/judge fields (IMPACT-ANALYSIS.md §10) are
           # OPTIONAL: they are empty in the single-model default and non-empty
           # only when the discover job fans the matrix across models. When
@@ -243,9 +423,6 @@ jobs:
           # leaving an effectively-empty skills_path to reach CLI-arg building.
           segs=()
           for seg in $ENTRY_SKILLS_PATH; do segs+=("$seg"); done
-          if [ ${#segs[@]} -eq 0 ]; then
-            echo "::error::Empty or whitespace-only skills_path in matrix entry"; exit 1
-          fi
           # Each segment must additionally belong to THIS entry's plugin: the
           # generic path_re only checks the shape, so bind the prefix to
           # ENTRY_PLUGIN via literal comparison (ENTRY_PLUGIN is already
@@ -265,6 +442,33 @@ jobs:
               echo "::error::skills_path segment '$seg' does not belong to plugin '$ENTRY_PLUGIN'"; exit 1
             fi
           done
+          agent_segs=()
+          for seg in $ENTRY_AGENTS_PATH; do agent_segs+=("$seg"); done
+          for seg in "${agent_segs[@]}"; do
+            if ! [[ "$seg" =~ $agent_path_re ]] || [[ "$seg" == *".."* ]]; then
+              echo "::error::Invalid agents_path segment '$seg' (must match $agent_path_re and not contain '..')"; exit 1
+            fi
+            if [[ "$seg" != "plugins/$ENTRY_PLUGIN/"* ]]; then
+              echo "::error::agents_path segment '$seg' does not belong to plugin '$ENTRY_PLUGIN'"; exit 1
+            fi
+          done
+          if [ -n "$ENTRY_EVAL_PATH" ]; then
+            if ! [[ "$ENTRY_EVAL_PATH" =~ $eval_path_re ]] || [[ "$ENTRY_EVAL_PATH" == *".."* ]]; then
+              echo "::error::Invalid eval_path '$ENTRY_EVAL_PATH' (must match $eval_path_re and not contain '..')"; exit 1
+            fi
+            if [[ "$ENTRY_EVAL_PATH" != "tests/$ENTRY_PLUGIN/"* ]]; then
+              echo "::error::eval_path '$ENTRY_EVAL_PATH' does not belong to plugin '$ENTRY_PLUGIN'"; exit 1
+            fi
+          fi
+          if [ "$ENTRY_TARGET_KIND" = "skill" ] && [ ${#segs[@]} -eq 0 ]; then
+            echo "::error::Skill matrix entry has an empty skills_path"; exit 1
+          fi
+          if [ "$ENTRY_TARGET_KIND" = "agent" ] && [ ${#agent_segs[@]} -eq 0 ]; then
+            echo "::error::Agent matrix entry has an empty agents_path"; exit 1
+          fi
+          if [ "$ENTRY_TARGET_KIND" = "agent" ] && [ -z "$ENTRY_EVAL_PATH" ]; then
+            echo "::error::Agent matrix entry has an empty eval_path"; exit 1
+          fi
 
       - name: Checkout skills content
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -277,14 +481,23 @@ jobs:
         env:
           DISPATCH_SKILL: ${{ inputs.skill }}
           PLUGIN: ${{ matrix.entry.plugin }}
+          TARGET_KIND: ${{ matrix.entry.target_kind }}
           SKILLS_PATH: ${{ matrix.entry.skills_path }}
+          AGENTS_PATH: ${{ matrix.entry.agents_path }}
+          EVAL_PATH: ${{ matrix.entry.eval_path }}
         run: |
           set -f
           # For workflow_dispatch targeting a specific skill, only run that one
           # eval. Otherwise derive the specs from this entry's skills_path so a
           # subset leg (per-skill PR entry or shard) reports has_evals accurately
           # instead of installing tools or probing a PAT only to exit empty.
-          if [ -n "$DISPATCH_SKILL" ]; then
+          if [ "$TARGET_KIND" = "agent" ]; then
+            if [ ! -f "$EVAL_PATH" ]; then
+              echo "::error::Agent eval spec does not exist at '$EVAL_PATH'"
+              exit 1
+            fi
+            EVALS="$EVAL_PATH"
+          elif [ -n "$DISPATCH_SKILL" ]; then
             CANDIDATE="tests/$PLUGIN/${DISPATCH_SKILL}/eval.yaml"
             if [ ! -f "$CANDIDATE" ]; then
               echo "::error::workflow_dispatch requested skill '${DISPATCH_SKILL}' in plugin '${PLUGIN}' but no eval spec exists at '${CANDIDATE}'. Check the skill name (case-sensitive) and that the eval file is present."
@@ -300,15 +513,10 @@ jobs:
               skill="${skill#/}"            # "" or "<skill>"
               if [ -n "$skill" ]; then
                 # Sharded/PR entry: only this skill's spec, if one exists.
-                # agent.* skills are excluded from the vally experiment.
-                case "$skill" in
-                  agent.*) continue ;;
-                esac
                 CANDIDATE="tests/$tp/$skill/eval.yaml"
                 [ -f "$CANDIDATE" ] && EVALS="${EVALS}${CANDIDATE}"$'\n'
               else
-                # Whole-plugin entry: every eval spec under the plugin, except
-                # agent.* skills, which the vally experiment glob excludes.
+                # Whole-plugin skill entry: agent evals have their own matrix legs.
                 FOUND=$(find "tests/$tp" -name "eval.yaml" -type f -not -path "*/agent.*/*")
                 [ -n "$FOUND" ] && EVALS="${EVALS}${FOUND}"$'\n'
               fi
@@ -619,7 +827,9 @@ jobs:
         env:
           RESULTS_DIR: artifacts/TestResults/vally/${{ matrix.entry.name }}
           PLUGIN: ${{ matrix.entry.plugin }}
+          TARGET_KIND: ${{ matrix.entry.target_kind }}
           SKILLS_PATH: ${{ matrix.entry.skills_path }}
+          AGENTS_PATH: ${{ matrix.entry.agents_path }}
           DISPATCH_SKILL: ${{ inputs.skill }}
           MODEL: ${{ steps.eval-models.outputs.model }}
           JUDGE_MODEL: ${{ steps.eval-models.outputs.judge-model }}
@@ -638,6 +848,52 @@ jobs:
           rm -f "$RUNNER_TEMP/evaluation-copilot-token"
           export GITHUB_TOKEN
 
+          if [ "$TARGET_KIND" = "agent" ]; then
+            # Vally 0.14 has no custom-agent registration field: its public
+            # EnvironmentConfig exposes skills/files/commands/MCP only, and its
+            # Copilot executor passes skillDirectories but no customAgents.
+            # Run the repository's native SDK agent evaluator instead, then
+            # adapt its evidence into the same schema/result tree as Vally.
+            AGENT_ARGS=()
+            for token in $AGENTS_PATH; do AGENT_ARGS+=("$token"); done
+            if [ ${#AGENT_ARGS[@]} -eq 0 ]; then
+              echo "::error::No custom-agent paths were supplied for $PLUGIN"
+              exit 1
+            fi
+
+            AGENT_RAW_DIR="$RESULTS_DIR/_agent-evaluation"
+            mkdir -p "$AGENT_RAW_DIR"
+            "$RUNNER_TEMP/trusted-validator/skill-validator" evaluate \
+              "${AGENT_ARGS[@]}" \
+              --tests-dir "tests/$PLUGIN" \
+              --model "$MODEL" \
+              --judge-model "$JUDGE_MODEL" \
+              --judge-mode pairwise \
+              --runs 1 \
+              --parallel-skills 1 \
+              --parallel-scenarios 3 \
+              --parallel-runs 1 \
+              --keep-sessions \
+              --verdict-warn-only \
+              --results-dir "$AGENT_RAW_DIR"
+
+            mapfile -d '' AGENT_RESULTS < <(find "$AGENT_RAW_DIR" -name results.json -type f -print0)
+            if [ ${#AGENT_RESULTS[@]} -ne 1 ]; then
+              echo "::error::Expected exactly one native agent results.json, found ${#AGENT_RESULTS[@]}"
+              exit 1
+            fi
+            node "$RUNNER_TEMP/trusted-validator-src/eng/vally-adapter/adapt-agent-results.mjs" \
+              --results-file "${AGENT_RESULTS[0]}" \
+              --output-root "$RESULTS_DIR" \
+              --repo-root "$GITHUB_WORKSPACE" \
+              --expected-evals "$RUNNER_TEMP/evaluation-expected-evals.txt" \
+              --model "$MODEL" \
+              --judge-model "$JUDGE_MODEL"
+            # The adapted schema-v5 result is now authoritative. Remove only
+            # the native aggregate result so recursive downstream collectors do
+            # not count it a second time; preserve sessions.db and session logs.
+            rm -f "${AGENT_RESULTS[0]}"
+          else
           # PLUGIN/SKILLS_PATH/DISPATCH_SKILL arrive via env (validated upstream)
           # rather than inline GitHub-expression interpolation, to avoid
           # injection; set -f keeps the SKILLS_PATH word-split glob-safe.
@@ -867,10 +1123,11 @@ jobs:
               --judge-model "$JUDGE2_MODEL" \
               "${OVERFIT_ARGS[@]}" || echo "::warning::secondary judge ($JUDGE2_MODEL) re-score failed for $PLUGIN"
           fi
+          fi
 
           # Surface how many verdicts were produced.
-          PRODUCED=$(find "$RESULTS_DIR" -name results.json -not -path "$EXPERIMENT_OUT/*" | wc -l | tr -d ' ')
-          echo "Produced $PRODUCED of $EXPECTED_EVAL_COUNT expected skill verdict(s) for $PLUGIN"
+          PRODUCED=$(find "$RESULTS_DIR" -name results.json -not -path "$RESULTS_DIR/_experiment/*" -not -path "$RESULTS_DIR/_agent-evaluation/*" | wc -l | tr -d ' ')
+          echo "Produced $PRODUCED of $EXPECTED_EVAL_COUNT expected target verdict(s) for $PLUGIN"
           if [ "$PRODUCED" -ne "$EXPECTED_EVAL_COUNT" ]; then
             echo "::error::vally produced $PRODUCED result file(s), but the pre-run manifest required $EXPECTED_EVAL_COUNT. The result set is incomplete or contains an unexpected eval."
             exit 1
@@ -935,9 +1192,9 @@ jobs:
         env:
           ENTRY_NAME: ${{ matrix.entry.name }}
         run: |
-          echo "## 🔬 Vally Evaluation Results: $ENTRY_NAME" >> $GITHUB_STEP_SUMMARY
+          echo "## 🔬 Evaluation Results: $ENTRY_NAME" >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
-          echo "Skilled vs baseline, judged head-to-head by \`vally compare\`. Each distinct stimulus gives one gate vote; repeated runs report reliability only. A pass needs a complete comparison, an exact one-sided sign test at p ≤ 0.05, and at least a 20% net win. ⚠️ marks an invalid or inconclusive result. 📉 marks a report-only LLM preference loss, not an objective completion regression." >> $GITHUB_STEP_SUMMARY
+          echo "Isolated target vs baseline, judged head-to-head. Skill targets use \`vally compare\`; custom-agent targets use the native SDK evaluator. Each distinct stimulus gives one gate vote; repeated runs report reliability only. A pass needs a complete comparison, an exact one-sided sign test at p ≤ 0.05, and at least a 20% net win. ⚠️ marks an invalid or inconclusive result. 📉 marks a report-only LLM preference loss, not an objective completion regression." >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
 
           RESULTS_DIR="artifacts/TestResults/vally/$ENTRY_NAME"

.github/workflows/evaluation-workflow-tests.yml

@@ -8,8 +8,10 @@ on:
       - ".github/workflows/evaluation-workflow-tests.yml"
       - "eng/evaluation-tools/**"
       - "eng/evaluation/test_token_failover.py"
+      - "eng/evaluation/path-safety.ps1"
       - "eng/dashboard/**"
       - "eng/vally-adapter/**"
+      - "eng/skill-validator/src/**"
   push:
     branches: [main]
     paths:
@@ -18,8 +20,10 @@ on:
       - ".github/workflows/evaluation-workflow-tests.yml"
       - "eng/evaluation-tools/**"
       - "eng/evaluation/test_token_failover.py"
+      - "eng/evaluation/path-safety.ps1"
       - "eng/dashboard/**"
       - "eng/vally-adapter/**"
+      - "eng/skill-validator/src/**"
   workflow_dispatch:
 
 permissions:

.github/workflows/evaluation.yml

@@ -6,8 +6,8 @@
 # branch (main), NOT from the PR branch, for every trigger below (issue_comment,
 # pull_request_review, pull_request_target, workflow_dispatch). Changes to this
 # file in a PR will not take effect until merged. The Vally harness runs the
-# skills content checked out from the gate-bound commit, so skill/test/eval.yaml
-# changes are evaluated before merge.
+# target content checked out from the gate-bound commit, so skill, agent, and
+# eval.yaml changes are evaluated before merge.
 #
 # For same-repository PRs:
 #   - On PR open/sync, the `pr-status` job posts an initial commit status:
@@ -25,7 +25,7 @@
 #     commit; no downstream job re-reads the live branch head.
 #
 # For scheduled runs:
-#   - Runs daily, evaluates all plugins with skills and tests.
+#   - Runs daily, evaluates all plugins with skill or custom-agent evals.
 #
 # Model for fork PRs:
 #   - Workflow YAML: always from the default branch (enforced by the
@@ -186,15 +186,16 @@ jobs:
           $changedFiles = git diff --name-only --diff-filter=ACMR $mergeBase $head
 
           $hasSkillChanges = $changedFiles |
-            Where-Object { $_ -match '^(plugins/[^/]+/skills|tests/[^/]+)/[^/]+/' } |
+            Where-Object { $_ -match '^(?:plugins/[^/]+/plugin\.json$|plugins/[^/]+/skills/[^/]+/|plugins/[^/]+/(?:[^/]+/)*[^/]+\.agent\.md$|tests/[^/]+/[^/]+/)' } |
             Select-Object -First 1
 
-          # Evaluation pipeline changes need a re-eval. The linter (skill-validator)
-          # is a separate workflow, so eng/skill-validator/ changes are NOT infra
-          # changes here. Documentation files don't affect evaluation.
+          # Evaluation pipeline changes need a re-eval. The native custom-agent
+          # lane executes eng/skill-validator/src, so changes there are evaluator
+          # infrastructure changes. Documentation files don't affect evaluation.
           $hasInfraChanges = $changedFiles |
             Where-Object {
               ($_ -match '^eng/vally-adapter/') -or
+              ($_ -match '^eng/skill-validator/src/') -or
               ($_ -match '^dotnet-skills\.experiment\.yaml$') -or
               $_ -match '^\.github/workflows/(evaluation|evaluation-run)\.yml$'
             } |
@@ -215,7 +216,7 @@ jobs:
             DESC="Submit a review with /evaluate, or comment /evaluate ${{ github.event.pull_request.head.sha }}"
           else
             STATE="success"
-            DESC="No skills to evaluate"
+            DESC="No skills or agents to evaluate"
           fi
 
           gh api "repos/${{ github.repository }}/statuses/${{ github.event.pull_request.head.sha }}" \
@@ -256,12 +257,13 @@ jobs:
           $changedFiles = git diff --name-only --diff-filter=ACMR $mergeBase $head
 
           $hasSkillChanges = $changedFiles |
-            Where-Object { $_ -match '^(plugins/[^/]+/skills|tests/[^/]+)/[^/]+/' } |
+            Where-Object { $_ -match '^(?:plugins/[^/]+/plugin\.json$|plugins/[^/]+/skills/[^/]+/|plugins/[^/]+/(?:[^/]+/)*[^/]+\.agent\.md$|tests/[^/]+/[^/]+/)' } |
             Select-Object -First 1
 
           $hasInfraChanges = $changedFiles |
             Where-Object {
               ($_ -match '^eng/vally-adapter/') -or
+              ($_ -match '^eng/skill-validator/src/') -or
               ($_ -match '^dotnet-skills\.experiment\.yaml$') -or
               $_ -match '^\.github/workflows/(evaluation|evaluation-run)\.yml$'
             } |
@@ -282,7 +284,7 @@ jobs:
             DESC="Fork PR evaluation requires a trusted branch"
           else
             STATE="success"
-            DESC="No skills to evaluate"
+            DESC="No skills or agents to evaluate"
           fi
 
           gh api "repos/${{ github.repository }}/statuses/${{ github.event.pull_request.head.sha }}" \
@@ -724,7 +726,7 @@ jobs:
 
   # ==========================================================================
   # DISCOVER JOB
-  # Find skills to evaluate based on changed files.
+  # Find skills and custom agents to evaluate based on changed files.
   # ==========================================================================
   discover:
     needs: gate
@@ -812,7 +814,7 @@ jobs:
         # head; the gate-bound SHA is used below instead.
         run: git fetch origin +refs/pull/${{ needs.gate.outputs.pr_number }}/head:refs/remotes/origin/pr-head
 
-      - name: Find skills to evaluate
+      - name: Find targets to evaluate
         if: github.event_name != 'schedule' || steps.check-changes.outputs.has_changes == 'true' || github.event_name == 'workflow_dispatch'
         id: find
         env:
@@ -825,6 +827,7 @@ jobs:
         run: |
           $entries = @()
           $plugins = @()
+          . (Join-Path $PWD "eng/evaluation/path-safety.ps1")
 
           # Build matrix entries for a full-plugin evaluation, sharding skills
           # that have eval specs by the optional `executionShard:` top-level tag
@@ -844,7 +847,14 @@ jobs:
             )
             $skillsDir = "plugins/$plugin/skills"
             $skillsRoot = Join-Path $contentRoot $skillsDir
-            $singleEntry = @{ name = $plugin; plugin = $plugin; skills_path = $skillsDir }
+            $singleEntry = @{
+              name = $plugin
+              plugin = $plugin
+              target_kind = "skill"
+              skills_path = $skillsDir
+              agents_path = ""
+              eval_path = ""
+            }
             if (-not (Test-Path $skillsRoot)) {
               return @()
             }
@@ -878,17 +888,125 @@ jobs:
               if ($selectedSkills.Count -gt 0) {
                 $paths = ($evalSkills | ForEach-Object { "$skillsDir/$_" }) -join ' '
                 $name = if ($evalSkills.Count -eq 1) { "$plugin--$($evalSkills[0])" } else { $plugin }
-                return @(@{ name = $name; plugin = $plugin; skills_path = $paths })
+                return @(@{
+                  name = $name
+                  plugin = $plugin
+                  target_kind = "skill"
+                  skills_path = $paths
+                  agents_path = ""
+                  eval_path = ""
+                })
               }
               return @($singleEntry)
             }
             return @($shardGroups.Keys | Sort-Object | ForEach-Object {
               $shardName = $_
               $paths = ($shardGroups[$shardName] | ForEach-Object { "$skillsDir/$_" }) -join ' '
-              @{ name = "$plugin--shard-$shardName"; plugin = $plugin; skills_path = $paths }
+              @{
+                name = "$plugin--shard-$shardName"
+                plugin = $plugin
+                target_kind = "skill"
+                skills_path = $paths
+                agents_path = ""
+                eval_path = ""
+              }
             })
           }
 
+          function Resolve-AgentEvalPath {
+            param(
+              [string]$plugin,
+              [string]$agent,
+              [string]$contentRoot = "."
+            )
+            $testsRoot = Join-Path $contentRoot "tests" $plugin
+            $candidates = @(
+              (Join-Path $testsRoot "agent.$agent" "eval.yaml"),
+              (Join-Path $testsRoot $agent "eval.yaml")
+            )
+            if (Test-Path $testsRoot) {
+              foreach ($subDir in Get-ChildItem -Path $testsRoot -Directory -ErrorAction SilentlyContinue) {
+                $candidates += Join-Path $subDir.FullName "agent.$agent" "eval.yaml"
+                $candidates += Join-Path $subDir.FullName $agent "eval.yaml"
+              }
+            }
+            foreach ($candidate in $candidates) {
+              if (Test-Path $candidate) {
+                return [IO.Path]::GetRelativePath(
+                  [IO.Path]::GetFullPath($contentRoot),
+                  [IO.Path]::GetFullPath($candidate)
+                ).Replace('\', '/')
+              }
+            }
+            return $null
+          }
+
+          function Get-PluginAgentEntries {
+            param(
+              [string]$plugin,
+              [string]$contentRoot = ".",
+              [string[]]$selectedAgents = @()
+            )
+            $pluginRoot = [IO.Path]::GetFullPath((Join-Path $contentRoot "plugins" $plugin))
+            $manifestPath = Join-Path $pluginRoot "plugin.json"
+            if (-not (Test-Path $manifestPath)) { return @() }
+            try {
+              $manifest = Get-Content $manifestPath -Raw | ConvertFrom-Json
+            } catch {
+              Write-Warning "Skipping agent discovery for '$plugin': malformed plugin.json"
+              return @()
+            }
+            $declaredPaths = @($manifest.agents | Where-Object { -not [string]::IsNullOrWhiteSpace("$_") })
+            if ($declaredPaths.Count -eq 0) { $declaredPaths = @("./agents/") }
+
+            $agentFiles = @()
+            foreach ($declaredPath in $declaredPaths) {
+              $candidate = [IO.Path]::GetFullPath((Join-Path $pluginRoot "$declaredPath"))
+              $pluginPrefix = $pluginRoot.TrimEnd([IO.Path]::DirectorySeparatorChar, [IO.Path]::AltDirectorySeparatorChar) + [IO.Path]::DirectorySeparatorChar
+              if (-not $candidate.StartsWith($pluginPrefix, [StringComparison]::Ordinal)) {
+                Write-Warning "Skipping agent path outside plugin '$plugin': $declaredPath"
+                continue
+              }
+              if (-not (Test-Path $candidate)) { continue }
+              if (Test-PathHasReparsePoint -allowedRoot $pluginRoot -path $candidate) {
+                throw "Agent path '$declaredPath' contains a symbolic link or reparse point"
+              }
+              if (Test-Path $candidate -PathType Container) {
+                $agentFiles += Get-ChildItem -Path $candidate -Filter "*.agent.md" -File -ErrorAction SilentlyContinue
+              } elseif ((Test-Path $candidate -PathType Leaf) -and $candidate.EndsWith(".agent.md", [StringComparison]::OrdinalIgnoreCase)) {
+                $agentFiles += Get-Item $candidate
+              }
+            }
+
+            $selected = @{}
+            foreach ($agent in $selectedAgents) { $selected[$agent] = $true }
+            return @($agentFiles |
+              Sort-Object FullName -Unique |
+              Sort-Object Name |
+              ForEach-Object {
+                $agent = $_.Name -replace '\.agent\.md$', ''
+                if (Test-PathHasReparsePoint -allowedRoot $pluginRoot -path $_.FullName) {
+                  throw "Agent file '$($_.FullName)' contains a symbolic link or reparse point"
+                }
+                $evalPath = Resolve-AgentEvalPath -plugin $plugin -agent $agent -contentRoot $contentRoot
+                if ($evalPath -and
+                    ($selectedAgents.Count -eq 0 -or $selected.ContainsKey($agent))) {
+                  $agentPath = [IO.Path]::GetRelativePath(
+                    [IO.Path]::GetFullPath($contentRoot),
+                    [IO.Path]::GetFullPath($_.FullName)
+                  ).Replace('\', '/')
+                  @{
+                    name = "$plugin--agent.$agent"
+                    plugin = $plugin
+                    target_kind = "agent"
+                    skills_path = ""
+                    agents_path = $agentPath
+                    eval_path = $evalPath
+                  }
+                }
+              })
+          }
+
           if ("${{ needs.gate.outputs.pr_number }}" -ne "") {
             # Single-PR run (/evaluate comment, /evaluate review, or the
             # evaluate-now label): detect individual changed skills against the
@@ -911,20 +1029,22 @@ jobs:
             $mergeBase = git merge-base $base $head
             $changedFiles = git diff --name-only --diff-filter=ACMR $mergeBase $head
 
-            # Check if any changed files are in infrastructure paths. The linter
-            # (skill-validator) is a separate workflow, so eng/skill-validator/
-            # changes are NOT infra changes here.
+            # Check if any changed files are in infrastructure paths. The native
+            # custom-agent lane executes eng/skill-validator/src, so those source
+            # changes are evaluation infrastructure changes.
             $hasInfraChanges = $changedFiles |
               Where-Object {
                 ($_ -match '^eng/vally-adapter/') -or
+                ($_ -match '^eng/evaluation/path-safety\.ps1$') -or
+                ($_ -match '^eng/skill-validator/src/') -or
                 ($_ -match '^dotnet-skills\.experiment\.yaml$') -or
                 $_ -match '^\.github/workflows/(evaluation|evaluation-run)\.yml$'
               } |
               Select-Object -First 1
 
-            # Also check for skill/test changes so we don't lose them
+            # Also check for skill, agent, and test changes so we don't lose them.
             $hasSkillChanges = $changedFiles |
-              Where-Object { $_ -match '^(?:plugins/([^/]+)/skills|tests/([^/]+))/([^/]+)/' } |
+              Where-Object { $_ -match '^(?:plugins/[^/]+/plugin\.json$|plugins/[^/]+/skills/[^/]+/|plugins/[^/]+/(?:[^/]+/)*[^/]+\.agent\.md$|tests/[^/]+/[^/]+/)' } |
               Select-Object -First 1
 
             if ($hasInfraChanges -and -not $hasSkillChanges) {
@@ -933,21 +1053,62 @@ jobs:
               # the smoke-test fast while still catching regressions.
               $allPlugins = @(Get-ChildItem -Path (Join-Path $contentRoot "plugins") -Directory -ErrorAction SilentlyContinue |
                 Where-Object {
-                  (Test-Path (Join-Path $_.FullName "skills")) -and
+                  (Test-Path (Join-Path $_.FullName "plugin.json")) -and
                   (Test-Path (Join-Path $contentRoot "tests" $_.Name))
                 } |
                 Select-Object -ExpandProperty Name)
               $plugins = @($allPlugins | Get-Random -Count ([Math]::Min(2, $allPlugins.Count)))
               Write-Host "Infrastructure changes detected, evaluating random subset: $($plugins -join ', ')"
-              $entries = @($plugins | ForEach-Object { Get-PluginShardEntries -plugin $_ -contentRoot $contentRoot })
+              $entries = @($plugins | ForEach-Object {
+                Get-PluginShardEntries -plugin $_ -contentRoot $contentRoot
+                Get-PluginAgentEntries -plugin $_ -contentRoot $contentRoot
+              })
             } else {
-              # Extract unique plugin/skill pairs from changed files
+              # Extract unique plugin/skill pairs from changed skill sources and
+              # non-agent eval directories.
               $changedPairs = @($changedFiles |
-                Where-Object { $_ -match '^(?:plugins/([^/]+)/skills|tests/([^/]+))/([^/]+)/' } |
-                ForEach-Object {
-                  $p = if ($Matches[1]) { $Matches[1] } else { $Matches[2] }
-                  "$p/$($Matches[3])"
+                Where-Object {
+                  $_ -match '^plugins/([^/]+)/skills/([^/]+)/' -or
+                  ($_ -match '^tests/([^/]+)/([^/]+)/' -and $Matches[2] -notlike 'agent.*')
                 } |
+                ForEach-Object {
+                  if ($_ -match '^plugins/([^/]+)/skills/([^/]+)/') {
+                    "$($Matches[1])/$($Matches[2])"
+                  } elseif ($_ -match '^tests/([^/]+)/([^/]+)/') {
+                    "$($Matches[1])/$($Matches[2])"
+                  }
+                } |
+                Sort-Object -Unique)
+
+              $changedAgentSourcePlugins = @($changedFiles |
+                ForEach-Object {
+                  if ($_ -match '^plugins/([^/]+)/(?:[^/]+/)*[^/]+\.agent\.md$') {
+                    $Matches[1]
+                  }
+                } |
+                Where-Object { $_ } |
+                Sort-Object -Unique)
+              $changedSkillSourcePlugins = @($changedFiles |
+                ForEach-Object {
+                  if ($_ -match '^plugins/([^/]+)/skills/[^/]+/') {
+                    $Matches[1]
+                  }
+                } |
+                Where-Object { $_ } |
+                Sort-Object -Unique)
+              $changedManifestPlugins = @($changedFiles |
+                ForEach-Object {
+                  if ($_ -match '^plugins/([^/]+)/plugin\.json$') {
+                    $Matches[1]
+                  }
+                } |
+                Where-Object { $_ } |
+                Sort-Object -Unique)
+              $changedTestPlugins = @($changedFiles |
+                ForEach-Object {
+                  if ($_ -match '^tests/([^/]+)/') { $Matches[1] }
+                } |
+                Where-Object { $_ } |
                 Sort-Object -Unique)
 
               # Filter to skills that have a SKILL.md and a tests directory,
@@ -965,10 +1126,28 @@ jobs:
                   $changedSkillsByPlugin[$plugin] += $skill
                 }
               }
-              $entries = @($changedSkillsByPlugin.Keys | Sort-Object | ForEach-Object {
+              $entries = @($changedSkillsByPlugin.Keys |
+                Where-Object { $_ -notin $changedManifestPlugins } |
+                Sort-Object |
+                ForEach-Object {
                 $plugin = $_
                 Get-PluginShardEntries -plugin $plugin -contentRoot $contentRoot -selectedSkills ($changedSkillsByPlugin[$plugin] | Sort-Object -Unique)
               })
+              $entries += @($changedManifestPlugins | ForEach-Object {
+                Get-PluginShardEntries -plugin $_ -contentRoot $contentRoot
+              })
+
+              $agentPlugins = @(
+                $changedAgentSourcePlugins + $changedSkillSourcePlugins + $changedManifestPlugins + $changedTestPlugins |
+                  Sort-Object -Unique)
+              $entries += @($agentPlugins | ForEach-Object {
+                $plugin = $_
+                # Any agent source can be a declared dependency of another evaluated
+                # agent. Skills and shared test fixtures can also be dependencies, and
+                # manifest changes alter the production registration surface. Rerun
+                # every agent eval in an affected plugin rather than miss an indirect edge.
+                Get-PluginAgentEntries -plugin $plugin -contentRoot $contentRoot
+              })
             }
 
             git worktree remove /tmp/pr-content --force 2>$null
@@ -988,18 +1167,25 @@ jobs:
               if ($dispatchPlugin -notmatch '^[a-zA-Z0-9._-]+$') {
                 throw "workflow_dispatch input plugin='$dispatchPlugin' must match ^[a-zA-Z0-9._-]+$ (single directory name, no path separators)"
               }
-              if (-not (Test-Path (Join-Path "plugins" $dispatchPlugin "skills")) -or
+              if (-not (Test-Path (Join-Path "plugins" $dispatchPlugin "plugin.json")) -or
                   -not (Test-Path (Join-Path "tests" $dispatchPlugin))) {
-                throw "workflow_dispatch input plugin='$dispatchPlugin' is not a valid plugin (must have plugins/<name>/skills and tests/<name>)"
+                throw "workflow_dispatch input plugin='$dispatchPlugin' is not a valid plugin (must have plugin.json and tests/<name>)"
               }
               $plugins = @($dispatchPlugin)
               Write-Host "workflow_dispatch: evaluating only $dispatchPlugin"
             } else {
               $plugins = @(Get-ChildItem -Path "plugins" -Directory |
-                Where-Object { (Test-Path (Join-Path $_.FullName "skills")) -and (Test-Path (Join-Path "tests" $_.Name)) -and ($_.Name -notin $excludeFromSchedule) } |
+                Where-Object {
+                  (Test-Path (Join-Path $_.FullName "plugin.json")) -and
+                  (Test-Path (Join-Path "tests" $_.Name)) -and
+                  ($_.Name -notin $excludeFromSchedule)
+                } |
                 Select-Object -ExpandProperty Name)
             }
-            $entries = @($plugins | ForEach-Object { Get-PluginShardEntries -plugin $_ })
+            $entries = @($plugins | ForEach-Object {
+              Get-PluginShardEntries -plugin $_
+              Get-PluginAgentEntries -plugin $_
+            })
           }
           # Only plugins represented by a non-empty eval entry are downstream
           # publication targets.
@@ -1126,10 +1312,13 @@ jobs:
                 $expanded += @{
                   name        = "$($e.name)--$m"
                   plugin      = $e.plugin
+                  target_kind = if ($e.target_kind) { $e.target_kind } else { "skill" }
                   skills_path = $e.skills_path
+                  agents_path = $e.agents_path
+                  eval_path   = $e.eval_path
                   model       = $m
                   judge       = $route.judge
-                  judge2      = $j2
+                  judge2      = if ($e.target_kind -eq "agent") { "" } else { $j2 }
                 }
               }
             }
@@ -1143,6 +1332,8 @@ jobs:
           # path-normalization token, so reject it for names and path segments.
           $namePattern = '\A[A-Za-z0-9._-]+\z'
           $pathPattern = '\Aplugins/[A-Za-z0-9._-]+/skills(/[A-Za-z0-9._-]+)?\z'
+          $agentPathPattern = '\Aplugins/[A-Za-z0-9._-]+/(?:[A-Za-z0-9._-]+/)*[A-Za-z0-9._-]+\.agent\.md\z'
+          $evalPathPattern = '\Atests/[A-Za-z0-9._-]+/(?:[A-Za-z0-9._-]+/)+eval\.yaml\z'
           foreach ($e in $entries) {
             if ("$($e.plugin)" -notmatch $namePattern -or "$($e.plugin)" -match '\.\.' -or "$($e.plugin)" -eq '.') {
               throw "Refusing unsafe matrix entry: plugin '$($e.plugin)' must match $namePattern, not be '.', and not contain '..'"
@@ -1150,6 +1341,9 @@ jobs:
             if ("$($e.name)" -notmatch $namePattern -or "$($e.name)" -match '\.\.' -or "$($e.name)" -eq '.') {
               throw "Refusing unsafe matrix entry: name '$($e.name)' must match $namePattern, not be '.', and not contain '..'"
             }
+            if ("$($e.target_kind)" -notin @('skill', 'agent')) {
+              throw "Refusing unsafe matrix entry: target_kind '$($e.target_kind)' must be 'skill' or 'agent'"
+            }
             # model/judge are always present; judge2 is optional (empty unless the
             # scheduled dual-judge cadence sets it). All flow into CLI args in the
             # runner, so hold them to the same strict allowlist as names.
@@ -1165,9 +1359,6 @@ jobs:
             # not a no-op. Filter on the trimmed value so whitespace-only
             # segments count as empty (matching the bash validation step).
             $spSegments = @("$($e.skills_path)" -split ' ' | Where-Object { $_.Trim() })
-            if ($spSegments.Count -eq 0) {
-              throw "Refusing unsafe matrix entry: skills_path for entry '$($e.name)' is empty"
-            }
             # Each segment must also belong to THIS entry's plugin: $pathPattern
             # only checks the generic shape, so bind the prefix to $e.plugin via
             # ordinal string comparison. This blocks a mismatched entry such as
@@ -1186,6 +1377,31 @@ jobs:
                 throw "Refusing unsafe matrix entry: skills_path segment '$sp' does not belong to plugin '$($e.plugin)'"
               }
             }
+            $agentSegments = @("$($e.agents_path)" -split ' ' | Where-Object { $_.Trim() })
+            foreach ($ap in $agentSegments) {
+              if ($ap -notmatch $agentPathPattern -or $ap -match '\.\.') {
+                throw "Refusing unsafe matrix entry: agents_path segment '$ap' must match $agentPathPattern and not contain '..'"
+              }
+              if (-not $ap.StartsWith("plugins/$($e.plugin)/", [System.StringComparison]::Ordinal)) {
+                throw "Refusing unsafe matrix entry: agents_path segment '$ap' does not belong to plugin '$($e.plugin)'"
+              }
+            }
+            $evalPath = "$($e.eval_path)"
+            if ($evalPath -and ($evalPath -notmatch $evalPathPattern -or $evalPath -match '\.\.')) {
+              throw "Refusing unsafe matrix entry: eval_path '$evalPath' must match $evalPathPattern and not contain '..'"
+            }
+            if ($evalPath -and -not $evalPath.StartsWith("tests/$($e.plugin)/", [System.StringComparison]::Ordinal)) {
+              throw "Refusing unsafe matrix entry: eval_path '$evalPath' does not belong to plugin '$($e.plugin)'"
+            }
+            if ($e.target_kind -eq 'skill' -and $spSegments.Count -eq 0) {
+              throw "Refusing unsafe matrix entry: skill target '$($e.name)' has no skills_path"
+            }
+            if ($e.target_kind -eq 'agent' -and $agentSegments.Count -eq 0) {
+              throw "Refusing unsafe matrix entry: agent target '$($e.name)' has no agents_path"
+            }
+            if ($e.target_kind -eq 'agent' -and -not $evalPath) {
+              throw "Refusing unsafe matrix entry: agent target '$($e.name)' has no eval_path"
+            }
           }
 
           # Output entries for evaluate matrix
@@ -1422,7 +1638,7 @@ jobs:
         run: |
           if [[ "${{ needs.evaluate.result }}" == "skipped" && "${{ needs.discover.result }}" == "success" && "${{ needs.discover.outputs.has_entries }}" != "true" ]]; then
             STATE="success"
-            DESC="No skills to evaluate"
+            DESC="No skills or agents to evaluate"
           elif [[ "${{ needs.gate.outputs.is_fork }}" == "true" ]]; then
             STATE="failure"
             DESC="Fork PR evaluation requires a trusted branch"
@@ -1452,7 +1668,7 @@ jobs:
           RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
 
           if [[ "${{ needs.discover.result }}" == "success" && "${{ needs.discover.outputs.has_entries }}" != "true" ]]; then
-            BODY="⏭️ No skills to evaluate — no changed skills with tests were found in this PR. [View workflow run](${RUN_URL})"
+            BODY="⏭️ No skills or agents to evaluate — no changed targets with eval specs were found in this PR. [View workflow run](${RUN_URL})"
           elif [[ "${{ needs.gate.outputs.is_fork }}" == "true" ]]; then
             BODY="🔒 Secret-backed evaluation is disabled for fork PRs. A maintainer must review and promote the change to a trusted repository branch before running \`/evaluate\`. [View workflow run](${RUN_URL})"
           else

CONTRIBUTING.md

@@ -340,6 +340,14 @@ stimuli:
 
 Each skill is evaluated in up to three variants — **baseline** (no skills), **skilled** (only the skill under test), and **plugin** (the whole plugin loaded) — and a skill "passes" only when the skilled run is a *credible* improvement over baseline. To assert that a skill should stay dormant for an out-of-scope task, add `expect_activation: false` to that stimulus. Dormancy is an isolated-skill activation contract: unexpected activation blocks a pass, while the stimulus's retained comparison does not vote in preference. See any existing `tests/*/*/eval.yaml` for a fuller example of the grader and stimulus format.
 
+Custom-agent evals live at `tests/<plugin>/agent.<name>/eval.yaml` and use the
+same baseline / isolated / plugin roles and verdict policy. Vally 0.14 cannot
+register custom agents, so CI executes those specs through the native Copilot
+SDK runner: isolated runs register the target agent plus declared dependencies,
+and plugin runs register the complete production plugin skill/agent surface.
+Agent results carry `skillKind: agent` and retain target activation, nested
+delegation, invoked skills, tools, completion, token, and wall-time evidence.
+
 #### Size the eval so it can return a verdict
 
 The pass gate gives each distinct stimulus one vote. Repeated runs collapse to one
@@ -386,6 +394,11 @@ Prerequisites: Node.js 20+ and the [GitHub CLI](https://cli.github.com) signed i
 # Run tests for a whole plugin
 ./eng/run-skill-evals.sh dotnet-msbuild
 
+# Exercise one custom-agent eval through the native SDK lane
+dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate \
+  plugins/dotnet-msbuild/agents/msbuild.agent.md \
+  --tests-dir tests/dotnet-msbuild --runs 1 --verdict-warn-only
+
 # Run every skill's tests
 ./eng/run-skill-evals.sh
 ```

dotnet-skills.experiment.yaml

@@ -8,8 +8,10 @@
 #   tests/<plugin>/<skill>/eval.yaml -> plugins/<plugin>/skills/<skill>
 # i.e. ${eval.grandparent}=<plugin> and ${eval.parent}=<skill>.
 #
-# The `agent.*` evals are excluded: they exercise orchestrator agents and do
-# not map to a single plugins/<plugin>/skills/<skill> directory.
+# The `agent.*` evals are excluded from this Vally experiment because Vally
+# 0.14 cannot register custom agents. evaluation-run.yml executes those specs
+# through the native Copilot SDK agent lane and adapts them into the same
+# result/dashboard schema.
 #
 # `overrides:` deliberately does NOT set `runs`. Precedence is
 # "CLI flags > experiment overrides > eval defaults", and the merge is a plain

eng/dashboard/dashboard.js

@@ -363,7 +363,8 @@
     }
     if (dormant) parts.push(`${dormant} dormant as expected`);
     if (active) parts.push(`${active} activated`);
-    return parts.length ? parts.join(' · ') : 'Activation evidence unavailable';
+    const prefix = verdict.skillKind === 'agent' ? 'Agent' : 'Skill';
+    return parts.length ? `${prefix}: ${parts.join(' · ')}` : `${prefix} activation evidence unavailable`;
   }
 
   function safeEvidenceUrl(value) {
@@ -413,6 +414,18 @@
         ? '; preference: excluded'
         : '; preference: eligible';
       const pluginStatus = s.plugin ? `; plugin: ${activationStatusLabel(s.plugin)}` : '';
+      const delegated = Array.isArray(s.delegatedAgents) && s.delegatedAgents.length
+        ? `; delegated: ${s.delegatedAgents.join(', ')}`
+        : '';
+      const skills = Array.isArray(s.invokedSkills) && s.invokedSkills.length
+        ? `; skills: ${s.invokedSkills.join(', ')}`
+        : '';
+      const tools = Array.isArray(s.isolatedTools) && s.isolatedTools.length
+        ? `; tools: ${s.isolatedTools.join(', ')}`
+        : '';
+      const completion = typeof s.isolatedCompleted === 'boolean'
+        ? `; completed: ${s.isolatedCompleted ? 'yes' : 'no'}`
+        : '';
       const activationOnly = [];
       if (s.isolatedActivationOnlyFailedRuns) {
         activationOnly.push(`isolated activation-only failures: ${s.isolatedActivationOnlyFailedRuns}`);
@@ -423,7 +436,7 @@
       const activationOnlyStatus = activationOnly.length
         ? `; ${activationOnly.join('; ')}`
         : '';
-      return `<li><strong>${escapeHtml(s.scenarioName)}</strong> (${escapeHtml(expectation)}): isolated: ${escapeHtml(activationStatusLabel(s.isolated))}${escapeHtml(pluginStatus)}${escapeHtml(activationOnlyStatus)}${escapeHtml(preference)}</li>`;
+      return `<li><strong>${escapeHtml(s.scenarioName)}</strong> (${escapeHtml(expectation)}): isolated: ${escapeHtml(activationStatusLabel(s.isolated))}${escapeHtml(pluginStatus)}${escapeHtml(delegated)}${escapeHtml(skills)}${escapeHtml(tools)}${escapeHtml(completion)}${escapeHtml(activationOnlyStatus)}${escapeHtml(preference)}</li>`;
     }).join('');
     return `
       <div>${escapeHtml(activationSummary(verdict))}</div>
@@ -515,6 +528,7 @@
           <th scope="row">
             ${escapeHtml(verdict.skillName)}
             ${verdict.skillKind === 'reference' ? '<span class="evidence-tag">reference</span>' : ''}
+            ${verdict.skillKind === 'agent' ? '<span class="evidence-tag">agent</span>' : ''}
           </th>
           <td>
             <span class="verdict-badge ${display.cls}">${escapeHtml(display.label)}</span>
@@ -532,7 +546,7 @@
           <div class="evidence-table-wrap">
             <table class="evidence-table">
               <caption>Authoritative verdict and supporting evidence for ${escapeHtml(model)}</caption>
-              <thead><tr><th>Skill</th><th>Verdict</th><th>Gate evidence</th><th>Activation</th><th>Judge evidence</th></tr></thead>
+              <thead><tr><th>Target</th><th>Verdict</th><th>Gate evidence</th><th>Activation</th><th>Judge evidence</th></tr></thead>
               <tbody>${rows}</tbody>
             </table>
           </div>

eng/dashboard/generate-benchmark-data.ps1

@@ -277,7 +277,8 @@ $verdictEvidence = [System.Collections.Generic.List[object]]::new()
 
 foreach ($verdict in $results.verdicts) {
     $skillName = $verdict.skillName
-    $isReferenceSkill = Test-ReferenceSkill -Plugin $PluginName -Skill $skillName
+    $isAgent = $verdict.PSObject.Properties['skillKind'] -and $verdict.skillKind -eq "agent"
+    $isReferenceSkill = -not $isAgent -and (Test-ReferenceSkill -Plugin $PluginName -Skill $skillName)
     $activationScenarios = [System.Collections.Generic.List[object]]::new()
     $judgeRationales = [System.Collections.Generic.List[object]]::new()
 
@@ -293,13 +294,46 @@ foreach ($verdict in $results.verdicts) {
         if ($scenario.PSObject.Properties['expectActivation'] -and $scenario.expectActivation -eq $false) {
             $expectActivation = $false
         }
-        # Support both old (skillActivation) and new (skillActivationIsolated) JSON schemas
-        $sa = if ($scenario.PSObject.Properties['skillActivationIsolated']) { $scenario.skillActivationIsolated } else { $scenario.skillActivation }
+        # Agent results have exact target-agent activation. Skill results retain
+        # the existing skillActivation fields and compatibility alias.
+        $sa = if ($isAgent -and $scenario.PSObject.Properties['agentActivationIsolated']) {
+            $scenario.agentActivationIsolated
+        } elseif ($scenario.PSObject.Properties['skillActivationIsolated']) {
+            $scenario.skillActivationIsolated
+        } else {
+            $scenario.skillActivation
+        }
         if ($sa -and -not $sa.activated -and $expectActivation -and -not $isReferenceSkill) {
             $notActivated = $true
         }
 
-        $saPluginForEvidence = if ($scenario.PSObject.Properties['skillActivationPlugin']) { $scenario.skillActivationPlugin } else { $null }
+        $saPluginForEvidence = if ($isAgent -and $scenario.PSObject.Properties['agentActivationPlugin']) {
+            $scenario.agentActivationPlugin
+        } elseif ($scenario.PSObject.Properties['skillActivationPlugin']) {
+            $scenario.skillActivationPlugin
+        } else {
+            $null
+        }
+        $invokedAgents = [object[]]@()
+        $delegatedAgents = [object[]]@()
+        $invokedSkills = [object[]]@()
+        $isolatedTools = [object[]]@()
+        $pluginTools = [object[]]@()
+        if ($isAgent) {
+            $invokedAgents = [object[]]@($sa.invokedAgents | Where-Object { $null -ne $_ })
+            $delegatedAgents = [object[]]@($sa.delegatedAgents | Where-Object { $null -ne $_ })
+            if ($scenario.PSObject.Properties['skillActivationIsolated']) {
+                $invokedSkills = [object[]]@(
+                    $scenario.skillActivationIsolated.detectedSkills |
+                        Where-Object { $null -ne $_ })
+            }
+            if ($scenario.skilledIsolated.metrics.toolCallBreakdown) {
+                $isolatedTools = [object[]]@($scenario.skilledIsolated.metrics.toolCallBreakdown.PSObject.Properties.Name)
+            }
+            if ($null -ne $scenario.skilledPlugin -and $scenario.skilledPlugin.metrics.toolCallBreakdown) {
+                $pluginTools = [object[]]@($scenario.skilledPlugin.metrics.toolCallBreakdown.PSObject.Properties.Name)
+            }
+        }
         $activationScenarios.Add([ordered]@{
             scenarioName = $scenario.scenarioName
             expectation  = if ($isReferenceSkill) { "reference" } elseif ($expectActivation) { "active" } else { "dormant" }
@@ -310,21 +344,30 @@ foreach ($verdict in $results.verdicts) {
                 $true
             }
             isolated     = Get-ActivationStatus -Activation $sa -ExpectActivation $expectActivation -IsReferenceSkill $isReferenceSkill
-            isolatedActivationOnlyFailedRuns = if ($sa -and $sa.PSObject.Properties['failedActivationOnlyCompletions']) {
-                [int]$sa.failedActivationOnlyCompletions
+            isolatedActivationOnlyFailedRuns = if ($scenario.skillActivationIsolated -and $scenario.skillActivationIsolated.PSObject.Properties['failedActivationOnlyCompletions']) {
+                [int]$scenario.skillActivationIsolated.failedActivationOnlyCompletions
             } else {
                 0
             }
-            plugin       = if ($null -ne $saPluginForEvidence) {
+            plugin       = if ($isAgent -and $null -ne $saPluginForEvidence) {
+                Get-ActivationStatus -Activation $saPluginForEvidence -ExpectActivation $expectActivation -IsReferenceSkill $false
+            } elseif ($null -ne $saPluginForEvidence) {
                 Get-PluginActivityStatus -Activation $saPluginForEvidence
             } else {
                 $null
             }
-            pluginActivationOnlyFailedRuns = if ($saPluginForEvidence -and $saPluginForEvidence.PSObject.Properties['failedActivationOnlyCompletions']) {
-                [int]$saPluginForEvidence.failedActivationOnlyCompletions
+            pluginActivationOnlyFailedRuns = if ($scenario.skillActivationPlugin -and $scenario.skillActivationPlugin.PSObject.Properties['failedActivationOnlyCompletions']) {
+                [int]$scenario.skillActivationPlugin.failedActivationOnlyCompletions
             } else {
                 0
             }
+            invokedAgents = $invokedAgents
+            delegatedAgents = $delegatedAgents
+            invokedSkills = $invokedSkills
+            isolatedTools = $isolatedTools
+            pluginTools = $pluginTools
+            isolatedCompleted = if ($isAgent) { $scenario.skilledIsolated.metrics.taskCompleted -eq $true } else { $null }
+            pluginCompleted = if ($isAgent -and $scenario.skilledPlugin) { $scenario.skilledPlugin.metrics.taskCompleted -eq $true } else { $null }
         })
 
         # Prefer paired-judge evidence because it explains the W/T/L vote. Fall
@@ -583,19 +626,39 @@ foreach ($verdict in $results.verdicts) {
     $links = [System.Collections.Generic.List[object]]::new()
     if ($commit.id -and "$($commit.id)" -match '^[0-9a-fA-F]{7,40}$') {
         $revision = "$($commit.id)"
-        $sourceRelativePath = if ($skillName.StartsWith("agent.")) {
+        $sourceRelativePath = if ($isAgent) {
             $agentName = $skillName.Substring("agent.".Length)
-            "plugins/$PluginName/agents/$agentName.agent.md"
+            $declaredPath = "$($verdict.skillPath)" -replace '\\', '/'
+            $pluginPattern = [Regex]::Escape($PluginName)
+            if ($declaredPath -match "^plugins/$pluginPattern/(?:[A-Za-z0-9._-]+/)*[A-Za-z0-9._-]+\.agent\.md$" -and
+                $declaredPath -notmatch '(^|/)\.\.(/|$)') {
+                $declaredPath
+            } else {
+                "plugins/$PluginName/agents/$agentName.agent.md"
+            }
         } else {
             "plugins/$PluginName/skills/$skillName/SKILL.md"
         }
         $links.Add([ordered]@{
-            label = if ($skillName.StartsWith("agent.")) { "Agent source" } else { "Skill source" }
+            label = if ($isAgent) { "Agent source" } else { "Skill source" }
             url   = "https://github.com/dotnet/skills/blob/$revision/$sourceRelativePath"
         })
+        $declaredEvalPath = if ($results.PSObject.Properties['evalFile']) {
+            "$($results.evalFile)" -replace '\\', '/'
+        } else {
+            ""
+        }
+        $evalRelativePath = if (
+            $declaredEvalPath -match "^tests/$pluginPattern/(?:[A-Za-z0-9._-]+/)+eval\.yaml$" -and
+            $declaredEvalPath -notmatch '(^|/)\.\.(/|$)'
+        ) {
+            $declaredEvalPath
+        } else {
+            "tests/$PluginName/$skillName/eval.yaml"
+        }
         $links.Add([ordered]@{
             label = "Eval source"
-            url   = "https://github.com/dotnet/skills/blob/$revision/tests/$PluginName/$skillName/eval.yaml"
+            url   = "https://github.com/dotnet/skills/blob/$revision/$evalRelativePath"
         })
     }
     if ($commit.url) {
@@ -604,7 +667,7 @@ foreach ($verdict in $results.verdicts) {
 
     $verdictEvidence.Add([ordered]@{
         skillName          = $skillName
-        skillKind          = if ($isReferenceSkill) { "reference" } else { "invocable" }
+        skillKind          = if ($isAgent) { "agent" } elseif ($isReferenceSkill) { "reference" } else { "invocable" }
         state              = if ($verdict.PSObject.Properties['state']) { $verdict.state } else { $null }
         stateReason        = if ($verdict.PSObject.Properties['stateReason']) { $verdict.stateReason } else { $null }
         passed             = $verdict.passed -eq $true
@@ -651,6 +714,7 @@ $skillValueKey = "SkillValue"
 $skillValueSkills = [System.Collections.Generic.List[object]]::new()
 foreach ($verdict in $results.verdicts) {
     $skillName = $verdict.skillName
+    $isAgent = $verdict.PSObject.Properties['skillKind'] -and $verdict.skillKind -eq "agent"
 
     $activationExpected = 0   # scenarios where the skill is expected to fire
     $activationFired    = 0   # of those, how many actually fired in the treatment arm
@@ -675,7 +739,13 @@ foreach ($verdict in $results.verdicts) {
         if ($scenario.PSObject.Properties['expectActivation'] -and $scenario.expectActivation -eq $false) {
             $expectActivation = $false
         }
-        $sa = if ($scenario.PSObject.Properties['skillActivationIsolated']) { $scenario.skillActivationIsolated } else { $scenario.skillActivation }
+        $sa = if ($isAgent -and $scenario.PSObject.Properties['agentActivationIsolated']) {
+            $scenario.agentActivationIsolated
+        } elseif ($scenario.PSObject.Properties['skillActivationIsolated']) {
+            $scenario.skillActivationIsolated
+        } else {
+            $scenario.skillActivation
+        }
         if ($expectActivation) {
             $activationExpected++
             if ($sa -and $sa.activated) { $activationFired++ }

eng/eval-quality/README.md

@@ -220,9 +220,10 @@ the gate errors on an entry that is stale, duplicated, or no longer needed, and
 *new* relative to the base branch. Without that second half, a PR could add a
 below-floor eval and exempt it in the same change — the defect the floor exists
 to prevent, relocated one file over. Renames are read from git, so moving a
-grandfathered eval is not treated as growth. `agent.*` evals are exempt
-outright: the experiment's `evals:` glob excludes them, so no verdict is ever
-computed and the floor has nothing to protect.
+grandfathered eval is not treated as growth. `agent.*` evals remain outside the
+Vally skill experiment but are not exempt from the quality floor: the native
+SDK agent lane adapts them into the same sign-test verdict schema, so they need
+the same minimum preference-eligible task breadth.
 
 ### 9. Duplicate key in a mapping
 

eng/eval-quality/check_eval_quality.py

@@ -50,7 +50,7 @@ state, declared numbers, or YAML shape/keys — so it cannot fire spuriously on
 well-written content.
 
 REPORTS warnings for pre-existing debt and judgement calls: grandfathered
-underpowered evals, orphaned fixtures, skills with no eval, and dormancy guards
+underpowered evals, orphaned fixtures, targets with no eval, and dormancy guards
 that appear to lack an anti-hijack rubric item. Warnings do not fail unless
 `--strict` is passed. That last one is deliberately a warning: detecting "the
 rubric says the skill should stay dormant" needs phrase matching, which will
@@ -413,8 +413,6 @@ def report_knife_edge(specs: list[str]) -> None:
     """
     band = []
     for spec in specs:
-        if os.path.basename(os.path.dirname(spec)).startswith("agent."):
-            continue
         try:
             with open(spec, encoding="utf-8") as fh:
                 doc = yaml.load(fh, NoDuplicateKeys) or {}
@@ -453,15 +451,9 @@ def check_power(specs: list[str]) -> None:
     allowed = load_allowlist()
     allowed_set = set(allowed)
     spec_set = set(specs)
-    thin, listed_thin, agent_specs = [], [], set()
+    thin, listed_thin = [], []
 
     for spec in specs:
-        # `agent.*` evals are excluded from dotnet-skills.experiment.yaml's
-        # `evals:` glob, so no verdict is ever computed for them and the floor
-        # has nothing to protect.
-        if os.path.basename(os.path.dirname(spec)).startswith("agent."):
-            agent_specs.add(spec)
-            continue
         with open(spec, encoding="utf-8") as fh:
             doc = yaml.safe_load(fh) or {}
         scenarios, dormancy, runs, paired_runs = eval_evidence_counts(doc)
@@ -495,11 +487,7 @@ def check_power(specs: list[str]) -> None:
 
     # Ratchet: the allowlist is a debt ledger, so it must only ever shrink.
     for spec in sorted(allowed_set - {s for _, _, _, _, s in listed_thin}):
-        if spec in agent_specs:
-            errors.append(
-                f"{ALLOWLIST} lists '{spec}', but agent.* evals are excluded from the experiment "
-                f"and never receive a verdict, so they never need an exemption. Remove the line.")
-        elif spec not in spec_set:
+        if spec not in spec_set:
             errors.append(
                 f"{ALLOWLIST} lists '{spec}', which is not an eval spec in this repo. "
                 f"Remove the stale line.")

eng/eval-quality/selftest_eval_quality.py

@@ -395,14 +395,13 @@ def runs_do_not_lift_a_single_scenario_over_the_floor(d):
     write_single_stimulus(d, runs=5)
 
 
-def agent_eval_exempted(d):
-    # agent.* evals never receive a verdict, so they never need an exemption —
-    # and an entry for one would otherwise sit in the ledger forever.
+def underpowered_agent_eval(d):
+    # Agent evals now receive the same sign-test verdict as skill evals, so the
+    # preference-eligible stimulus floor applies to them too.
     ev = os.path.join(d, "tests", "demo", "agent.widget")
     os.makedirs(ev)
     with open(os.path.join(ev, "eval.yaml"), "w") as f:
         f.write("name: agent-widget\nstimuli:\n  - name: One\n    prompt: go\n    rubric:\n      - Did it\n")
-    write_allowlist(d, "tests/demo/agent.widget/eval.yaml")
 
 
 def commit(d, message):
@@ -486,7 +485,7 @@ results = [
                 "(4 preference paired run(s))"),
     case("stale exemption for an eval that now qualifies", allowlisted_eval_that_now_meets_the_floor, expect_fail=True),
     case("exemption for a spec that no longer exists", allowlist_entry_for_a_spec_that_does_not_exist, expect_fail=True),
-    case("exemption for an agent.* eval that never needs one", agent_eval_exempted, expect_fail=True),
+    case("underpowered agent eval", underpowered_agent_eval, expect_fail=True),
     case("runs cannot lift one scenario over the floor",
          runs_do_not_lift_a_single_scenario_over_the_floor, expect_fail=True),
     case("ledger unchanged since its base", allowlist_unchanged_since_base,

eng/evaluation/path-safety.ps1

@@ -0,0 +1,43 @@
+function Test-PathHasReparsePoint {
+  param(
+    [Parameter(Mandatory = $true)]
+    [string]$AllowedRoot,
+    [Parameter(Mandatory = $true)]
+    [string]$Path
+  )
+
+  $root = [IO.Path]::TrimEndingDirectorySeparator(
+    [IO.Path]::GetFullPath($AllowedRoot))
+  $fullPath = [IO.Path]::GetFullPath($Path)
+  try {
+    if (([IO.File]::GetAttributes($root) -band [IO.FileAttributes]::ReparsePoint) -ne 0) {
+      return $true
+    }
+  } catch {
+    return $true
+  }
+
+  $relative = [IO.Path]::GetRelativePath($root, $fullPath)
+  if ($relative -eq ".") { return $false }
+  if ([IO.Path]::IsPathRooted($relative) -or
+      $relative -eq ".." -or
+      $relative.StartsWith(".." + [IO.Path]::DirectorySeparatorChar) -or
+      $relative.StartsWith(".." + [IO.Path]::AltDirectorySeparatorChar)) {
+    return $true
+  }
+
+  $current = $root
+  foreach ($segment in $relative.Split(
+    @([IO.Path]::DirectorySeparatorChar, [IO.Path]::AltDirectorySeparatorChar),
+    [StringSplitOptions]::RemoveEmptyEntries)) {
+    $current = Join-Path $current $segment
+    try {
+      if (([IO.File]::GetAttributes($current) -band [IO.FileAttributes]::ReparsePoint) -ne 0) {
+        return $true
+      }
+    } catch {
+      return $true
+    }
+  }
+  return $false
+}

eng/evaluation/test_token_failover.py

@@ -2,6 +2,8 @@
 
 import json
 import os
+import re
+import shutil
 import stat
 import subprocess
 import sys
@@ -20,11 +22,26 @@ REPO_ROOT = Path(__file__).resolve().parents[2]
 WORKFLOW = REPO_ROOT / ".github" / "workflows" / "evaluation-run.yml"
 CALLER_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "evaluation.yml"
 TEST_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "evaluation-workflow-tests.yml"
+DASHBOARD_GENERATOR = REPO_ROOT / "eng" / "dashboard" / "generate-benchmark-data.ps1"
+PATH_SAFETY_SCRIPT = REPO_ROOT / "eng" / "evaluation" / "path-safety.ps1"
 STEP_NAME = "Select available Copilot token from pool"
 GIT_BASH = Path(os.environ.get("ProgramFiles", r"C:\Program Files")) / "Git" / "bin" / "bash.exe"
 BASH = str(GIT_BASH) if os.name == "nt" and GIT_BASH.exists() else "bash"
 
 
+def create_symlink_or_skip(
+    test_case: unittest.TestCase,
+    link: Path,
+    target: Path,
+    *,
+    target_is_directory: bool = False,
+) -> None:
+    try:
+        link.symlink_to(target, target_is_directory=target_is_directory)
+    except OSError as error:
+        test_case.skipTest(f"Symlinks are unavailable: {error}")
+
+
 def selection_script() -> str:
     workflow = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
     try:
@@ -63,7 +80,8 @@ class TokenFailoverTests(unittest.TestCase):
         end = discover_script.index("# Validate every entry", start)
         script = (
             "$ErrorActionPreference = 'Stop'\n"
-            "$entries = @(@{name='fixture'; plugin='fixture'; skills_path='plugins/fixture/skills'})\n"
+            "$entries = @(@{name='fixture'; plugin='fixture'; target_kind='skill'; "
+            "skills_path='plugins/fixture/skills'; agents_path=''})\n"
             + discover_script[start:end]
             + "\nConvertTo-Json -InputObject @($entries) -Compress\n"
         )
@@ -541,6 +559,73 @@ esac
             smoke_script,
         )
 
+    def test_path_safety_helper_changes_run_workflow_tests(self) -> None:
+        workflow = yaml.safe_load(TEST_WORKFLOW.read_text(encoding="utf-8"))
+        triggers = workflow.get("on", workflow.get(True))
+        helper_path = "eng/evaluation/path-safety.ps1"
+        for event in ("pull_request", "push"):
+            self.assertEqual(triggers[event]["paths"].count(helper_path), 1)
+
+    def test_manual_dispatch_does_not_execute_pr_path_safety_helper(self) -> None:
+        workflow = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
+        build_script = next(
+            step["run"]
+            for step in workflow["jobs"]["prepare"]["steps"]
+            if step.get("id") == "build"
+        )
+
+        self.assertNotIn('eng/evaluation/path-safety.ps1', build_script)
+        self.assertIn("function Test-PathHasReparsePoint", build_script)
+        self.assertIn("github.workflow_sha", build_script)
+
+    def test_path_safety_helper_rejects_linked_allowed_root(self) -> None:
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp)
+            target = root / "target"
+            target.mkdir()
+            (target / "child.txt").write_text("content", encoding="utf-8")
+            linked_root = root / "linked-root"
+            create_symlink_or_skip(
+                self, linked_root, target, target_is_directory=True)
+
+            quote = lambda path: str(path).replace("'", "''")
+            script = (
+                f". '{quote(PATH_SAFETY_SCRIPT)}'\n"
+                f"Test-PathHasReparsePoint -AllowedRoot '{quote(linked_root)}' "
+                f"-Path '{quote(linked_root)}'\n"
+                f"Test-PathHasReparsePoint -AllowedRoot '{quote(linked_root)}' "
+                f"-Path '{quote(linked_root / 'child.txt')}'\n"
+            )
+            result = subprocess.run(
+                ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", script],
+                capture_output=True,
+                text=True,
+                timeout=30,
+            )
+
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            self.assertEqual(result.stdout.strip().splitlines(), ["True", "True"])
+
+    def test_path_safety_helper_preserves_filesystem_root(self) -> None:
+        with tempfile.TemporaryDirectory() as temp:
+            path = Path(temp)
+            root = Path(path.anchor)
+            quote = lambda value: str(value).replace("'", "''")
+            script = (
+                f". '{quote(PATH_SAFETY_SCRIPT)}'\n"
+                f"Test-PathHasReparsePoint -AllowedRoot '{quote(root)}' "
+                f"-Path '{quote(path)}'\n"
+            )
+            result = subprocess.run(
+                ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", script],
+                capture_output=True,
+                text=True,
+                timeout=30,
+            )
+
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            self.assertEqual(result.stdout.strip(), "False")
+
     def test_adapter_fault_injection_runs_in_pr_ci(self) -> None:
         workflow = yaml.safe_load(TEST_WORKFLOW.read_text(encoding="utf-8"))
         triggers = workflow.get("on", workflow.get(True))
@@ -815,7 +900,7 @@ esac
             run_script.count(
                 '--expected-evals "$RUNNER_TEMP/evaluation-expected-evals.txt"'
             ),
-            2,
+            3,
         )
         self.assertIn(
             'if [ "$PRODUCED" -ne "$EXPECTED_EVAL_COUNT" ]',
@@ -880,8 +965,520 @@ esac
         trusted_adapter = '"$RUNNER_TEMP/trusted-validator-src/eng/vally-adapter/'
         self.assertIn(f"node {trusted_adapter}gen-experiment.mjs", run_script)
         self.assertIn(f"node {trusted_adapter}adapt.mjs", run_script)
+        self.assertIn(f"node {trusted_adapter}adapt-agent-results.mjs", run_script)
+        self.assertIn('"$RUNNER_TEMP/trusted-validator/skill-validator" evaluate', run_script)
+        self.assertIn('rm -f "${AGENT_RESULTS[0]}"', run_script)
+        self.assertGreater(
+            run_script.index('rm -f "${AGENT_RESULTS[0]}"'),
+            run_script.index(f"node {trusted_adapter}adapt-agent-results.mjs"),
+        )
         self.assertNotIn("node eng/vally-adapter/", run_script)
 
+    def test_discovery_creates_first_class_agent_matrix_entries(self) -> None:
+        caller = yaml.safe_load(CALLER_WORKFLOW.read_text(encoding="utf-8"))
+        discover_script = next(
+            step["run"]
+            for step in caller["jobs"]["discover"]["steps"]
+            if "function Get-PluginAgentEntries" in step.get("run", "")
+        )
+        self.assertIn('target_kind = "agent"', discover_script)
+        self.assertIn("$manifest.agents", discover_script)
+        self.assertIn("Resolve-AgentEvalPath", discover_script)
+        self.assertIn("agents_path = $agentPath", discover_script)
+        self.assertIn("eval_path = $evalPath", discover_script)
+        self.assertIn("^plugins/([^/]+)/(?:[^/]+/)*[^/]+\\.agent\\.md$", discover_script)
+        self.assertIn("$changedAgentSourcePlugins", discover_script)
+        self.assertIn("every agent eval in an affected plugin", discover_script)
+
+        runner = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
+        steps = {step.get("name"): step for step in runner["jobs"]["vally-evaluate"]["steps"]}
+        validate = steps["Validate matrix entry"]["run"]
+        self.assertIn("ENTRY_TARGET_KIND", steps["Validate matrix entry"]["env"])
+        self.assertIn("ENTRY_EVAL_PATH", steps["Validate matrix entry"]["env"])
+        self.assertIn("agent_path_re=", validate)
+        self.assertIn("eval_path_re=", validate)
+        self.assertIn('Agent matrix entry has an empty agents_path', validate)
+        self.assertIn('Agent matrix entry has an empty eval_path', validate)
+
+        find = steps["Find eval specs"]["run"]
+        self.assertIn('if [ "$TARGET_KIND" = "agent" ]', find)
+        self.assertIn('EVALS="$EVAL_PATH"', find)
+
+        run = steps["Run vally evaluations"]["run"]
+        self.assertIn('if [ "$TARGET_KIND" = "agent" ]', run)
+        self.assertIn("--verdict-warn-only", run)
+        self.assertIn("--keep-sessions", run)
+
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp)
+            (root / "plugins" / "demo" / "skills" / "skill-a").mkdir(parents=True)
+            (root / "plugins" / "demo" / "custom-agents").mkdir(parents=True)
+            (root / "tests" / "demo" / "skill-a").mkdir(parents=True)
+            (root / "tests" / "demo" / "nested" / "agent.router").mkdir(parents=True)
+            (root / "plugins" / "demo" / "skills" / "skill-a" / "SKILL.md").write_text(
+                "# Skill", encoding="utf-8")
+            (root / "plugins" / "demo" / "custom-agents" / "router.agent.md").write_text(
+                "---\nname: router\ndescription: Routes.\n---\nRoute.", encoding="utf-8")
+            (root / "plugins" / "demo" / "plugin.json").write_text(
+                json.dumps({
+                    "name": "demo",
+                    "version": "1.0.0",
+                    "description": "Demo",
+                    "skills": ["./skills/"],
+                    "agents": ["./custom-agents/router.agent.md"],
+                }),
+                encoding="utf-8",
+            )
+            (root / "tests" / "demo" / "skill-a" / "eval.yaml").write_text(
+                "name: skill-a\nstimuli: []\n", encoding="utf-8")
+            (root / "tests" / "demo" / "nested" / "agent.router" / "eval.yaml").write_text(
+                "name: agent.router\nstimuli: []\n", encoding="utf-8")
+
+            start = discover_script.index("function Get-PluginShardEntries")
+            end = discover_script.index(
+                'if ("${{ needs.gate.outputs.pr_number }}"', start)
+            functions = discover_script[start:end]
+            script = (
+                "$ErrorActionPreference = 'Stop'\n"
+                + f". '{str(PATH_SAFETY_SCRIPT).replace(chr(39), chr(39) * 2)}'\n"
+                + functions
+                + f"\n$root = '{str(root).replace(chr(39), chr(39) * 2)}'\n"
+                + "$entries = @(\n"
+                + "  Get-PluginShardEntries -plugin demo -contentRoot $root\n"
+                + "  Get-PluginAgentEntries -plugin demo -contentRoot $root\n"
+                + ")\n"
+                + "ConvertTo-Json -InputObject @($entries) -Compress\n"
+            )
+            result = subprocess.run(
+                ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", script],
+                capture_output=True, text=True, timeout=30,
+            )
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            entries = json.loads(result.stdout.strip().splitlines()[-1])
+            self.assertEqual(
+                {(entry["target_kind"], entry["name"]) for entry in entries},
+                {("skill", "demo"), ("agent", "demo--agent.router")},
+            )
+            agent_entry = next(entry for entry in entries if entry["target_kind"] == "agent")
+            self.assertEqual(
+                agent_entry["agents_path"],
+                "plugins/demo/custom-agents/router.agent.md",
+            )
+            self.assertEqual(
+                agent_entry["eval_path"],
+                "tests/demo/nested/agent.router/eval.yaml",
+            )
+
+            outside_agent = root / "outside.agent.md"
+            outside_agent.write_text(
+                "---\nname: router\ndescription: External.\n---\nExternal.",
+                encoding="utf-8",
+            )
+            (root / "plugins" / "demo" / "custom-agents" / "router.agent.md").unlink()
+            create_symlink_or_skip(
+                self,
+                root / "plugins" / "demo" / "custom-agents" / "router.agent.md",
+                outside_agent,
+            )
+            unsafe_result = subprocess.run(
+                ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", script],
+                capture_output=True,
+                text=True,
+                timeout=30,
+            )
+            self.assertNotEqual(
+                unsafe_result.returncode,
+                0,
+                unsafe_result.stdout + unsafe_result.stderr,
+            )
+
+    def test_manual_agent_dispatch_resolves_manifest_paths(self) -> None:
+        workflow = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
+        prepare = workflow["jobs"]["prepare"]
+        steps = {step.get("name", step.get("id")): step for step in prepare["steps"]}
+        self.assertIn("Checkout evaluation content", steps)
+        build_script = steps["build"]["run"]
+
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp)
+            agent_dir = root / "plugins" / "demo" / "custom-agents"
+            eval_dir = root / "tests" / "demo" / "nested" / "agent.router"
+            agent_dir.mkdir(parents=True)
+            eval_dir.mkdir(parents=True)
+            (root / "plugins" / "demo" / "plugin.json").write_text(
+                json.dumps({
+                    "name": "demo",
+                    "version": "1.0.0",
+                    "description": "Demo",
+                    "agents": ["./custom-agents/router.agent.md"],
+                }),
+                encoding="utf-8",
+            )
+            (agent_dir / "router.agent.md").write_text(
+                "---\nname: router\ndescription: Routes.\n---\nRoute.",
+                encoding="utf-8",
+            )
+            (eval_dir / "eval.yaml").write_text(
+                "name: agent.router\nstimuli: []\n",
+                encoding="utf-8",
+            )
+            path_safety_dir = root / "eng" / "evaluation"
+            path_safety_dir.mkdir(parents=True)
+            shutil.copy2(PATH_SAFETY_SCRIPT, path_safety_dir / PATH_SAFETY_SCRIPT.name)
+            output_file = root / "github-output.txt"
+            env = dict(
+                os.environ,
+                PLUGIN="demo",
+                SKILL="agent.router",
+                GITHUB_OUTPUT=str(output_file),
+            )
+            result = subprocess.run(
+                ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", build_script],
+                cwd=root,
+                env=env,
+                capture_output=True,
+                text=True,
+                timeout=30,
+            )
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            output_line = output_file.read_text(encoding="utf-8").strip()
+            entries = json.loads(output_line.removeprefix("entries="))
+            self.assertEqual(entries[0]["agents_path"], "plugins/demo/custom-agents/router.agent.md")
+            self.assertEqual(entries[0]["eval_path"], "tests/demo/nested/agent.router/eval.yaml")
+
+            outside_agent = root / "outside.agent.md"
+            outside_agent.write_text(
+                "---\nname: router\ndescription: External.\n---\nExternal.",
+                encoding="utf-8",
+            )
+            (agent_dir / "router.agent.md").unlink()
+            create_symlink_or_skip(
+                self, agent_dir / "router.agent.md", outside_agent)
+            output_file.unlink()
+            unsafe_result = subprocess.run(
+                ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", build_script],
+                cwd=root,
+                env=env,
+                capture_output=True,
+                text=True,
+                timeout=30,
+            )
+            self.assertNotEqual(
+                unsafe_result.returncode,
+                0,
+                unsafe_result.stdout + unsafe_result.stderr,
+            )
+
+    def test_all_pr_discovery_gates_match_direct_agent_sources(self) -> None:
+        caller = yaml.safe_load(CALLER_WORKFLOW.read_text(encoding="utf-8"))
+        discovery_scripts = {
+            job_name: next(
+                step["run"]
+                for step in caller["jobs"][job_name]["steps"]
+                if "$hasSkillChanges = $changedFiles" in step.get("run", "")
+            )
+            for job_name in ("pr-status", "fork-pr-status", "discover")
+        }
+        changed_files = [
+            "plugins/dotnet-test/plugin.json",
+            "plugins/dotnet-test/agents/test-quality-auditor.agent.md",
+            "plugins/dotnet-test/custom-agents/helper.agent.md",
+            "plugins/dotnet-test/skills/test-smell-detection/SKILL.md",
+            "tests/dotnet-test/agent.test-quality-auditor/eval.yaml",
+            "tests/dotnet-test/test-smell-detection/eval.yaml",
+            "plugins/dotnet-test/README.md",
+        ]
+        expected = changed_files[:6]
+
+        for job_name, script in discovery_scripts.items():
+            with self.subTest(job=job_name):
+                match = re.search(
+                    r"\$hasSkillChanges = \$changedFiles \|\s*"
+                    r"Where-Object \{ \$_ -match '([^']+)' \}",
+                    script,
+                )
+                self.assertIsNotNone(match)
+                env = dict(os.environ, DISCOVERY_PATTERN=match.group(1))
+                powershell = (
+                    "$changedFiles = @("
+                    + ",".join(
+                        f"'{path.replace(chr(39), chr(39) * 2)}'"
+                        for path in changed_files
+                    )
+                    + "); "
+                    "$matches = @($changedFiles | "
+                    "Where-Object { $_ -match $env:DISCOVERY_PATTERN }); "
+                    "ConvertTo-Json -InputObject $matches -Compress"
+                )
+                result = subprocess.run(
+                    [
+                        "pwsh",
+                        "-NoLogo",
+                        "-NoProfile",
+                        "-NonInteractive",
+                        "-Command",
+                        powershell,
+                    ],
+                    env=env,
+                    capture_output=True,
+                    text=True,
+                    timeout=30,
+                )
+                self.assertEqual(
+                    result.returncode,
+                    0,
+                    result.stdout + result.stderr,
+                )
+                self.assertEqual(json.loads(result.stdout.strip()), expected)
+
+        matrix_script = discovery_scripts["discover"]
+        self.assertIn("$changedManifestPlugins", matrix_script)
+        self.assertIn(
+            "$changedAgentSourcePlugins + $changedSkillSourcePlugins + $changedManifestPlugins + $changedTestPlugins",
+            matrix_script,
+        )
+        self.assertIn(
+            "every agent eval in an affected plugin",
+            matrix_script,
+        )
+
+    def test_manual_whole_plugin_dispatch_includes_agent_entries(self) -> None:
+        workflow = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
+        build_script = next(
+            step["run"]
+            for step in workflow["jobs"]["prepare"]["steps"]
+            if step.get("id") == "build"
+        )
+
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp)
+            agent_dir = root / "plugins" / "demo" / "custom-agents"
+            eval_dir = root / "tests" / "demo" / "agent.router"
+            agent_dir.mkdir(parents=True)
+            eval_dir.mkdir(parents=True)
+            (root / "plugins" / "demo" / "plugin.json").write_text(
+                json.dumps({
+                    "name": "demo",
+                    "version": "1.0.0",
+                    "description": "Demo",
+                    "agents": ["./custom-agents/"],
+                }),
+                encoding="utf-8",
+            )
+            (agent_dir / "router.agent.md").write_text(
+                "---\nname: router\ndescription: Routes.\n---\nRoute.",
+                encoding="utf-8",
+            )
+            (eval_dir / "eval.yaml").write_text(
+                "name: agent.router\nstimuli: []\n",
+                encoding="utf-8",
+            )
+            path_safety_dir = root / "eng" / "evaluation"
+            path_safety_dir.mkdir(parents=True)
+            shutil.copy2(PATH_SAFETY_SCRIPT, path_safety_dir / PATH_SAFETY_SCRIPT.name)
+            output_file = root / "github-output.txt"
+            env = dict(
+                os.environ,
+                PLUGIN="demo",
+                SKILL="",
+                GITHUB_OUTPUT=str(output_file),
+            )
+
+            result = subprocess.run(
+                ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", build_script],
+                cwd=root,
+                env=env,
+                capture_output=True,
+                text=True,
+                timeout=30,
+            )
+
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            entries = json.loads(
+                output_file.read_text(encoding="utf-8").strip().removeprefix("entries=")
+            )
+            self.assertEqual(
+                {(entry["target_kind"], entry["name"]) for entry in entries},
+                {("skill", "demo"), ("agent", "demo--agent.router")},
+            )
+
+    def test_dashboard_preserves_agent_identity_and_delegation(self) -> None:
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp)
+            results = root / "results.json"
+            output = root / "out"
+            results.write_text(json.dumps({
+                "schemaVersion": 5,
+                "model": "executor",
+                "judgeModel": "judge",
+                "evalFile": "tests/demo/nested/agent.router/eval.yaml",
+                "verdicts": [{
+                    "skillName": "agent.router",
+                    "skillPath": "plugins/demo/custom-agents/router.agent.md",
+                    "skillKind": "agent",
+                    "state": "VALID_PASS",
+                    "passed": True,
+                    "reason": "credible preference improvement",
+                    "signTest": {
+                        "wins": 5, "ties": 0, "losses": 0,
+                        "discordant": 5, "direction": "better",
+                        "pValue": 0.03125, "alpha": 0.05,
+                    },
+                    "netWin": 1,
+                    "practicalSignificance": {"minimum": 0.2},
+                    "scenarios": [{
+                        "scenarioName": "routes work",
+                        "expectActivation": True,
+                        "preferenceGateEligible": True,
+                        "agentActivationIsolated": {
+                            "activated": True,
+                            "invokedAgents": ["router", "helper"],
+                            "delegatedAgents": ["helper"],
+                        },
+                        "agentActivationPlugin": {
+                            "activated": True,
+                            "invokedAgents": ["router", "helper"],
+                            "delegatedAgents": ["helper"],
+                        },
+                        "skillActivationIsolated": {
+                            "activated": False,
+                            "detectedSkills": ["routing-skill"],
+                        },
+                        "baseline": {
+                            "judgeResult": {"overallScore": 2},
+                            "metrics": {"wallTimeMs": 100, "tokenEstimate": 20},
+                        },
+                        "skilledIsolated": {
+                            "judgeResult": {"overallScore": 4},
+                            "metrics": {
+                                "wallTimeMs": 200,
+                                "tokenEstimate": 30,
+                                "taskCompleted": True,
+                                "toolCallBreakdown": {"skill": 1},
+                            },
+                        },
+                        "skilledPlugin": {
+                            "judgeResult": {"overallScore": 4},
+                            "metrics": {
+                                "wallTimeMs": 220,
+                                "tokenEstimate": 35,
+                                "taskCompleted": True,
+                                "toolCallBreakdown": {"skill": 1, "agent": 1},
+                            },
+                        },
+                        "trials": [{
+                            "winner": "treatment",
+                            "errored": False,
+                            "baselinePassed": False,
+                            "treatmentPassed": True,
+                            "evidence": "The agent routed correctly.",
+                        }],
+                    }],
+                }],
+            }), encoding="utf-8")
+
+            result = subprocess.run([
+                "pwsh", "-NoLogo", "-NoProfile", "-NonInteractive",
+                "-File", str(DASHBOARD_GENERATOR),
+                "-ResultsFile", str(results),
+                "-PluginName", "demo",
+                "-OutputDir", str(output),
+                "-CommitJson", json.dumps({
+                    "id": "abcdef1234567890",
+                    "url": "https://github.com/dotnet/skills/commit/abcdef1234567890",
+                }),
+            ], capture_output=True, text=True, timeout=30)
+
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            dashboard = json.loads((output / "demo.json").read_text(encoding="utf-8-sig"))
+            evidence = dashboard["entries"]["Quality"][-1]["verdictEvidence"][0]
+            self.assertEqual(evidence["skillKind"], "agent")
+            scenario = evidence["activationScenarios"][0]
+            self.assertEqual(scenario["isolated"], "activated")
+            self.assertEqual(scenario["delegatedAgents"], ["helper"])
+            self.assertEqual(scenario["invokedSkills"], ["routing-skill"])
+            self.assertEqual(scenario["isolatedTools"], ["skill"])
+            self.assertTrue(scenario["isolatedCompleted"])
+            skill_value = dashboard["entries"]["SkillValue"][-1]["skills"][0]
+            self.assertEqual(skill_value["activationExpected"], 1)
+            self.assertEqual(skill_value["activationFired"], 1)
+            agent_link = next(
+                link for link in evidence["links"] if link["label"] == "Agent source"
+            )
+            self.assertIn(
+                "/plugins/demo/custom-agents/router.agent.md",
+                agent_link["url"],
+            )
+            eval_link = next(
+                link for link in evidence["links"] if link["label"] == "Eval source"
+            )
+            self.assertIn(
+                "/tests/demo/nested/agent.router/eval.yaml",
+                eval_link["url"],
+            )
+
+    def test_dashboard_agent_evidence_allows_missing_plugin_role(self) -> None:
+        with tempfile.TemporaryDirectory() as temp:
+            root = Path(temp)
+            results = root / "results.json"
+            output = root / "out"
+            results.write_text(json.dumps({
+                "schemaVersion": 5,
+                "model": "executor",
+                "judgeModel": "judge",
+                "verdicts": [{
+                    "skillName": "agent.router",
+                    "skillKind": "agent",
+                    "state": "INVALID_INCONCLUSIVE",
+                    "passed": False,
+                    "reason": "plugin evidence missing",
+                    "scenarios": [{
+                        "scenarioName": "routes work",
+                        "expectActivation": True,
+                        "agentActivationIsolated": {
+                            "activated": True,
+                            "invokedAgents": None,
+                            "delegatedAgents": None,
+                        },
+                        "skillActivationIsolated": {
+                            "activated": False,
+                            "detectedSkills": None,
+                        },
+                        "baseline": {
+                            "judgeResult": {"overallScore": 2},
+                            "metrics": {"wallTimeMs": 100, "tokenEstimate": 20},
+                        },
+                        "skilledIsolated": {
+                            "judgeResult": {"overallScore": 4},
+                            "metrics": {
+                                "wallTimeMs": 200,
+                                "tokenEstimate": 30,
+                                "taskCompleted": True,
+                                "toolCallBreakdown": {"skill": 1},
+                            },
+                        },
+                    }],
+                }],
+            }), encoding="utf-8")
+
+            result = subprocess.run([
+                "pwsh", "-NoLogo", "-NoProfile", "-NonInteractive",
+                "-File", str(DASHBOARD_GENERATOR),
+                "-ResultsFile", str(results),
+                "-PluginName", "demo",
+                "-OutputDir", str(output),
+            ], capture_output=True, text=True, timeout=30)
+
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            dashboard = json.loads((output / "demo.json").read_text(encoding="utf-8-sig"))
+            evidence = dashboard["entries"]["Quality"][-1]["verdictEvidence"][0]
+            scenario = evidence["activationScenarios"][0]
+            self.assertEqual(scenario["invokedAgents"], [])
+            self.assertEqual(scenario["delegatedAgents"], [])
+            self.assertEqual(scenario["invokedSkills"], [])
+            self.assertEqual(scenario["pluginTools"], [])
+            self.assertIsNone(scenario["pluginCompleted"])
+
     def test_result_consumers_use_explicit_verdict_states(self) -> None:
         workflow = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
         steps = workflow["jobs"]["vally-evaluate"]["steps"]

eng/skill-validator/src/Evaluate/AgentRunner.cs

@@ -1,5 +1,6 @@
 using System.Collections.Concurrent;
 using System.Diagnostics;
+using System.Text;
 using System.Text.Json;
 using System.Text.Json.Nodes;
 using SkillValidator.Shared;
@@ -20,7 +21,46 @@ public sealed record RunOptions(
     string? SessionsDir = null,
     string? SessionId = null,
     AgentInfo? Agent = null,
-    IReadOnlyList<AgentInfo>? AdditionalAgents = null);
+    IReadOnlyList<AgentInfo>? AdditionalAgents = null,
+    bool SelectAgentAsPrimary = true);
+
+internal sealed class RunEventBuffer
+{
+    private readonly Lock _sync = new();
+    private readonly List<AgentEvent> _events = [];
+    private readonly StringBuilder _agentOutput = new();
+
+    internal void Record(string type, Action<AgentEvent, StringBuilder> populate)
+    {
+        lock (_sync)
+        {
+            var agentEvent = new AgentEvent(
+                type,
+                DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
+                []);
+            populate(agentEvent, _agentOutput);
+            _events.Add(agentEvent);
+        }
+    }
+
+    internal void Add(AgentEvent agentEvent)
+    {
+        lock (_sync)
+            _events.Add(agentEvent);
+    }
+
+    internal bool ContainsType(string type)
+    {
+        lock (_sync)
+            return _events.Any(agentEvent => agentEvent.Type == type);
+    }
+
+    internal (List<AgentEvent> Events, string AgentOutput) Snapshot()
+    {
+        lock (_sync)
+            return ([.. _events], _agentOutput.ToString());
+    }
+}
 
 public static class AgentRunner
 {
@@ -130,14 +170,15 @@ public static class AgentRunner
 
     /// <summary>
     /// Extracts a file path from PreToolUseHookInput.ToolArgs for permission sandboxing.
-    /// Checks common arg keys: path, fileName, fullCommandText.
+    /// Checks common path arg keys. Shell command text is not itself a path; shell
+    /// paths are checked from PermissionRequestShell.PossiblePaths instead.
     /// </summary>
     internal static string? ExtractPathFromToolArgs(PreToolUseHookInput input)
     {
         if (input.ToolArgs is not JsonElement args || args.ValueKind != JsonValueKind.Object)
             return null;
 
-        foreach (var key in new[] { "path", "fileName", "fullCommandText" })
+        foreach (var key in new[] { "path", "fileName" })
         {
             if (args.TryGetProperty(key, out var val) && val.ValueKind == JsonValueKind.String)
                 return val.GetString();
@@ -146,6 +187,46 @@ public static class AgentRunner
         return null;
     }
 
+    internal static bool IsShellTool(string? toolName) =>
+        toolName is not null &&
+        (toolName.Equals("bash", StringComparison.OrdinalIgnoreCase) ||
+         toolName.Equals("powershell", StringComparison.OrdinalIgnoreCase) ||
+         toolName.Equals("local_shell", StringComparison.OrdinalIgnoreCase));
+
+    internal static bool CheckPermissions(IEnumerable<string>? reqPaths, string workDir, string? skillPath, Action<string>? log, string? runLabel = null, string? pluginRoot = null, IReadOnlyList<string>? additionalAllowedDirs = null)
+    {
+        return reqPaths is null || reqPaths.All(path =>
+            CheckPermission(path, workDir, skillPath, log, runLabel, pluginRoot, additionalAllowedDirs));
+    }
+
+    internal static bool CheckShellPermission(
+        PermissionRequestShell request,
+        string workDir,
+        string? skillPath,
+        Action<string>? log,
+        string? runLabel = null,
+        string? pluginRoot = null,
+        IReadOnlyList<string>? additionalAllowedDirs = null)
+    {
+        var hasUrl = request.PossibleUrls is { Length: > 0 }
+            || request.FullCommandText?.Contains("://", StringComparison.OrdinalIgnoreCase) == true;
+        if (hasUrl)
+        {
+            var labelSuffix = runLabel is not null ? $" ({runLabel})" : "";
+            log?.Invoke($"      ❌ Denying shell permission request with network URL{labelSuffix}");
+            return false;
+        }
+
+        return CheckPermissions(
+            request.PossiblePaths,
+            workDir,
+            skillPath,
+            log,
+            runLabel,
+            pluginRoot,
+            additionalAllowedDirs);
+    }
+
     public static bool CheckPermission(string? reqPath, string workDir, string? skillPath, Action<string>? log, string? runLabel = null, string? pluginRoot = null, IReadOnlyList<string>? additionalAllowedDirs = null)
     {
         var labelSuffix = runLabel is not null ? $" ({runLabel})" : "";
@@ -205,8 +286,13 @@ public static class AgentRunner
             string normalizedDir = Path.EndsInDirectorySeparator(dir)
                 ? dir
                 : dir + Path.DirectorySeparatorChar;
-            return resolved.Equals(normalizedDir, comparison) ||
-                   resolved.StartsWith(normalizedDir, comparison);
+            var lexicallyContained = resolved.Equals(normalizedDir, comparison)
+                || resolved.StartsWith(normalizedDir, comparison);
+            return lexicallyContained
+                && !PathSafety.ContainsReparsePoint(
+                    Path.TrimEndingDirectorySeparator(dir),
+                    Path.TrimEndingDirectorySeparator(resolved),
+                    missingPathIsUnsafe: false);
         });
 
         if (!anyAllowed)
@@ -347,7 +433,7 @@ public static class AgentRunner
         string[] skillDirs;
         if (pluginRoot is not null)
         {
-            skillDirs = ResolvePluginSkillDirectories(pluginRoot);
+            skillDirs = await StagePluginSkillDirectories(pluginRoot);
         }
         else if (skill is not null)
         {
@@ -406,7 +492,7 @@ public static class AgentRunner
         // In plugin mode: register all plugin agents. In isolated mode: register
         // only the target agent (+ additional declared agents). In baseline: none.
         List<CustomAgentConfig>? customAgents = null;
-        if (pluginRoot is not null)
+        if (pluginRoot is not null && agent is not null)
         {
             // Plugin run: discover and register all agents in the plugin
             var pluginAgents = await AgentDiscovery.DiscoverAgentsInPlugin(pluginRoot);
@@ -420,12 +506,11 @@ public static class AgentRunner
         else if (agent is not null)
         {
             // Isolated agent run: register only the target agent + declared dependencies
-            customAgents = [BuildCustomAgentConfig(agent)];
-            if (additionalAgents is { Count: > 0 })
-            {
-                foreach (var dep in additionalAgents)
-                    customAgents.Add(BuildCustomAgentConfig(dep));
-            }
+            customAgents = new[] { agent }
+                .Concat(additionalAgents ?? [])
+                .DistinctBy(item => item.Name, StringComparer.OrdinalIgnoreCase)
+                .Select(BuildCustomAgentConfig)
+                .ToList();
             if (verbose)
                 log?.Invoke($"      🤖 Registered agent(s) (isolated): {string.Join(", ", customAgents.Select(a => a.Name))}");
         }
@@ -437,6 +522,11 @@ public static class AgentRunner
                 log?.Invoke($"      🤖 Registered additional agent(s): {string.Join(", ", customAgents.Select(a => a.Name))}");
         }
 
+        var runLabel =
+            agent is not null
+                ? (pluginRoot is not null ? "agent-plugin" : "agent-isolated")
+                : (skill is not null ? "skilled" : "baseline");
+
         return new SessionConfig
         {
             Model = model,
@@ -453,20 +543,37 @@ public static class AgentRunner
             CreateSessionFsProvider = _ => new LocalSessionFsHandler(configDir),
             OnPermissionRequest = (request, _) =>
             {
-                // PermissionRequest carries per-kind data (e.g. Read.Path,
-                // Write.FileName, Shell.FullCommandText/PossiblePaths), but we
-                // don't use it here: permission sandboxing is enforced via
-                // Hooks.OnPreToolUse instead, so this handler approves all.
+                if (request is PermissionRequestShell shellRequest)
+                {
+                    var allowed = CheckShellPermission(
+                        shellRequest,
+                        workDir,
+                        effectiveSkillPath,
+                        verbose ? log : null,
+                        runLabel,
+                        pluginRoot,
+                        additionalAllowedDirs);
+                    return Task.FromResult(
+                        allowed
+                            ? GitHub.Copilot.Rpc.PermissionDecision.ApproveOnce()
+                            : GitHub.Copilot.Rpc.PermissionDecision.Reject("Path outside allowed directories"));
+                }
+
                 return Task.FromResult(GitHub.Copilot.Rpc.PermissionDecision.ApproveOnce());
             },
             Hooks = new SessionHooks
             {
                 OnPreToolUse = (input, invocation) =>
                 {
-                    var runLabel =
-                        agent is not null
-                            ? (pluginRoot is not null ? "agent-plugin" : "agent-isolated")
-                            : (skill is not null ? "skilled" : "baseline");
+                    if (IsShellTool(input.ToolName))
+                    {
+                        return Task.FromResult<PreToolUseHookOutput?>(new PreToolUseHookOutput
+                        {
+                            PermissionDecision = "ask",
+                            PermissionDecisionReason = "Validate shell command paths",
+                        });
+                    }
+
                     var reqPath = ExtractPathFromToolArgs(input);
                     var allowed = CheckPermission(reqPath, workDir, effectiveSkillPath, verbose ? log : null, runLabel, pluginRoot, additionalAllowedDirs);
                     return Task.FromResult<PreToolUseHookOutput?>(new PreToolUseHookOutput
@@ -526,6 +633,28 @@ public static class AgentRunner
         return dirs.ToArray();
     }
 
+    private static async Task<string[]> StagePluginSkillDirectories(string pluginRoot)
+    {
+        var stagedRoots = new List<string>();
+        foreach (var sourceRoot in ResolvePluginSkillDirectories(pluginRoot))
+        {
+            var skills = await SkillDiscovery.DiscoverSkills(sourceRoot, pluginRoot);
+            if (skills.Count == 0)
+                continue;
+
+            var stageDir = Path.Combine(Path.GetTempPath(), $"sv-plugin-{Guid.NewGuid():N}");
+            Directory.CreateDirectory(stageDir);
+            _workDirs.Add(stageDir);
+            foreach (var discoveredSkill in skills)
+            {
+                var stagedSkillDir = Path.Combine(stageDir, Path.GetFileName(discoveredSkill.Path));
+                CopyDirectory(discoveredSkill.Path, stagedSkillDir);
+            }
+            stagedRoots.Add(stageDir);
+        }
+        return stagedRoots.ToArray();
+    }
+
     public static async Task<RunMetrics> RunAgent(RunOptions options, CancellationToken cancellationToken = default)
     {
         // Validate mutual exclusivity
@@ -554,8 +683,7 @@ public static class AgentRunner
             write($"      📂 Work dir: {workDir} ({(options.Skill is not null ? "skilled" : "baseline")})");
         }
 
-        var events = new List<AgentEvent>();
-        string agentOutput = "";
+        var eventBuffer = new RunEventBuffer();
         var startTime = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
         bool timedOut = false;
 
@@ -581,115 +709,115 @@ public static class AgentRunner
             // from the agent selection is captured in the events list.
             session.On<SessionEvent>(evt =>
             {
-                var agentEvent = new AgentEvent(
-                    evt.Type,
-                    DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
-                    []);
-
-                // Copy known event data
-                switch (evt)
+                eventBuffer.Record(evt.Type, (agentEvent, agentOutput) =>
                 {
-                    case AssistantMessageDeltaEvent delta:
-                        agentEvent.Data["deltaContent"] = JsonValue.Create(delta.Data.DeltaContent);
-                        agentOutput += delta.Data.DeltaContent ?? "";
-                        break;
-                    case AssistantMessageEvent msg:
-                        agentEvent.Data["content"] = JsonValue.Create(msg.Data.Content);
-                        if (!string.IsNullOrEmpty(msg.Data.Content))
-                            agentOutput = msg.Data.Content;
-                        break;
-                    case ToolExecutionStartEvent toolStart:
-                        agentEvent.Data["toolName"] = JsonValue.Create(toolStart.Data.ToolName);
-                        agentEvent.Data["arguments"] = JsonValue.Create(toolStart.Data.Arguments?.ToString());
-                        if (options.Verbose)
-                        {
-                            var write = options.Log ?? (m => Console.Error.WriteLine(m));
-                            write($"      🔧 {toolStart.Data.ToolName}");
-                        }
-                        break;
-                    case ToolExecutionCompleteEvent toolComplete:
-                        agentEvent.Data["success"] = JsonValue.Create(toolComplete.Data.Success.ToString());
-                        agentEvent.Data["result"] = JsonValue.Create(toolComplete.Data.Result?.Content ?? toolComplete.Data.Error?.Message ?? "");
-                        break;
-                    case SkillInvokedEvent skillInvoked:
-                        agentEvent.Data["name"] = JsonValue.Create(skillInvoked.Data.Name);
-                        agentEvent.Data["path"] = JsonValue.Create(skillInvoked.Data.Path);
-                        if (skillInvoked.Data.AllowedTools is { } allowedTools)
-                        {
-                            var arr = new JsonArray();
-                            foreach (var tool in allowedTools)
-                                arr.Add((JsonNode?)JsonValue.Create(tool));
-                            agentEvent.Data["allowedTools"] = arr;
-                        }
-                        if (options.Verbose)
-                        {
-                            var write = options.Log ?? (m => Console.Error.WriteLine(m));
-                            write($"      📘 Skill invoked: {skillInvoked.Data.Name}");
-                        }
-                        break;
-                    case SubagentStartedEvent subagentStarted:
-                        agentEvent.Data["agentName"] = JsonValue.Create(subagentStarted.Data.AgentName);
-                        agentEvent.Data["agentDisplayName"] = JsonValue.Create(subagentStarted.Data.AgentDisplayName);
-                        agentEvent.Data["agentDescription"] = JsonValue.Create(subagentStarted.Data.AgentDescription);
-                        agentEvent.Data["toolCallId"] = JsonValue.Create(subagentStarted.Data.ToolCallId);
-                        if (options.Verbose)
-                        {
-                            var write = options.Log ?? (m => Console.Error.WriteLine(m));
-                            write($"      🤖 Subagent started: {subagentStarted.Data.AgentName}");
-                        }
-                        break;
-                    case SubagentCompletedEvent subagentCompleted:
-                        agentEvent.Data["agentName"] = JsonValue.Create(subagentCompleted.Data.AgentName);
-                        agentEvent.Data["agentDisplayName"] = JsonValue.Create(subagentCompleted.Data.AgentDisplayName);
-                        agentEvent.Data["toolCallId"] = JsonValue.Create(subagentCompleted.Data.ToolCallId);
-                        if (options.Verbose)
-                        {
-                            var write = options.Log ?? (m => Console.Error.WriteLine(m));
-                            write($"      ✅ Subagent completed: {subagentCompleted.Data.AgentName}");
-                        }
-                        break;
-                    case SubagentFailedEvent subagentFailed:
-                        agentEvent.Data["agentName"] = JsonValue.Create(subagentFailed.Data.AgentName);
-                        agentEvent.Data["agentDisplayName"] = JsonValue.Create(subagentFailed.Data.AgentDisplayName);
-                        agentEvent.Data["toolCallId"] = JsonValue.Create(subagentFailed.Data.ToolCallId);
-                        agentEvent.Data["error"] = JsonValue.Create(subagentFailed.Data.Error);
-                        if (options.Verbose)
-                        {
-                            var write = options.Log ?? (m => Console.Error.WriteLine(m));
-                            write($"      ❌ Subagent failed: {subagentFailed.Data.AgentName}");
-                        }
-                        break;
-                    case SubagentSelectedEvent subagentSelected:
-                        agentEvent.Data["agentName"] = JsonValue.Create(subagentSelected.Data.AgentName);
-                        agentEvent.Data["agentDisplayName"] = JsonValue.Create(subagentSelected.Data.AgentDisplayName);
-                        break;
-                    case SubagentDeselectedEvent:
-                        break;
-                    case AssistantUsageEvent usage:
-                        agentEvent.Data["inputTokens"] = JsonValue.Create(usage.Data.InputTokens);
-                        agentEvent.Data["outputTokens"] = JsonValue.Create(usage.Data.OutputTokens);
-                        agentEvent.Data["cacheReadTokens"] = JsonValue.Create(usage.Data.CacheReadTokens);
-                        agentEvent.Data["cacheWriteTokens"] = JsonValue.Create(usage.Data.CacheWriteTokens);
-                        agentEvent.Data["model"] = JsonValue.Create(usage.Data.Model);
-                        break;
-                    case UserMessageEvent userMsg:
-                        agentEvent.Data["content"] = JsonValue.Create(userMsg.Data.Content);
-                        break;
-                    case SessionIdleEvent:
-                        done.TrySetResult();
-                        break;
-                    case SessionErrorEvent err:
-                        agentEvent.Data["message"] = JsonValue.Create(err.Data.Message);
-                        done.TrySetException(new InvalidOperationException(err.Data.Message ?? "Session error"));
-                        break;
-                }
-
-                events.Add(agentEvent);
+                    // Copy known event data
+                    switch (evt)
+                    {
+                        case AssistantMessageDeltaEvent delta:
+                            agentEvent.Data["deltaContent"] = JsonValue.Create(delta.Data.DeltaContent);
+                            agentOutput.Append(delta.Data.DeltaContent);
+                            break;
+                        case AssistantMessageEvent msg:
+                            agentEvent.Data["content"] = JsonValue.Create(msg.Data.Content);
+                            if (!string.IsNullOrEmpty(msg.Data.Content))
+                            {
+                                agentOutput.Clear();
+                                agentOutput.Append(msg.Data.Content);
+                            }
+                            break;
+                        case ToolExecutionStartEvent toolStart:
+                            agentEvent.Data["toolName"] = JsonValue.Create(toolStart.Data.ToolName);
+                            agentEvent.Data["arguments"] = JsonValue.Create(toolStart.Data.Arguments?.ToString());
+                            if (options.Verbose)
+                            {
+                                var write = options.Log ?? (m => Console.Error.WriteLine(m));
+                                write($"      🔧 {toolStart.Data.ToolName}");
+                            }
+                            break;
+                        case ToolExecutionCompleteEvent toolComplete:
+                            agentEvent.Data["success"] = JsonValue.Create(toolComplete.Data.Success);
+                            agentEvent.Data["result"] = JsonValue.Create(toolComplete.Data.Result?.Content ?? toolComplete.Data.Error?.Message ?? "");
+                            break;
+                        case SkillInvokedEvent skillInvoked:
+                            agentEvent.Data["name"] = JsonValue.Create(skillInvoked.Data.Name);
+                            agentEvent.Data["path"] = JsonValue.Create(skillInvoked.Data.Path);
+                            if (skillInvoked.Data.AllowedTools is { } allowedTools)
+                            {
+                                var arr = new JsonArray();
+                                foreach (var tool in allowedTools)
+                                    arr.Add((JsonNode?)JsonValue.Create(tool));
+                                agentEvent.Data["allowedTools"] = arr;
+                            }
+                            if (options.Verbose)
+                            {
+                                var write = options.Log ?? (m => Console.Error.WriteLine(m));
+                                write($"      📘 Skill invoked: {skillInvoked.Data.Name}");
+                            }
+                            break;
+                        case SubagentStartedEvent subagentStarted:
+                            agentEvent.Data["agentName"] = JsonValue.Create(subagentStarted.Data.AgentName);
+                            agentEvent.Data["agentDisplayName"] = JsonValue.Create(subagentStarted.Data.AgentDisplayName);
+                            agentEvent.Data["agentDescription"] = JsonValue.Create(subagentStarted.Data.AgentDescription);
+                            agentEvent.Data["toolCallId"] = JsonValue.Create(subagentStarted.Data.ToolCallId);
+                            if (options.Verbose)
+                            {
+                                var write = options.Log ?? (m => Console.Error.WriteLine(m));
+                                write($"      🤖 Subagent started: {subagentStarted.Data.AgentName}");
+                            }
+                            break;
+                        case SubagentCompletedEvent subagentCompleted:
+                            agentEvent.Data["agentName"] = JsonValue.Create(subagentCompleted.Data.AgentName);
+                            agentEvent.Data["agentDisplayName"] = JsonValue.Create(subagentCompleted.Data.AgentDisplayName);
+                            agentEvent.Data["toolCallId"] = JsonValue.Create(subagentCompleted.Data.ToolCallId);
+                            if (options.Verbose)
+                            {
+                                var write = options.Log ?? (m => Console.Error.WriteLine(m));
+                                write($"      ✅ Subagent completed: {subagentCompleted.Data.AgentName}");
+                            }
+                            break;
+                        case SubagentFailedEvent subagentFailed:
+                            agentEvent.Data["agentName"] = JsonValue.Create(subagentFailed.Data.AgentName);
+                            agentEvent.Data["agentDisplayName"] = JsonValue.Create(subagentFailed.Data.AgentDisplayName);
+                            agentEvent.Data["toolCallId"] = JsonValue.Create(subagentFailed.Data.ToolCallId);
+                            agentEvent.Data["error"] = JsonValue.Create(subagentFailed.Data.Error);
+                            if (options.Verbose)
+                            {
+                                var write = options.Log ?? (m => Console.Error.WriteLine(m));
+                                write($"      ❌ Subagent failed: {subagentFailed.Data.AgentName}");
+                            }
+                            break;
+                        case SubagentSelectedEvent subagentSelected:
+                            agentEvent.Data["agentName"] = JsonValue.Create(subagentSelected.Data.AgentName);
+                            agentEvent.Data["agentDisplayName"] = JsonValue.Create(subagentSelected.Data.AgentDisplayName);
+                            break;
+                        case SubagentDeselectedEvent:
+                            break;
+                        case AssistantUsageEvent usage:
+                            agentEvent.Data["inputTokens"] = JsonValue.Create(usage.Data.InputTokens);
+                            agentEvent.Data["outputTokens"] = JsonValue.Create(usage.Data.OutputTokens);
+                            agentEvent.Data["cacheReadTokens"] = JsonValue.Create(usage.Data.CacheReadTokens);
+                            agentEvent.Data["cacheWriteTokens"] = JsonValue.Create(usage.Data.CacheWriteTokens);
+                            agentEvent.Data["model"] = JsonValue.Create(usage.Data.Model);
+                            break;
+                        case UserMessageEvent userMsg:
+                            agentEvent.Data["content"] = JsonValue.Create(userMsg.Data.Content);
+                            break;
+                        case SessionIdleEvent:
+                            done.TrySetResult();
+                            break;
+                        case SessionErrorEvent err:
+                            agentEvent.Data["message"] = JsonValue.Create(err.Data.Message);
+                            done.TrySetException(new InvalidOperationException(err.Data.Message ?? "Session error"));
+                            break;
+                    }
+                });
             });
 
-            // For agent evaluation: explicitly select the agent as primary persona.
-            // Must happen after CreateSessionAsync and event handler setup, before SendAsync.
-            if (options.Agent is not null)
+            // Legacy callers may explicitly select the target agent as the primary
+            // persona. The first-class CI agent lane leaves the default parent
+            // selected so target activation and delegation remain observable.
+            if (options.Agent is not null && options.SelectAgentAsPrimary)
             {
                 await session.Rpc.Agent.SelectAsync(options.Agent.Name);
                 if (options.Verbose)
@@ -705,7 +833,7 @@ public static class AgentRunner
         catch (TimeoutException te)
         {
             timedOut = true;
-            events.Add(new AgentEvent(
+            eventBuffer.Add(new AgentEvent(
                 "runner.error",
                 DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
                 new Dictionary<string, JsonNode?> { ["message"] = JsonValue.Create(te.ToString()) }));
@@ -729,15 +857,15 @@ public static class AgentRunner
                 || msg.Contains("timed out", StringComparison.OrdinalIgnoreCase))
             {
                 // Timeout: record a dedicated event (the timer fired, no session.error exists)
-                events.Add(new AgentEvent(
+                eventBuffer.Add(new AgentEvent(
                     "runner.timeout",
                     DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
                     new Dictionary<string, JsonNode?> { ["message"] = JsonValue.Create(msg) }));
             }
-            else if (!events.Any(e => e.Type == "session.error"))
+            else if (!eventBuffer.ContainsType("session.error"))
             {
                 // Only add runner.error when there isn't already a session.error event
-                events.Add(new AgentEvent(
+                eventBuffer.Add(new AgentEvent(
                     "runner.error",
                     DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
                     new Dictionary<string, JsonNode?> { ["message"] = JsonValue.Create(msg) }));
@@ -745,12 +873,13 @@ public static class AgentRunner
         }
 
         var wallTimeMs = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() - startTime;
+        var (events, agentOutput) = eventBuffer.Snapshot();
         var metrics = MetricsCollector.CollectMetrics(events, agentOutput, wallTimeMs, workDir);
         metrics.TimedOut = timedOut;
         return metrics;
     }
 
-    private static async Task<string> SetupWorkDir(EvalScenario scenario, string? skillPath, string? evalPath)
+    internal static async Task<string> SetupWorkDir(EvalScenario scenario, string? skillPath, string? evalPath)
     {
         var workDir = Path.Combine(Path.GetTempPath(), $"sv-{Guid.NewGuid():N}");
         Directory.CreateDirectory(workDir);
@@ -763,6 +892,21 @@ public static class AgentRunner
             foreach (var entry in new DirectoryInfo(evalDir).EnumerateFileSystemInfos())
             {
                 if (entry.Name == "eval.yaml") continue;
+                FileAttributes attributes;
+                try
+                {
+                    attributes = entry.Attributes;
+                }
+                catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
+                {
+                    Console.Error.WriteLine($"Unable to inspect setup entry, skipping: {entry.FullName}");
+                    continue;
+                }
+                if ((attributes & FileAttributes.ReparsePoint) != 0)
+                {
+                    Console.Error.WriteLine($"Setup entry is a symbolic link or reparse point, skipping: {entry.FullName}");
+                    continue;
+                }
                 var dest = Path.Combine(workDir, entry.Name);
                 if (entry is DirectoryInfo dir)
                     CopyDirectory(dir.FullName, dest);
@@ -807,7 +951,15 @@ public static class AgentRunner
                     {
                         continue;
                     }
-                    File.Copy(resolvedSource, targetPath, true);
+                    if (Directory.Exists(resolvedSource))
+                    {
+                        Directory.CreateDirectory(targetPath);
+                        CopyDirectory(resolvedSource, targetPath);
+                    }
+                    else
+                    {
+                        File.Copy(resolvedSource, targetPath, true);
+                    }
                 }
             }
         }
@@ -884,17 +1036,47 @@ public static class AgentRunner
 
         var canonicalBaseDir = Path.TrimEndingDirectorySeparator(Path.GetFullPath(baseDir));
         var sourcePath = Path.GetFullPath(Path.Combine(baseDir, source));
-        // Prevent path traversal: source must stay inside the base directory
-        if (!sourcePath.StartsWith(canonicalBaseDir + Path.DirectorySeparatorChar, pathComparison)
-            && !sourcePath.Equals(canonicalBaseDir, pathComparison))
+        var allowedRoot = evalPath is null
+            ? canonicalBaseDir
+            : FindRepositoryRoot(canonicalBaseDir) ?? canonicalBaseDir;
+        var normalizedAllowedRoot = Path.TrimEndingDirectorySeparator(Path.GetFullPath(allowedRoot));
+        // Vally fixture paths are relative to eval.yaml and may intentionally
+        // reference a sibling eval's shared fixtures. Keep them inside the
+        // repository root; standalone legacy evals remain confined to their
+        // own eval/skill directory when no repository root can be identified.
+        if (!sourcePath.StartsWith(normalizedAllowedRoot + Path.DirectorySeparatorChar, pathComparison)
+            && !sourcePath.Equals(normalizedAllowedRoot, pathComparison))
         {
-            Console.Error.WriteLine($"Setup file source escapes base directory, skipping: {source}");
+            Console.Error.WriteLine($"Setup file source escapes the allowed repository directory, skipping: {source}");
+            return null;
+        }
+        if (PathSafety.ContainsReparsePoint(
+            normalizedAllowedRoot,
+            sourcePath,
+            missingPathIsUnsafe: false))
+        {
+            Console.Error.WriteLine($"Setup file source contains a symbolic link or reparse point, skipping: {source}");
             return null;
         }
 
         return sourcePath;
     }
 
+    private static string? FindRepositoryRoot(string startDirectory)
+    {
+        var current = new DirectoryInfo(startDirectory);
+        while (current is not null)
+        {
+            if (Directory.Exists(Path.Combine(current.FullName, "plugins"))
+                && Directory.Exists(Path.Combine(current.FullName, "tests")))
+            {
+                return current.FullName;
+            }
+            current = current.Parent;
+        }
+        return null;
+    }
+
     private static readonly string[] SensitiveEnvKeys =
     [
         "GITHUB_TOKEN",
@@ -1035,6 +1217,18 @@ public static class AgentRunner
     /// </summary>
     private static void CopyDirectory(string source, string destination)
     {
+        FileAttributes attributes;
+        try
+        {
+            attributes = File.GetAttributes(source);
+        }
+        catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
+        {
+            throw new IOException($"Unable to inspect source directory '{source}'.", ex);
+        }
+        if ((attributes & FileAttributes.ReparsePoint) != 0)
+            throw new IOException($"Refusing to copy symbolic link or reparse-point directory '{source}'.");
+
         var sourceRoot = Path.GetFullPath(source);
         if (!Path.EndsInDirectorySeparator(sourceRoot))
             sourceRoot += Path.DirectorySeparatorChar;

eng/skill-validator/src/Evaluate/AssertionEvaluator.cs

@@ -11,12 +11,14 @@ public static class AssertionEvaluator
         IReadOnlyList<Assertion> assertions,
         string agentOutput,
         string workDir,
-        int scenarioTimeoutSeconds = EvalSchema.DefaultScenarioTimeoutSeconds)
+        int scenarioTimeoutSeconds = EvalSchema.DefaultScenarioTimeoutSeconds,
+        RunMetrics? metrics = null)
     {
         var results = new List<AssertionResult>();
         foreach (var assertion in assertions)
         {
-            var result = await EvaluateAssertion(assertion, agentOutput, workDir, scenarioTimeoutSeconds);
+            var result = await EvaluateAssertion(
+                assertion, agentOutput, workDir, scenarioTimeoutSeconds, metrics);
             results.Add(result);
         }
         return results;
@@ -86,7 +88,8 @@ public static class AssertionEvaluator
         Assertion assertion,
         string agentOutput,
         string workDir,
-        int scenarioTimeoutSeconds)
+        int scenarioTimeoutSeconds,
+        RunMetrics? metrics)
     {
         return assertion.Type switch
         {
@@ -98,7 +101,7 @@ public static class AssertionEvaluator
             AssertionType.OutputNotContains => EvalOutputNotContains(assertion, agentOutput),
             AssertionType.OutputMatches => EvalOutputMatches(assertion, agentOutput),
             AssertionType.OutputNotMatches => EvalOutputNotMatches(assertion, agentOutput),
-            AssertionType.ExitSuccess => EvalExitSuccess(assertion, agentOutput),
+            AssertionType.ExitSuccess => EvalExitSuccess(assertion, agentOutput, metrics),
             AssertionType.RunCommandAndAssert => await EvalRunCommandAndAssert(assertion, workDir, scenarioTimeoutSeconds),
             _ => new AssertionResult(assertion, false, $"Unknown assertion type: {assertion.Type}"),
         };
@@ -230,13 +233,23 @@ public static class AssertionEvaluator
         }
     }
 
-    private static AssertionResult EvalExitSuccess(Assertion a, string agentOutput)
+    private static AssertionResult EvalExitSuccess(
+        Assertion a,
+        string agentOutput,
+        RunMetrics? metrics)
     {
-        bool success = agentOutput.Length > 0;
+        var hasOutput = agentOutput.Length > 0;
+        var completedCleanly = metrics is null
+            || (!metrics.TimedOut
+                && metrics.ErrorCount == 0
+                && metrics.Events?.Any(evt => evt.Type == "session.idle") == true);
+        bool success = hasOutput && completedCleanly;
         return new AssertionResult(a, success,
-            success
-                ? "Agent completed successfully"
-                : "Agent produced no output");
+            success ? "Agent completed successfully"
+            : !hasOutput ? "Agent produced no output"
+            : metrics?.TimedOut == true ? "Agent produced output but timed out"
+            : metrics?.ErrorCount > 0 ? $"Agent produced output but recorded {metrics.ErrorCount} error(s)"
+            : "Agent produced output but did not reach session.idle");
     }
 
     private const int MaxOutputLength = 4096;
@@ -259,15 +272,28 @@ public static class AssertionEvaluator
             return new AssertionResult(a, false, $"Invalid timeout value {timeoutSeconds}s. Timeout must be greater than 0.");
         }
 
-        var processStartInfo = new ProcessStartInfo(command, cmd.CommandArguments ?? string.Empty)
+        var processStartInfo = new ProcessStartInfo
         {
+            FileName = command,
             WorkingDirectory = workDir,
             RedirectStandardOutput = true,
             RedirectStandardError = true,
             UseShellExecute = false,
         };
+        if (cmd.ArgumentList is { Length: > 0 })
+        {
+            foreach (var argument in cmd.ArgumentList)
+                processStartInfo.ArgumentList.Add(argument);
+        }
+        else
+        {
+            processStartInfo.Arguments = cmd.CommandArguments ?? string.Empty;
+        }
 
         AgentRunner.ScrubSensitiveEnvironment(processStartInfo);
+        var displayedArguments = cmd.ArgumentList is { Length: > 0 }
+            ? string.Join(" ", cmd.ArgumentList)
+            : cmd.CommandArguments;
 
         Process process;
         try
@@ -275,13 +301,13 @@ public static class AssertionEvaluator
             var started = Process.Start(processStartInfo);
             if (started is null)
             {
-                return new AssertionResult(a, false, $"Failed to start process '{command}' {cmd.CommandArguments}");
+                return new AssertionResult(a, false, $"Failed to start process '{command}' {displayedArguments}");
             }
             process = started;
         }
         catch (Exception ex)
         {
-            return new AssertionResult(a, false, $"Failed to start process '{command}' {cmd.CommandArguments}: {ex.Message}");
+            return new AssertionResult(a, false, $"Failed to start process '{command}' {displayedArguments}: {ex.Message}");
         }
 
         using (process)

eng/skill-validator/src/Evaluate/BaselineStore.cs

@@ -220,7 +220,7 @@ internal sealed class BaselineStore
                 else if (f.Source is not null)
                 {
                     var resolved = AgentRunner.ResolveSourcePath(f.Source, evalPath, skillPath: null);
-                    sb.Append("s:").Append(resolved is not null && File.Exists(resolved) ? HashFile(resolved) : "missing");
+                    AppendSourceIdentity(sb, resolved);
                 }
                 sb.Append('\n');
             }
@@ -236,6 +236,29 @@ internal sealed class BaselineStore
         return Sha256Hex(Encoding.UTF8.GetBytes(sb.ToString()));
     }
 
+    private static void AppendSourceIdentity(StringBuilder sb, string? resolved)
+    {
+        if (resolved is not null && File.Exists(resolved))
+        {
+            sb.Append("s:file:").Append(HashFile(resolved));
+            return;
+        }
+
+        if (resolved is not null && Directory.Exists(resolved))
+        {
+            sb.Append("s:directory:");
+            var sourceRoot = Path.TrimEndingDirectorySeparator(Path.GetFullPath(resolved));
+            foreach (var (rel, full) in EnumerateDirFixtures(resolved, "", sourceRoot)
+                         .OrderBy(x => x.Rel, StringComparer.Ordinal))
+            {
+                sb.Append('\n').Append(rel).Append('=').Append(HashFile(full));
+            }
+            return;
+        }
+
+        sb.Append("s:missing");
+    }
+
     private static readonly StringComparison PathComparison =
         OperatingSystem.IsWindows() ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
 
@@ -272,7 +295,9 @@ internal sealed class BaselineStore
         {
             if ((entry.Attributes & FileAttributes.ReparsePoint) != 0)
                 continue;
-            var rel = string.Concat(relBase, "/", entry.Name);
+            var rel = string.IsNullOrEmpty(relBase)
+                ? entry.Name
+                : string.Concat(relBase, "/", entry.Name);
             if (entry is DirectoryInfo sub)
             {
                 var subFull = Path.TrimEndingDirectorySeparator(Path.GetFullPath(sub.FullName));
@@ -312,6 +337,7 @@ internal sealed class BaselineStore
                   .Append(a.Value ?? "").Append('|').Append(a.Pattern ?? "").Append('|');
                 if (a.CommandArgs is { } ca)
                     sb.Append(ca.CommandToRun).Append(';').Append(ca.CommandArguments ?? "").Append(';')
+                      .Append(ca.ArgumentList is null ? "" : string.Join('\u001f', ca.ArgumentList)).Append(';')
                       .Append(ca.ExpectedExitCode?.ToString() ?? "").Append(';').Append(ca.ExpectedStdOutContains ?? "").Append(';')
                       .Append(ca.ExpectedStdErrorContains ?? "").Append(';').Append(ca.ExpectedStdOutMatches ?? "").Append(';')
                       .Append(ca.ExpectedStdErrorMatches ?? "").Append(';').Append(ca.Timeout?.ToString() ?? "");

eng/skill-validator/src/Evaluate/Comparator.cs

@@ -68,7 +68,36 @@ public static class Comparator
         IReadOnlyList<ScenarioComparison> comparisons,
         double minImprovement,
         bool requireCompletion,
-        double confidenceLevel = 0.95)
+        double confidenceLevel = 0.95) =>
+        ComputeVerdictCore(
+            skill,
+            comparisons,
+            minImprovement,
+            requireCompletion,
+            confidenceLevel,
+            pluginIsDiagnosticOnly: false);
+
+    public static SkillVerdict ComputeAgentVerdict(
+        SkillInfo agent,
+        IReadOnlyList<ScenarioComparison> comparisons,
+        double minImprovement,
+        bool requireCompletion,
+        double confidenceLevel = 0.95) =>
+        ComputeVerdictCore(
+            agent,
+            comparisons,
+            minImprovement,
+            requireCompletion,
+            confidenceLevel,
+            pluginIsDiagnosticOnly: true);
+
+    private static SkillVerdict ComputeVerdictCore(
+        SkillInfo skill,
+        IReadOnlyList<ScenarioComparison> comparisons,
+        double minImprovement,
+        bool requireCompletion,
+        double confidenceLevel,
+        bool pluginIsDiagnosticOnly)
     {
         if (comparisons.Count == 0)
         {
@@ -89,7 +118,7 @@ public static class Comparator
             .ToList();
 
         double overallImprovementScore = comparisons.Average(c => c.ImprovementScore);
-        double normalizedGain = ComputeNormalizedGain(comparisons);
+        double normalizedGain = ComputeNormalizedGain(comparisons, pluginIsDiagnosticOnly);
 
         var ci = Statistics.BootstrapConfidenceInterval(allPerRunScores, confidenceLevel);
         bool significant = Statistics.IsStatisticallySignificant(ci);
@@ -98,7 +127,8 @@ public static class Comparator
         {
             bool regressed = comparisons.Any(c =>
                 c.Baseline.Metrics.TaskCompleted &&
-                (!c.SkilledIsolated.Metrics.TaskCompleted || (c.SkilledPlugin is not null && !c.SkilledPlugin.Metrics.TaskCompleted)));
+                (!c.SkilledIsolated.Metrics.TaskCompleted
+                    || (!pluginIsDiagnosticOnly && c.SkilledPlugin is not null && !c.SkilledPlugin.Metrics.TaskCompleted)));
             if (regressed)
             {
                 return new SkillVerdict
@@ -174,7 +204,9 @@ public static class Comparator
     /// Normalized gain: g = (post - pre) / (1 - pre)
     /// Per Hake (1998), used in SkillsBench to control for ceiling effects.
     /// </summary>
-    private static double ComputeNormalizedGain(IReadOnlyList<ScenarioComparison> comparisons)
+    private static double ComputeNormalizedGain(
+        IReadOnlyList<ScenarioComparison> comparisons,
+        bool pluginIsDiagnosticOnly)
     {
         if (comparisons.Count == 0) return 0;
 
@@ -188,7 +220,9 @@ public static class Comparator
             // then use that run's overall score so this aligns with the effective
             // comparison used for pass/fail and reporting.
             double effectiveScore;
-            if (c.SkilledPlugin is not null && c.PluginImprovementScore < c.IsolatedImprovementScore)
+            if (!pluginIsDiagnosticOnly
+                && c.SkilledPlugin is not null
+                && c.PluginImprovementScore < c.IsolatedImprovementScore)
                 effectiveScore = c.SkilledPlugin.JudgeResult.OverallScore;
             else
                 effectiveScore = c.SkilledIsolated.JudgeResult.OverallScore;

eng/skill-validator/src/Evaluate/EvalSchema.cs

@@ -39,10 +39,12 @@ public static class EvalSchema
     /// (<c>stimuli:</c>/<c>graders:</c>) or the legacy skill-validator format
     /// (<c>scenarios:</c>), returning null when neither yields any scenario.
     ///
-    /// Unlike <see cref="ParseEvalConfig"/>, this never throws on an unrecognized
-    /// or empty schema — the standalone overfitting judge treats an unparseable
-    /// eval as "skip, don't fail". The Vally format is tried first because it is
-    /// the schema every eval.yaml in this repo now uses.
+    /// Unlike <see cref="ParseEvalConfig"/>, this returns null for an
+    /// unrecognized or empty schema. Invalid values in a recognized schema
+    /// still throw so the native evaluator fails explicitly. The standalone
+    /// overfitting command catches those errors and treats them as "skip, don't
+    /// fail". The Vally format is tried first because it is the schema every
+    /// eval.yaml in this repo now uses.
     /// </summary>
     public static EvalConfig? ParseEvalConfigFlexible(string yamlContent)
     {
@@ -69,9 +71,9 @@ public static class EvalSchema
 
     /// <summary>
     /// Map a Vally-native eval (<c>stimuli</c> with per-stimulus <c>prompt</c>,
-    /// <c>graders</c>, and <c>rubric</c>) onto the internal <see cref="EvalConfig"/>
-    /// shape the overfitting judge consumes. Each stimulus becomes a scenario
-    /// (name + prompt + rubric), and recognized output graders map to assertions.
+    /// environment, graders, constraints, and rubric) onto the internal
+    /// <see cref="EvalConfig"/> used by the native custom-agent lane and the
+    /// standalone overfitting judge.
     /// Returns null when the YAML has no <c>stimuli</c>.
     /// </summary>
     internal static EvalConfig? TryParseVallyEvalConfig(string yamlContent)
@@ -89,6 +91,9 @@ public static class EvalSchema
         if (raw?.Stimuli is not { Count: > 0 })
             return null;
 
+        var defaults = raw.Defaults ?? raw.Config;
+        var defaultTimeout = ParseDurationSeconds(defaults?.Timeout)
+            ?? DefaultScenarioTimeoutSeconds;
         var scenarios = new List<EvalScenario>();
         foreach (var stimulus in raw.Stimuli)
         {
@@ -106,30 +111,110 @@ public static class EvalSchema
                 }
             }
 
+            SetupConfig? setup = null;
+            if (stimulus.Environment is not null)
+            {
+                var files = stimulus.Environment.Files?.Select(file =>
+                    new SetupFile(file.Dest, file.Src)).ToList();
+                setup = new SetupConfig(
+                    Files: files,
+                    Commands: stimulus.Environment.Commands,
+                    AdditionalRequiredSkills: stimulus.Environment.Skills,
+                    AdditionalRequiredAgents: stimulus.Environment.Agents);
+            }
+
             scenarios.Add(new EvalScenario(
                 Name: stimulus.Name,
                 Prompt: stimulus.Prompt,
+                Setup: setup,
                 Assertions: assertions,
-                Rubric: stimulus.Rubric is { Count: > 0 } ? stimulus.Rubric : null));
+                Rubric: stimulus.Rubric is { Count: > 0 } ? stimulus.Rubric : null,
+                Timeout: ParseDurationSeconds(stimulus.Constraints?.MaxDuration) ?? defaultTimeout,
+                ExpectTools: stimulus.Constraints?.ExpectTools,
+                RejectTools: stimulus.Constraints?.RejectTools,
+                MaxTurns: stimulus.Constraints?.MaxTurns,
+                MaxTokens: stimulus.Constraints?.MaxTokens,
+                ExpectActivation: stimulus.ExpectActivation ?? true));
         }
 
         return scenarios.Count > 0 ? new EvalConfig(scenarios) : null;
     }
 
     /// <summary>
-    /// Best-effort map of a Vally output grader onto an <see cref="Assertion"/>.
-    /// The overfitting judge sends the raw eval YAML to the LLM regardless, so
-    /// unrecognized grader types (e.g. <c>prompt</c>, the LLM-rubric grader) are
-    /// simply skipped rather than treated as errors.
+    /// Map deterministic Vally graders onto native assertions. The
+    /// <c>prompt</c> grader is represented by the evaluator's rubric judge.
     /// </summary>
-    private static Assertion? MapVallyGrader(RawVallyGrader grader) => grader.Type switch
+    private static Assertion? MapVallyGrader(RawVallyGrader grader)
     {
-        "output-contains" => new Assertion(AssertionType.OutputContains, Value: grader.Config?.Substring),
-        "output-not-contains" => new Assertion(AssertionType.OutputNotContains, Value: grader.Config?.Substring),
-        "output-matches" => new Assertion(AssertionType.OutputMatches, Pattern: grader.Config?.Pattern),
-        "output-not-matches" => new Assertion(AssertionType.OutputNotMatches, Pattern: grader.Config?.Pattern),
-        _ => null,
-    };
+        var config = grader.Config;
+        return grader.Type switch
+        {
+            "file-exists" => new Assertion(AssertionType.FileExists, Path: config?.Path),
+            "file-not-exists" => new Assertion(AssertionType.FileNotExists, Path: config?.Path),
+            "file-contains" => new Assertion(AssertionType.FileContains, Path: config?.Path, Value: config?.Value),
+            "file-not-contains" => new Assertion(AssertionType.FileNotContains, Path: config?.Path, Value: config?.Value),
+            "output-contains" => new Assertion(AssertionType.OutputContains, Value: config?.Substring),
+            "output-not-contains" => new Assertion(AssertionType.OutputNotContains, Value: config?.Substring),
+            "output-matches" => new Assertion(AssertionType.OutputMatches, Pattern: config?.Pattern),
+            "output-not-matches" => new Assertion(AssertionType.OutputNotMatches, Pattern: config?.Pattern),
+            "exit-success" => new Assertion(AssertionType.ExitSuccess),
+            "run-command" when !string.IsNullOrWhiteSpace(config?.Command) =>
+                new Assertion(
+                    AssertionType.RunCommandAndAssert,
+                    CommandArgs: BuildShellCommandAssertion(config)),
+            // The prompt grader is handled by the legacy evaluator's rubric judge.
+            "prompt" => null,
+            _ => null,
+        };
+    }
+
+    private static CommandAssertionArgs BuildShellCommandAssertion(RawVallyGraderConfig config)
+    {
+        var command = config.Command!;
+        return new CommandAssertionArgs(
+            CommandToRun: OperatingSystem.IsWindows() ? "cmd.exe" : "/bin/sh",
+            ExpectedExitCode: config.ExpectedExitCode ?? 0,
+            ExpectedStdOutContains: config.StdoutContains,
+            ExpectedStdOutMatches: config.StdoutMatches,
+            Timeout: ParseDurationSeconds(config.Timeout),
+            CommandArguments: OperatingSystem.IsWindows()
+                ? $"/d /s /c \"{command}\""
+                : null,
+            ArgumentList: OperatingSystem.IsWindows() ? null : ["-c", command]);
+    }
+
+    internal static int? ParseDurationSeconds(string? value)
+    {
+        if (string.IsNullOrWhiteSpace(value))
+            return null;
+
+        var text = value.Trim();
+        if (int.TryParse(text, out var seconds) && seconds > 0)
+            return seconds;
+
+        var match = System.Text.RegularExpressions.Regex.Match(
+            text, @"^(\d+)(ms|s|m|h)$", System.Text.RegularExpressions.RegexOptions.IgnoreCase);
+        if (!match.Success || !long.TryParse(match.Groups[1].Value, out var amount) || amount <= 0)
+            throw new InvalidOperationException($"Invalid duration '{value}'. Use a positive value such as '90s', '5m', or '1h'.");
+
+        try
+        {
+            var totalSeconds = match.Groups[2].Value.ToLowerInvariant() switch
+            {
+                "ms" => Math.Max(1, amount / 1000 + (amount % 1000 == 0 ? 0 : 1)),
+                "s" => amount,
+                "m" => checked(amount * 60L),
+                "h" => checked(amount * 3600L),
+                _ => throw new InvalidOperationException($"Invalid duration unit in '{value}'."),
+            };
+            return checked((int)totalSeconds);
+        }
+        catch (OverflowException)
+        {
+            throw new InvalidOperationException(
+                $"Duration '{value}' exceeds the supported maximum of {int.MaxValue} seconds.");
+        }
+    }
 
     private static EvalScenario ParseScenario(RawScenario raw)
     {
@@ -304,15 +389,50 @@ public static class EvalSchema
 
     internal sealed class RawVallyEvalConfig
     {
+        public RawVallyDefaults? Defaults { get; set; }
+        public RawVallyDefaults? Config { get; set; }
         public List<RawVallyStimulus>? Stimuli { get; set; }
     }
 
+    internal sealed class RawVallyDefaults
+    {
+        public string? Timeout { get; set; }
+    }
+
     internal sealed class RawVallyStimulus
     {
         public string Name { get; set; } = "";
         public string Prompt { get; set; } = "";
+        public RawVallyEnvironment? Environment { get; set; }
         public List<RawVallyGrader>? Graders { get; set; }
         public List<string>? Rubric { get; set; }
+        public RawVallyConstraints? Constraints { get; set; }
+        public bool? ExpectActivation { get; set; }
+    }
+
+    internal sealed class RawVallyEnvironment
+    {
+        public List<RawVallyFile>? Files { get; set; }
+        public List<string>? Commands { get; set; }
+        public List<string>? Skills { get; set; }
+        // Repository extension used by the native agent lane. Vally 0.14 does
+        // not support agent registration, so the SDK runner consumes this field.
+        public List<string>? Agents { get; set; }
+    }
+
+    internal sealed class RawVallyFile
+    {
+        public string Src { get; set; } = "";
+        public string Dest { get; set; } = "";
+    }
+
+    internal sealed class RawVallyConstraints
+    {
+        public string? MaxDuration { get; set; }
+        public List<string>? ExpectTools { get; set; }
+        public List<string>? RejectTools { get; set; }
+        public int? MaxTurns { get; set; }
+        public int? MaxTokens { get; set; }
     }
 
     internal sealed class RawVallyGrader
@@ -325,5 +445,12 @@ public static class EvalSchema
     {
         public string? Substring { get; set; }
         public string? Pattern { get; set; }
+        public string? Path { get; set; }
+        public string? Value { get; set; }
+        public string? Command { get; set; }
+        public int? ExpectedExitCode { get; set; }
+        public string? Timeout { get; set; }
+        public string? StdoutContains { get; set; }
+        public string? StdoutMatches { get; set; }
     }
 }

eng/skill-validator/src/Evaluate/EvaluateCommand.cs

@@ -312,7 +312,9 @@ public static class EvaluateCommand
             if (evalPath is not null && File.Exists(evalPath))
             {
                 var content = await File.ReadAllTextAsync(evalPath);
-                evalConfig = EvalSchema.ParseEvalConfig(content);
+                evalConfig = EvalSchema.ParseEvalConfigFlexible(content)
+                    ?? throw new InvalidOperationException(
+                        $"Agent eval '{evalPath}' does not contain any valid stimuli or scenarios.");
             }
             var mcpServers = await FindPluginMcpServers(agent.Path);
             allTargets.Add(new EvalTargetInfo(
@@ -543,7 +545,9 @@ public static class EvaluateCommand
 
     /// <summary>
     /// Evaluates a custom agent using the same three-way comparison pattern as skills:
-    /// baseline (no agent), agent-isolated (agent selected), agent-plugin (full plugin + agent selected).
+    /// baseline (no agent), agent-isolated (target registered), and agent-plugin
+    /// (full production plugin surface registered). The default parent remains
+    /// selected so routing and delegation are measured rather than forced.
     /// </summary>
     private static async Task<SkillVerdict?> EvaluateAgent(
         EvalTargetInfo target,
@@ -624,16 +628,30 @@ public static class EvaluateCommand
             return null;
         }
 
-        var verdict = Comparator.ComputeVerdict(
+        var verdict = Comparator.ComputeAgentVerdict(
             new SkillInfo(agent.Name, agent.Description, agent.Path, agent.Path, agent.AgentMdContent),
             comparisons, config.MinImprovement, config.RequireCompletion, config.ConfidenceLevel);
+        verdict.SkillKind = "agent";
+        ApplyAgentActivationGate(verdict, comparisons, agent.Name, log);
 
-        // Check agent activation via SubagentSelectedEvent (not SkillInvokedEvent)
+        log($"{(verdict.Passed ? "✅" : "❌")} Done (score: {verdict.OverallImprovementScore * 100:F1}%)");
+        return verdict;
+    }
+
+    internal static void ApplyAgentActivationGate(
+        SkillVerdict verdict,
+        IReadOnlyList<ScenarioComparison> comparisons,
+        string agentName,
+        Action<string> log)
+    {
+        // Check target-agent activation via subagent events (not SkillInvokedEvent).
+        // Only the isolated arm participates in the agent verdict. The plugin arm
+        // is production-surface telemetry, matching ComputeAgentVerdict's score gate.
         var notActivatedIsolated = comparisons.Where(c =>
-            c.SubagentActivationIsolated is { } sa && !sa.InvokedAgents.Any(n => n.Equals(agent.Name, StringComparison.OrdinalIgnoreCase))
+            c.SubagentActivationIsolated is { } sa && !sa.InvokedAgents.Any(n => n.Equals(agentName, StringComparison.OrdinalIgnoreCase))
             && c.ExpectActivation).ToList();
         var notActivatedPlugin = comparisons.Where(c =>
-            c.SubagentActivationPlugin is { } sa && !sa.InvokedAgents.Any(n => n.Equals(agent.Name, StringComparison.OrdinalIgnoreCase))
+            c.SubagentActivationPlugin is { } sa && !sa.InvokedAgents.Any(n => n.Equals(agentName, StringComparison.OrdinalIgnoreCase))
             && c.ExpectActivation).ToList();
 
         if (notActivatedIsolated.Count > 0)
@@ -649,14 +667,8 @@ public static class EvaluateCommand
         {
             var names = string.Join(", ", notActivatedPlugin.Select(c => c.ScenarioName));
             log($"{Ansi.Yellow}⚠️  Agent NOT activated (plugin) in: {names}{Ansi.Reset}");
-            verdict.SkillNotActivated = true;
-            verdict.Passed = false;
-            verdict.FailureKind = FailureKind.SkillNotActivated;
             verdict.Reason += $" [AGENT NOT ACTIVATED (plugin) in {notActivatedPlugin.Count} scenario(s)]";
         }
-
-        log($"{(verdict.Passed ? "✅" : "❌")} Done (score: {verdict.OverallImprovementScore * 100:F1}%)");
-        return verdict;
     }
 
     /// <summary>
@@ -726,23 +738,15 @@ public static class EvaluateCommand
         var perRunPairwise = runResults.Select(r => r.Pairwise).ToList();
 
         var perRunIsolatedScores = new List<double>();
-        var perRunPluginScores = new List<double>();
         for (int i = 0; i < baselineRuns.Count; i++)
         {
             var pw = perRunPairwise[i];
             bool pairwiseFromPlugin = runResults[i].PairwiseFromPlugin;
             var isoComp = Comparator.CompareScenario(scenario.Name, baselineRuns[i], isolatedRuns[i],
                 pairwiseFromPlugin ? null : pw);
-            var plgComp = Comparator.CompareScenario(scenario.Name, baselineRuns[i], pluginRuns[i],
-                pairwiseFromPlugin ? pw : null);
             perRunIsolatedScores.Add(isoComp.ImprovementScore);
-            perRunPluginScores.Add(plgComp.ImprovementScore);
         }
 
-        var perRunScores = perRunIsolatedScores
-            .Zip(perRunPluginScores, (iso, plg) => Math.Min(iso, plg))
-            .ToList();
-
         var avgBaseline = AverageResults(baselineRuns);
         var avgIsolated = AverageResults(isolatedRuns);
         var avgPlugin = AverageResults(pluginRuns);
@@ -777,17 +781,16 @@ public static class EvaluateCommand
             Baseline = avgBaseline,
             SkilledIsolated = avgIsolated,
             SkilledPlugin = avgPlugin,
-            ImprovementScore = Math.Min(isoComparison.ImprovementScore, plgComparison.ImprovementScore),
+            ImprovementScore = isoComparison.ImprovementScore,
             IsolatedImprovementScore = isoComparison.ImprovementScore,
             PluginImprovementScore = plgComparison.ImprovementScore,
-            Breakdown = isoComparison.ImprovementScore <= plgComparison.ImprovementScore
-                ? isoComparison.Breakdown : plgComparison.Breakdown,
+            Breakdown = isoComparison.Breakdown,
             IsolatedBreakdown = isoComparison.Breakdown,
             PluginBreakdown = plgComparison.Breakdown,
             PairwiseResult = bestPairwise,
         };
-        comparison.PerRunScores = perRunScores;
-        comparison.VarianceCV = Statistics.CoefficientOfVariation(perRunScores);
+        comparison.PerRunScores = perRunIsolatedScores;
+        comparison.VarianceCV = Statistics.CoefficientOfVariation(perRunIsolatedScores);
         comparison.HighVariance = comparison.VarianceCV is > 0.5;
 
         // Aggregate subagent activation across runs (primary activation signal for agents)
@@ -879,18 +882,27 @@ public static class EvaluateCommand
         IReadOnlyList<AgentInfo>? additionalAgents = null;
         if (scenario.Setup is not null && pluginRoot is not null)
         {
-            additionalSkills = await ResolveAdditionalSkills(scenario.Setup.AdditionalRequiredSkills, pluginRoot);
-            additionalAgents = await ResolveAdditionalAgents(scenario.Setup.AdditionalRequiredAgents, pluginRoot);
+            additionalSkills = await ResolveAdditionalSkills(
+                scenario.Setup.AdditionalRequiredSkills, pluginRoot, target.EvalPath);
+        }
+        if (pluginRoot is not null)
+        {
+            var agentDependencies = (agent.Agents ?? [])
+                .Concat(scenario.Setup?.AdditionalRequiredAgents ?? [])
+                .Distinct(StringComparer.OrdinalIgnoreCase)
+                .ToList();
+            additionalAgents = await ResolveAdditionalAgents(agentDependencies, pluginRoot, target.EvalPath);
         }
 
-        // 2. Agent-isolated: target agent only (+ scenario deps)
+        // 2. Agent-isolated: target agent only (+ declared skill/agent dependencies).
         var isolatedTask = AgentRunner.RunAgent(new RunOptions(scenario, null, target.EvalPath, config.Model, config.Verbose,
             PluginRoot: null, Log: runLog, McpServers: target.McpServers, SessionsDir: sessionsDir,
-            SessionId: isolatedSessionId, Agent: agent, AdditionalSkills: additionalSkills, AdditionalAgents: additionalAgents), cancellationToken);
-        // 3. Agent-plugin: full plugin context + agent selected
+            SessionId: isolatedSessionId, Agent: agent, AdditionalSkills: additionalSkills,
+            AdditionalAgents: additionalAgents, SelectAgentAsPrimary: false), cancellationToken);
+        // 3. Agent-plugin: full production plugin skills and agents.
         var pluginTask = AgentRunner.RunAgent(new RunOptions(scenario, null, target.EvalPath, config.Model, config.Verbose,
             PluginRoot: pluginRoot, Log: runLog, McpServers: target.McpServers, SessionsDir: sessionsDir,
-            SessionId: pluginSessionId, Agent: agent), cancellationToken);
+            SessionId: pluginSessionId, Agent: agent, SelectAgentAsPrimary: false), cancellationToken);
 
         RunMetrics baselineMetrics;
         RunMetrics isolatedMetrics;
@@ -930,9 +942,9 @@ public static class EvaluateCommand
         if (scenario.Assertions is { Count: > 0 })
         {
             if (reusedBaseline is null)
-                baselineMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, baselineMetrics.AgentOutput, baselineMetrics.WorkDir, scenario.Timeout);
-            isolatedMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, isolatedMetrics.AgentOutput, isolatedMetrics.WorkDir, scenario.Timeout);
-            pluginMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, pluginMetrics.AgentOutput, pluginMetrics.WorkDir, scenario.Timeout);
+                baselineMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, baselineMetrics.AgentOutput, baselineMetrics.WorkDir, scenario.Timeout, baselineMetrics);
+            isolatedMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, isolatedMetrics.AgentOutput, isolatedMetrics.WorkDir, scenario.Timeout, isolatedMetrics);
+            pluginMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, pluginMetrics.AgentOutput, pluginMetrics.WorkDir, scenario.Timeout, pluginMetrics);
         }
 
         var baselineConstraints = reusedBaseline is null ? AssertionEvaluator.EvaluateConstraints(scenario, baselineMetrics) : [];
@@ -1006,8 +1018,9 @@ public static class EvaluateCommand
         bool pairwiseFromPlugin = false;
         if (usePairwise)
         {
-            pairwiseFromPlugin = pluginJudge.OverallScore < isolatedJudge.OverallScore;
-            var worseSkilled = pairwiseFromPlugin ? pluginMetrics : isolatedMetrics;
+            // Agent preference is always baseline vs isolated target. The full
+            // plugin arm is diagnostic telemetry, matching the Vally skill lane.
+            var worseSkilled = isolatedMetrics;
             try
             {
                 // Reused baseline work dir no longer exists; run the judge in the skilled
@@ -1493,8 +1506,10 @@ public static class EvaluateCommand
         IReadOnlyList<AgentInfo>? additionalAgents = null;
         if (scenario.Setup is not null && pluginRoot is not null)
         {
-            additionalSkills = await ResolveAdditionalSkills(scenario.Setup.AdditionalRequiredSkills, pluginRoot);
-            additionalAgents = await ResolveAdditionalAgents(scenario.Setup.AdditionalRequiredAgents, pluginRoot);
+            additionalSkills = await ResolveAdditionalSkills(
+                scenario.Setup.AdditionalRequiredSkills, pluginRoot, evalSkill.EvalPath);
+            additionalAgents = await ResolveAdditionalAgents(
+                scenario.Setup.AdditionalRequiredAgents, pluginRoot, evalSkill.EvalPath);
         }
 
         // 2. Skilled-isolated: target skill + declared dependencies
@@ -1502,8 +1517,9 @@ public static class EvaluateCommand
             PluginRoot: null, Log: runLog, McpServers: evalSkill.McpServers, SessionsDir: sessionsDir,
             SessionId: isolatedSessionId, AdditionalSkills: additionalSkills, AdditionalAgents: additionalAgents), cancellationToken);
         // 3. Skilled-plugin: load entire plugin from plugin root directory
-        var pluginTask = AgentRunner.RunAgent(new RunOptions(scenario, skill, evalSkill.EvalPath, config.Model, config.Verbose,
-            PluginRoot: pluginRoot, Log: runLog, McpServers: evalSkill.McpServers, SessionsDir: sessionsDir, SessionId: pluginSessionId), cancellationToken);
+        var pluginTask = AgentRunner.RunAgent(CreateSkillPluginRunOptions(
+            scenario, evalSkill, config, pluginRoot, runLog, sessionsDir,
+            pluginSessionId, additionalAgents), cancellationToken);
 
         RunMetrics baselineMetrics;
         RunMetrics isolatedMetrics;
@@ -1542,9 +1558,9 @@ public static class EvaluateCommand
         if (scenario.Assertions is { Count: > 0 })
         {
             if (reusedBaseline is null)
-                baselineMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, baselineMetrics.AgentOutput, baselineMetrics.WorkDir, scenario.Timeout);
-            isolatedMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, isolatedMetrics.AgentOutput, isolatedMetrics.WorkDir, scenario.Timeout);
-            pluginMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, pluginMetrics.AgentOutput, pluginMetrics.WorkDir, scenario.Timeout);
+                baselineMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, baselineMetrics.AgentOutput, baselineMetrics.WorkDir, scenario.Timeout, baselineMetrics);
+            isolatedMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, isolatedMetrics.AgentOutput, isolatedMetrics.WorkDir, scenario.Timeout, isolatedMetrics);
+            pluginMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(scenario.Assertions, pluginMetrics.AgentOutput, pluginMetrics.WorkDir, scenario.Timeout, pluginMetrics);
         }
 
         // Evaluate constraints on the skilled runs (baseline constraints are cached when reused)
@@ -1665,6 +1681,7 @@ public static class EvaluateCommand
                     sessionDb.SavePairwiseResult(baselineSessionId, JsonSerializer.Serialize(pairwise, SkillValidatorJsonContext.Default.PairwiseJudgeResult));
                 }
             }
+
             catch (Exception error)
             {
                 runLog($"⚠️  Pairwise judge failed: {error}");
@@ -1702,6 +1719,28 @@ public static class EvaluateCommand
             pairwiseFromPlugin, isolatedActivation, pluginActivation, isolatedSubagent, pluginSubagent);
     }
 
+    internal static RunOptions CreateSkillPluginRunOptions(
+        EvalScenario scenario,
+        EvalSkillInfo evalSkill,
+        ValidatorConfig config,
+        string? pluginRoot,
+        Action<string> runLog,
+        string? sessionsDir,
+        string pluginSessionId,
+        IReadOnlyList<AgentInfo>? additionalAgents) =>
+        new(
+            scenario,
+            evalSkill.Skill,
+            evalSkill.EvalPath,
+            config.Model,
+            config.Verbose,
+            PluginRoot: pluginRoot,
+            Log: runLog,
+            McpServers: evalSkill.McpServers,
+            SessionsDir: sessionsDir,
+            SessionId: pluginSessionId,
+            AdditionalAgents: additionalAgents);
+
     private static async Task<(JudgeResult Result, TokenUsage Tokens)> SafeJudge(Task<(JudgeResult Result, TokenUsage Tokens)> task, string label, Action<string> runLog)
     {
         try
@@ -1841,9 +1880,9 @@ public static class EvaluateCommand
                         if (scenario.Assertions is { Count: > 0 })
                         {
                             skillOnlyMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(
-                                scenario.Assertions, skillOnlyMetrics.AgentOutput, skillOnlyMetrics.WorkDir, scenario.Timeout);
+                                scenario.Assertions, skillOnlyMetrics.AgentOutput, skillOnlyMetrics.WorkDir, scenario.Timeout, skillOnlyMetrics);
                             allSkillsMetrics.AssertionResults = await AssertionEvaluator.EvaluateAssertions(
-                                scenario.Assertions, allSkillsMetrics.AgentOutput, allSkillsMetrics.WorkDir, scenario.Timeout);
+                                scenario.Assertions, allSkillsMetrics.AgentOutput, allSkillsMetrics.WorkDir, scenario.Timeout, allSkillsMetrics);
                         }
                         var soConstraints = AssertionEvaluator.EvaluateConstraints(scenario, skillOnlyMetrics);
                         var asConstraints = AssertionEvaluator.EvaluateConstraints(scenario, allSkillsMetrics);
@@ -2086,7 +2125,7 @@ public static class EvaluateCommand
                         {
                             var refPath = serversEl.GetString()!;
                             if (!Path.IsPathRooted(refPath) && !refPath.Contains(".."))
-                                mcpObject = await ResolveMcpFile(Path.Combine(dir, refPath));
+                                mcpObject = await ResolveMcpFile(dir, Path.Combine(dir, refPath));
                         }
                         else if (serversEl.ValueKind == JsonValueKind.Object)
                         {
@@ -2126,8 +2165,13 @@ public static class EvaluateCommand
     /// Resolve a .mcp.json file path and return the mcpServers object element, or null.
     /// Codex plugins use a string path in plugin.json to reference an external .mcp.json file.
     /// </summary>
-    private static async Task<JsonElement?> ResolveMcpFile(string mcpPath)
+    private static async Task<JsonElement?> ResolveMcpFile(string pluginRoot, string mcpPath)
     {
+        if (PathSafety.ContainsReparsePoint(pluginRoot, mcpPath))
+        {
+            Console.Error.WriteLine($"Refusing to read .mcp.json through a symbolic link or reparse point: {mcpPath}");
+            return null;
+        }
         if (!File.Exists(mcpPath)) return null;
         try
         {
@@ -2252,7 +2296,7 @@ public static class EvaluateCommand
     /// These are author-declared names in eval.yaml that must map to skills in the plugin.
     /// </summary>
     internal static async Task<IReadOnlyList<SkillInfo>?> ResolveAdditionalSkills(
-        IReadOnlyList<string>? skillNames, string pluginRoot)
+        IReadOnlyList<string>? skillNames, string pluginRoot, string? evalPath = null)
     {
         if (skillNames is not { Count: > 0 })
             return null;
@@ -2261,25 +2305,53 @@ public static class EvaluateCommand
         var allSkills = new List<SkillInfo>();
         foreach (var dir in pluginSkillDirs)
         {
-            var skills = await SkillDiscovery.DiscoverSkills(dir);
+            var skills = await SkillDiscovery.DiscoverSkills(dir, pluginRoot);
             allSkills.AddRange(skills);
         }
 
         var resolved = new List<SkillInfo>();
-        foreach (var name in skillNames)
+        foreach (var reference in skillNames)
         {
-            var match = allSkills.FirstOrDefault(s =>
-                s.Name.Equals(name, StringComparison.OrdinalIgnoreCase)
-                || Path.GetFileName(s.Path).Equals(name, StringComparison.OrdinalIgnoreCase));
+            SkillInfo? match = null;
+            if (LooksLikePath(reference) && evalPath is not null)
+            {
+                var pluginsRoot = Directory.GetParent(Path.GetFullPath(pluginRoot))?.FullName;
+                var candidate = ResolveDeclaredDependencyPath(reference, pluginsRoot, evalPath);
+                if (pluginsRoot is null || candidate is null || !IsWithinDirectory(candidate, pluginsRoot))
+                {
+                    throw new InvalidOperationException(
+                        $"environment.skills path '{reference}' resolves outside the repository plugins directory.");
+                }
+
+                var matches = (await SkillDiscovery.DiscoverSkills(candidate, pluginsRoot)).ToList();
+                if (matches.Count > 1)
+                {
+                    throw new InvalidOperationException(
+                        $"Required skill path '{reference}' resolves to '{candidate}', which contains multiple skills: "
+                        + $"{string.Join(", ", matches.Select(skill => $"'{skill.Name}'").Order())}. "
+                        + "Point to a specific skill directory.");
+                }
+                match = matches.SingleOrDefault();
+            }
+            else
+            {
+                match = allSkills.FirstOrDefault(s =>
+                    s.Name.Equals(reference, StringComparison.OrdinalIgnoreCase)
+                    || Path.GetFileName(s.Path).Equals(reference, StringComparison.OrdinalIgnoreCase));
+            }
+
             if (match is not null)
                 resolved.Add(match);
             else
                 throw new InvalidOperationException(
-                    $"additional_required_skills: '{name}' not found in plugin at '{pluginRoot}'. "
-                    + "Check that the skill name matches a skill directory under the plugin's skills/ folder.");
+                    $"Required skill {DescribeReference(reference)} could not be resolved for plugin at '{pluginRoot}'. "
+                    + "Use a bare skill name or an eval-relative path such as "
+                    + "'../../plugins/<plugin>/skills/<skill>'.");
         }
 
-        return resolved.Count > 0 ? resolved : null;
+        return resolved
+            .DistinctBy(skill => Path.GetFullPath(skill.Path), StringComparer.OrdinalIgnoreCase)
+            .ToList();
     }
 
     /// <summary>
@@ -2287,26 +2359,112 @@ public static class EvaluateCommand
     /// These are author-declared names in eval.yaml that must map to agents in the plugin.
     /// </summary>
     internal static async Task<IReadOnlyList<AgentInfo>?> ResolveAdditionalAgents(
-        IReadOnlyList<string>? agentNames, string pluginRoot)
+        IReadOnlyList<string>? agentNames, string pluginRoot, string? evalPath = null)
     {
         if (agentNames is not { Count: > 0 })
             return null;
 
         var allAgents = await AgentDiscovery.DiscoverAgentsInPlugin(pluginRoot);
         var resolved = new List<AgentInfo>();
-        foreach (var name in agentNames)
+        var pending = new Queue<string>(agentNames);
+        var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+        while (pending.TryDequeue(out var reference))
         {
-            var match = allAgents.FirstOrDefault(a =>
-                a.Name.Equals(name, StringComparison.OrdinalIgnoreCase));
+            AgentInfo? match = null;
+            if (LooksLikePath(reference) && evalPath is not null)
+            {
+                var pluginsRoot = Directory.GetParent(Path.GetFullPath(pluginRoot))?.FullName;
+                var candidate = ResolveDeclaredDependencyPath(reference, pluginsRoot, evalPath);
+                if (pluginsRoot is null || candidate is null || !IsWithinDirectory(candidate, pluginsRoot))
+                {
+                    throw new InvalidOperationException(
+                        $"environment.agents path '{reference}' resolves outside the repository plugins directory.");
+                }
+
+                var matches = (await AgentDiscovery.DiscoverAgentsInDirectory(candidate, pluginsRoot)).ToList();
+                if (matches.Count > 1)
+                {
+                    throw new InvalidOperationException(
+                        $"Required agent path '{reference}' resolves to '{candidate}', which contains multiple agents: "
+                        + $"{string.Join(", ", matches.Select(agent => $"'{agent.Name}'").Order())}. "
+                        + "Point to a specific agent file.");
+                }
+                match = matches.SingleOrDefault();
+            }
+            else
+            {
+                match = allAgents.FirstOrDefault(a =>
+                    a.Name.Equals(reference, StringComparison.OrdinalIgnoreCase));
+            }
+
             if (match is not null)
+            {
+                var fullPath = Path.GetFullPath(match.Path);
+                if (!seen.Add(fullPath))
+                    continue;
                 resolved.Add(match);
+                foreach (var dependency in match.Agents ?? [])
+                    pending.Enqueue(dependency);
+            }
             else
                 throw new InvalidOperationException(
-                    $"additional_required_agents: '{name}' not found in plugin at '{pluginRoot}'. "
-                    + "Check that the agent name matches an .agent.md file under the plugin's agents/ folder.");
+                    $"Required agent {DescribeReference(reference)} could not be resolved for plugin at '{pluginRoot}'. "
+                    + "Use a bare agent name or an eval-relative path such as "
+                    + "'../../plugins/<plugin>/agents/<agent>.agent.md'.");
         }
 
-        return resolved.Count > 0 ? resolved : null;
+        return resolved;
+    }
+
+    private static bool LooksLikePath(string reference) =>
+        reference.Contains(Path.DirectorySeparatorChar)
+        || reference.Contains(Path.AltDirectorySeparatorChar)
+        || reference.StartsWith(".", StringComparison.Ordinal);
+
+    private static string DescribeReference(string reference) =>
+        LooksLikePath(reference)
+            ? $"path '{reference}'"
+            : $"name '{reference}'";
+
+    private static string? ResolveDeclaredDependencyPath(
+        string reference, string? pluginsRoot, string evalPath)
+    {
+        if (pluginsRoot is null)
+            return null;
+
+        // Existing agent evals spell repository plugin dependencies as
+        // ../../plugins/<plugin>/..., even though Vally never executed them.
+        // Resolve the stable plugins/ suffix from the repository root so those
+        // declarations are portable across test-directory depth.
+        var normalized = reference.Replace('\\', '/');
+        var marker = normalized.StartsWith("plugins/", StringComparison.OrdinalIgnoreCase)
+            ? 0
+            : normalized.IndexOf("/plugins/", StringComparison.OrdinalIgnoreCase);
+        if (marker > 0)
+            marker++;
+        if (marker >= 0)
+        {
+            var repoRoot = Directory.GetParent(pluginsRoot)?.FullName;
+            return repoRoot is null
+                ? null
+                : Path.GetFullPath(Path.Combine(
+                    repoRoot,
+                    normalized[marker..].Replace('/', Path.DirectorySeparatorChar)));
+        }
+
+        return Path.GetFullPath(
+            Path.Combine(Path.GetDirectoryName(Path.GetFullPath(evalPath))!, reference));
+    }
+
+    private static bool IsWithinDirectory(string path, string directory)
+    {
+        var comparison = OperatingSystem.IsWindows()
+            ? StringComparison.OrdinalIgnoreCase
+            : StringComparison.Ordinal;
+        var normalizedDirectory = Path.TrimEndingDirectorySeparator(Path.GetFullPath(directory))
+            + Path.DirectorySeparatorChar;
+        var normalizedPath = Path.GetFullPath(path);
+        return normalizedPath.StartsWith(normalizedDirectory, comparison);
     }
 
     internal static (Dictionary<string, (PluginInfo Plugin, List<SkillInfo> Skills)> Groups, List<string> Errors)

eng/skill-validator/src/Evaluate/MetricsCollector.cs

@@ -156,6 +156,13 @@ public static class MetricsCollector
                     errorCount++;
                     break;
                 }
+
+                case "tool.execution_complete":
+                {
+                    if (GetBooleanValue(evt.Data, "success") == false)
+                        errorCount++;
+                    break;
+                }
             }
         }
 
@@ -198,6 +205,21 @@ public static class MetricsCollector
         return null;
     }
 
+    private static bool? GetBooleanValue(Dictionary<string, JsonNode?> data, string key)
+    {
+        if (!data.TryGetValue(key, out var value) || value is null)
+            return null;
+
+        try
+        {
+            return value.GetValue<bool>();
+        }
+        catch (InvalidOperationException)
+        {
+            return bool.TryParse(value.ToString(), out var parsed) ? parsed : null;
+        }
+    }
+
     private static int GetIntValue(Dictionary<string, JsonNode?> data, string key)
     {
         if (data.TryGetValue(key, out var value) && value is not null)

eng/skill-validator/src/Evaluate/Models.cs

@@ -56,7 +56,8 @@ public sealed record CommandAssertionArgs(
     string? ExpectedStdErrorContains = null,
     string? ExpectedStdOutMatches = null,
     string? ExpectedStdErrorMatches = null,
-    int? Timeout = null);
+    int? Timeout = null,
+    string[]? ArgumentList = null);
 
 public sealed record Assertion(
     AssertionType Type,
@@ -358,6 +359,7 @@ public sealed class SkillVerdict
         get => _schemaVersion ?? LegacySkillValidatorResultsSchema.CurrentVersion;
         init => _schemaVersion = value;
     }
+    public string SkillKind { get; set; } = "skill";
     public required string SkillName { get; init; }
     public required string SkillPath { get; init; }
     public required bool Passed { get; set; }

eng/skill-validator/src/README.md

@@ -61,6 +61,11 @@ skill-validator evaluate --help
 # Evaluate a skill (--tests-dir is required)
 skill-validator evaluate --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills/my-skill
 
+# Evaluate a custom agent without forcing it as the primary persona. The
+# default parent must route to the registered target agent.
+skill-validator evaluate --runs 1 --verdict-warn-only \
+  --tests-dir ./tests/my-plugin ./plugins/my-plugin/agents/my-agent.agent.md
+
 # Verbose output with per-scenario breakdowns
 skill-validator evaluate --verbose --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills
 
@@ -140,7 +145,7 @@ of Codex's evolving server parser in `skill-validator`.
 
 | Flag | Default | Description |
 |------|---------|-------------|
-| `<paths...>` | *(required)* | Paths to skill directories or parent directories |
+| `<paths...>` | *(required)* | Paths to skill directories, agent files/directories, or their parent directories |
 | `--tests-dir <path>` | *(required)* | Directory containing test subdirectories |
 | `--model <name>` | `claude-opus-4.6` | Model for agent runs |
 | `--judge-model <name>` | same as `--model` | Model for LLM judge (can be different) |
@@ -249,6 +254,12 @@ Both settings are optional. When omitted, the CLI defaults (`--parallel-scenario
 
 ### Scenarios
 
+Custom-agent evaluation also accepts the repository's Vally-native
+`stimuli:`/`graders:` format. `environment.skills` declares isolated-run skill
+dependencies, while an agent frontmatter `agents:` list declares custom-agent
+dependencies recursively. The plugin arm ignores those narrowed declarations
+and registers the complete production plugin skill and agent surface.
+
 ```yaml
 scenarios:
   - name: "Descriptive name of the scenario"

eng/skill-validator/src/Shared/AgentDiscovery.cs

@@ -60,13 +60,15 @@ public static class AgentDiscovery
 
             if (!PluginDiscovery.TryGetSafeSubdirectory(pluginRoot, relativePath, out var fullPath, out _))
                 continue;
+            if (PathSafety.ContainsReparsePoint(pluginRoot, fullPath!))
+                continue;
             if (Directory.Exists(fullPath!))
             {
-                agents.AddRange(await DiscoverAgentsInDirectory(fullPath!));
+                agents.AddRange(await DiscoverAgentsInDirectory(fullPath!, pluginRoot));
             }
             else
             {
-                var agent = await DiscoverAgentAt(fullPath!);
+                var agent = await DiscoverAgentAt(fullPath!, pluginRoot);
                 if (agent is not null)
                     agents.Add(agent);
             }
@@ -77,12 +79,17 @@ public static class AgentDiscovery
     /// <summary>
     /// Discover agent files (.agent.md) in the given directory, or a single agent if a file path is provided.
     /// </summary>
-    public static async Task<IReadOnlyList<AgentInfo>> DiscoverAgentsInDirectory(string agentsDir)
+    public static async Task<IReadOnlyList<AgentInfo>> DiscoverAgentsInDirectory(
+        string agentsDir,
+        string? allowedRoot = null)
     {
+        if (allowedRoot is not null && PathSafety.ContainsReparsePoint(allowedRoot, agentsDir))
+            return [];
+
         // If the path is a file, try to discover it directly
         if (File.Exists(agentsDir))
         {
-            var agent = await DiscoverAgentAt(agentsDir);
+            var agent = await DiscoverAgentAt(agentsDir, allowedRoot);
             return agent is not null ? [agent] : [];
         }
 
@@ -92,25 +99,28 @@ public static class AgentDiscovery
         var agents = new List<AgentInfo>();
         foreach (var file in Directory.GetFiles(agentsDir, "*.agent.md"))
         {
-            var agent = await DiscoverAgentAt(file);
+            var agent = await DiscoverAgentAt(file, allowedRoot);
             if (agent is not null)
                 agents.Add(agent);
         }
         return agents;
     }
 
-    private static async Task<AgentInfo?> DiscoverAgentAt(string filePath)
+    private static async Task<AgentInfo?> DiscoverAgentAt(
+        string filePath,
+        string? allowedRoot = null)
     {
         if (!File.Exists(filePath))
             return null;
+        if (allowedRoot is not null && PathSafety.ContainsReparsePoint(allowedRoot, filePath))
+            return null;
 
         var content = await File.ReadAllTextAsync(filePath);
         var (metadata, _) = ParseAgentFrontmatter(content);
         var fileName = Path.GetFileName(filePath);
         var name = metadata.Name ?? "";
         var description = metadata.Description ?? "";
-
-        return new AgentInfo(name, description, filePath, content, fileName, metadata.Tools);
+        return new AgentInfo(name, description, filePath, content, fileName, metadata.Tools, metadata.Agents);
     }
 
     internal static (AgentFrontmatter Metadata, string Body) ParseAgentFrontmatter(string content)
@@ -125,4 +135,3 @@ public static class AgentDiscovery
         return (metadata, body);
     }
 }
-

eng/skill-validator/src/Shared/Models.cs

@@ -31,7 +31,8 @@ public sealed record AgentInfo(
     string Path,
     string AgentMdContent,
     string FileName,
-    IReadOnlyList<string>? Tools = null);
+    IReadOnlyList<string>? Tools = null,
+    IReadOnlyList<string>? Agents = null);
 
 // --- Plugin info ---
 
@@ -64,4 +65,5 @@ public sealed record AgentFrontmatter
     public string? Name { get; set; }
     public string? Description { get; set; }
     public List<string>? Tools { get; set; }
+    public List<string>? Agents { get; set; }
 }

eng/skill-validator/src/Shared/PathSafety.cs

@@ -0,0 +1,53 @@
+namespace SkillValidator.Shared;
+
+internal static class PathSafety
+{
+    internal static bool ContainsReparsePoint(
+        string allowedRoot,
+        string path,
+        bool missingPathIsUnsafe = true)
+    {
+        var root = Path.TrimEndingDirectorySeparator(Path.GetFullPath(allowedRoot));
+        var fullPath = Path.GetFullPath(path);
+        if (IsReparsePointOrUnreadable(root, missingPathIsUnsafe))
+            return true;
+
+        var relative = Path.GetRelativePath(root, fullPath);
+        if (relative == ".")
+            return false;
+        if (Path.IsPathRooted(relative)
+            || relative == ".."
+            || relative.StartsWith($"..{Path.DirectorySeparatorChar}", StringComparison.Ordinal)
+            || relative.StartsWith($"..{Path.AltDirectorySeparatorChar}", StringComparison.Ordinal))
+        {
+            return true;
+        }
+
+        var current = root;
+        foreach (var segment in relative.Split(
+            [Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar],
+            StringSplitOptions.RemoveEmptyEntries))
+        {
+            current = Path.Combine(current, segment);
+            if (IsReparsePointOrUnreadable(current, missingPathIsUnsafe))
+                return true;
+        }
+        return false;
+    }
+
+    private static bool IsReparsePointOrUnreadable(string path, bool missingPathIsUnsafe)
+    {
+        try
+        {
+            return (File.GetAttributes(path) & FileAttributes.ReparsePoint) != 0;
+        }
+        catch (Exception ex) when (ex is FileNotFoundException or DirectoryNotFoundException)
+        {
+            return missingPathIsUnsafe;
+        }
+        catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
+        {
+            return true;
+        }
+    }
+}

eng/skill-validator/src/Shared/SkillDiscovery.cs

@@ -6,14 +6,19 @@ public static class SkillDiscovery
 {
     private static readonly IDeserializer FrontmatterDeserializer = SkillValidatorYamlContext.UnderscoredDeserializer;
 
-    public static async Task<IReadOnlyList<SkillInfo>> DiscoverSkills(string targetPath)
+    public static async Task<IReadOnlyList<SkillInfo>> DiscoverSkills(
+        string targetPath,
+        string? allowedRoot = null)
     {
+        if (allowedRoot is not null && PathSafety.ContainsReparsePoint(allowedRoot, targetPath))
+            return [];
+
         // If pointing at a SKILL.md file, use its parent directory
         if (File.Exists(targetPath) && Path.GetFileName(targetPath).Equals("SKILL.md", StringComparison.OrdinalIgnoreCase))
             targetPath = Path.GetDirectoryName(targetPath)!;
 
         // Check if the target itself is a skill
-        var directSkill = await DiscoverSkillAt(targetPath);
+        var directSkill = await DiscoverSkillAt(targetPath, allowedRoot);
         if (directSkill is not null)
             return [directSkill];
 
@@ -27,7 +32,7 @@ public static class SkillDiscovery
             if (Path.GetFileName(dir).StartsWith('.'))
                 continue;
 
-            var skill = await DiscoverSkillAt(dir);
+            var skill = await DiscoverSkillAt(dir, allowedRoot);
             if (skill is not null)
                 skills.Add(skill);
         }
@@ -58,11 +63,17 @@ public static class SkillDiscovery
         return skills;
     }
 
-    private static async Task<SkillInfo?> DiscoverSkillAt(string dirPath)
+    private static async Task<SkillInfo?> DiscoverSkillAt(
+        string dirPath,
+        string? allowedRoot = null)
     {
+        if (allowedRoot is not null && PathSafety.ContainsReparsePoint(allowedRoot, dirPath))
+            return null;
         var skillMdPath = Path.Combine(dirPath, "SKILL.md");
         if (!File.Exists(skillMdPath))
             return null;
+        if (allowedRoot is not null && PathSafety.ContainsReparsePoint(allowedRoot, skillMdPath))
+            return null;
 
         var skillMdContent = await File.ReadAllTextAsync(skillMdPath);
         var (metadata, _) = ParseFrontmatter(skillMdContent);

eng/skill-validator/src/SkillValidatorYamlContext.cs

@@ -15,7 +15,11 @@ namespace SkillValidator;
 [YamlSerializable(typeof(EvalSchema.RawSetupFile))]
 [YamlSerializable(typeof(EvalSchema.RawAssertion))]
 [YamlSerializable(typeof(EvalSchema.RawVallyEvalConfig))]
+[YamlSerializable(typeof(EvalSchema.RawVallyDefaults))]
 [YamlSerializable(typeof(EvalSchema.RawVallyStimulus))]
+[YamlSerializable(typeof(EvalSchema.RawVallyEnvironment))]
+[YamlSerializable(typeof(EvalSchema.RawVallyFile))]
+[YamlSerializable(typeof(EvalSchema.RawVallyConstraints))]
 [YamlSerializable(typeof(EvalSchema.RawVallyGrader))]
 [YamlSerializable(typeof(EvalSchema.RawVallyGraderConfig))]
 public partial class SkillValidatorYamlContext : StaticContext

eng/skill-validator/src/docs/InvestigatingResults.md

@@ -1,6 +1,11 @@
 # Investigating Evaluation Results
 
-> **⚠️ Skill evaluations now run on the Vally harness.** As of the Vally migration, the LLM eval pipeline (`evaluation.yml`) no longer uses `skill-validator evaluate`; it runs Vally via `eng/vally-adapter/` and uploads `vally-results-*` artifacts. For investigating current eval failures, use the guide at `eng/vally-adapter/InvestigatingResults.md` in the repository root instead. This document describes the legacy `skill-validator evaluate` schema and is retained for historical results and reference. (The `skill-validator check` **linter** is unaffected and still runs via `skill-check.yml`.)
+> **⚠️ Skill evaluations run on Vally; custom-agent evaluations use this runner
+> as an execution lane.** The CI pipeline adapts native agent results through
+> `eng/vally-adapter/adapt-agent-results.mjs` before publishing them, so use
+> `eng/vally-adapter/InvestigatingResults.md` for the final schema. This
+> document describes the raw `skill-validator evaluate` output retained under
+> `_agent-evaluation/` for custom-agent diagnosis and for historical results.
 >
 > The current Vally workflow makes one targeted recovery attempt for executor
 > `session.idle` timeouts before adaptation. See
@@ -95,7 +100,8 @@ Each verdict contains:
 | Field | Description |
 |-------|-------------|
 | `schemaOwner` / `schemaVersion` | The same legacy schema identity, repeated so standalone `verdict.json` files are self-describing |
-| `skillName` | Name of the skill being evaluated |
+| `skillKind` | `skill` or `agent`; native custom-agent runs set `agent` before CI adaptation |
+| `skillName` | Compatibility field containing the skill or custom-agent name |
 | `passed` | Overall pass/fail |
 | `scenarios[]` | Array of per-scenario comparisons |
 | `overfittingResult` | Overfitting analysis (if enabled) |
@@ -116,9 +122,16 @@ Each scenario includes two required runs (baseline + isolated). It may also incl
 | `isolatedBreakdown` | Per-metric contribution to the score (see below) |
 | `pluginBreakdown` | Per-metric contribution to the score (see below); optional and only populated when a plugin run is present |
 | `pairwiseResult` | Judge's rubric-by-rubric comparison |
-| `perRunScores` | Per-run improvement scores as a flat array of numbers (one per run); when a plugin run is present, each value is `min(isolated, plugin)` for that run; when no plugin run is present (`skilledPlugin` is null), each value is the isolated improvement score for that run |
+| `perRunScores` | Per-run improvement scores used by the statistical gate. Agent evaluations always use isolated-vs-baseline scores because the plugin arm is diagnostic. Skill evaluations use `min(isolated, plugin)` when a plugin run is present and the isolated score otherwise |
 
-> **Note:** Scenarios do not have a `passed` field. To determine pass/fail for an individual scenario, check whether `improvementScore >= 0`. This is the effective score: when no plugin run is present it equals `isolatedImprovementScore`; when a plugin run is present it is the min of isolated and plugin scores. The `passed` field exists only at the verdict level (per-skill).
+> **Note:** Scenarios do not have a `passed` field. To determine pass/fail for an individual scenario, check whether `improvementScore >= 0`. For skills, this effective score is the minimum of isolated and plugin scores when both arms exist. For agents, it is always the isolated score; `pluginImprovementScore` and `pluginBreakdown` remain diagnostic production-surface telemetry. The `passed` field exists only at the verdict level.
+
+> **Agent activation:** Expected target-agent activation in the isolated arm is a verdict gate. Missing target activation in the plugin arm is diagnostic telemetry and is included in logs and reason text, but does not set `skillNotActivated`, change `failureKind`, or fail the verdict.
+
+> **Plugin skill staging:** Plugin runs load staged copies of manifest-declared
+> skills rather than exposing the source directories directly. Skill directories
+> and `SKILL.md` files must remain inside the plugin without symlink/reparse-point
+> components, and linked descendants are omitted while copying the skill tree.
 
 > **Reused baselines:** When the run was invoked with `--baseline-from`, the `baseline` arm is not executed — its `metrics` and `judgeResult` come from the shared baseline file produced earlier with `--baseline-out` (computed once, honoring `--runs`). Such scenarios are reported with the `baseline-reused` session phase and a `reused` baseline status. The baseline file is keyed on `--model` and `--judge-model` plus, per scenario, a SHA-256 of the prompt and a composite SHA-256 over its setup inputs (copied test files, explicit setup files, and setup commands) and its evaluation criteria (rubric, assertions, expect/reject tools, and turn/token/timeout limits); reuse fails fast if the agent model, judge model, or any prompt-plus-setup-plus-criteria identity is missing, so the baseline you compare against is always identity-matched and a shared prompt across cases with different fixtures or rubrics cannot cross-contaminate. Because the baseline output is identical across every skill/agent that consumes the same file, this acts as a shared control group and removes baseline run-to-run variance from cross-skill comparisons.
 

eng/skill-validator/tests/Check/AgentPluginTests.cs

@@ -15,6 +15,135 @@ public class AgentProfilerTests
         return new AgentInfo(name, description, $"/tmp/agents/{fileName}", content, fileName);
     }
 
+    public class AgentDiscoveryPathSafetyTests
+    {
+        [Fact]
+        public async Task PluginDiscoveryRejectsDeclaredAgentFileSymlink()
+        {
+            var root = Path.Combine(Path.GetTempPath(), $"agent-file-link-{Guid.NewGuid():N}");
+            var pluginRoot = Path.Combine(root, "plugin");
+            var agentsDir = Path.Combine(pluginRoot, "agents");
+            var outsideDir = Path.Combine(root, "outside");
+            Directory.CreateDirectory(agentsDir);
+            Directory.CreateDirectory(outsideDir);
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {
+                  "name": "demo",
+                  "version": "1.0.0",
+                  "description": "Demo",
+                  "agents": ["./agents/leak.agent.md"]
+                }
+                """);
+            var outsideAgent = Path.Combine(outsideDir, "leak.agent.md");
+            File.WriteAllText(outsideAgent, """
+                ---
+                name: leak
+                description: External agent.
+                ---
+                External.
+                """);
+            if (!SymlinkTestHelper.TryCreateFile(Path.Combine(agentsDir, "leak.agent.md"), outsideAgent))
+            {
+                Directory.Delete(root, true);
+                return;
+            }
+            try
+            {
+                Assert.Empty(await AgentDiscovery.DiscoverAgentsInPlugin(pluginRoot));
+            }
+            finally
+            {
+                Directory.Delete(root, true);
+            }
+        }
+
+        [Fact]
+        public async Task PluginDiscoveryRejectsDeclaredDirectorySymlink()
+        {
+            var root = Path.Combine(Path.GetTempPath(), $"agent-dir-link-{Guid.NewGuid():N}");
+            var pluginRoot = Path.Combine(root, "plugin");
+            var outsideDir = Path.Combine(root, "outside");
+            Directory.CreateDirectory(pluginRoot);
+            Directory.CreateDirectory(outsideDir);
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {
+                  "name": "demo",
+                  "version": "1.0.0",
+                  "description": "Demo",
+                  "agents": ["./linked/"]
+                }
+                """);
+            File.WriteAllText(Path.Combine(outsideDir, "outside.agent.md"), """
+                ---
+                name: outside
+                description: External agent.
+                ---
+                External.
+                """);
+            if (!SymlinkTestHelper.TryCreateDirectory(Path.Combine(pluginRoot, "linked"), outsideDir))
+            {
+                Directory.Delete(root, true);
+                return;
+            }
+            try
+            {
+                Assert.Empty(await AgentDiscovery.DiscoverAgentsInPlugin(pluginRoot));
+            }
+            finally
+            {
+                Directory.Delete(root, true);
+            }
+        }
+
+        [Fact]
+        public async Task PluginDiscoverySkipsLinkedAgentInsideConventionalDirectory()
+        {
+            var root = Path.Combine(Path.GetTempPath(), $"agent-mixed-link-{Guid.NewGuid():N}");
+            var pluginRoot = Path.Combine(root, "plugin");
+            var agentsDir = Path.Combine(pluginRoot, "agents");
+            var outsideDir = Path.Combine(root, "outside");
+            Directory.CreateDirectory(agentsDir);
+            Directory.CreateDirectory(outsideDir);
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {
+                  "name": "demo",
+                  "version": "1.0.0",
+                  "description": "Demo",
+                  "agents": ["./agents/"]
+                }
+                """);
+            File.WriteAllText(Path.Combine(agentsDir, "real.agent.md"), """
+                ---
+                name: real
+                description: Real agent.
+                ---
+                Real.
+                """);
+            var outsideAgent = Path.Combine(outsideDir, "linked.agent.md");
+            File.WriteAllText(outsideAgent, """
+                ---
+                name: linked
+                description: External agent.
+                ---
+                External.
+                """);
+            if (!SymlinkTestHelper.TryCreateFile(Path.Combine(agentsDir, "linked.agent.md"), outsideAgent))
+            {
+                Directory.Delete(root, true);
+                return;
+            }
+            try
+            {
+                var agent = Assert.Single(await AgentDiscovery.DiscoverAgentsInPlugin(pluginRoot));
+                Assert.Equal("real", agent.Name);
+            }
+            finally
+            {
+                Directory.Delete(root, true);
+            }
+        }
+    }
+
     [Fact]
     public void ValidAgentProducesNoErrors()
     {
@@ -89,6 +218,34 @@ public class AgentProfilerTests
         Assert.DoesNotContain(profile.Errors, e => e.Contains("does not match filename"));
     }
 
+    [Fact]
+    public async Task DiscoveryPreservesDeclaredAgentDependencies()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"agent-discovery-{Guid.NewGuid():N}");
+        Directory.CreateDirectory(root);
+        try
+        {
+            File.WriteAllText(Path.Combine(root, "parent.agent.md"), """
+                ---
+                name: parent
+                description: Parent agent.
+                agents:
+                  - child-a
+                  - child-b
+                ---
+                # Parent
+                """);
+
+            var agent = Assert.Single(await AgentDiscovery.DiscoverAgentsInDirectory(root));
+
+            Assert.Equal(["child-a", "child-b"], agent.Agents);
+        }
+        finally
+        {
+            Directory.Delete(root, true);
+        }
+    }
+
     [Fact]
     public void NameWithUppercaseErrors()
     {

eng/skill-validator/tests/Evaluate/AssertionsTests.cs

@@ -1,3 +1,4 @@
+using System.Text.Json.Nodes;
 using SkillValidator.Evaluate;
 using SkillValidator.Shared;
 
@@ -63,6 +64,130 @@ public class EvaluateAssertionsTests
         Assert.False(results[0].Passed);
     }
 
+    [Fact]
+    public async Task ExitSuccessFailsWhenRunTimedOutAfterOutput()
+    {
+        var metrics = new RunMetrics
+        {
+            TimedOut = true,
+            AgentOutput = "partial output",
+            Events = [new AgentEvent("assistant.message", 0, [])],
+        };
+
+        var results = await AssertionEvaluator.EvaluateAssertions(
+            [new Assertion(AssertionType.ExitSuccess)],
+            metrics.AgentOutput,
+            WorkDir,
+            metrics: metrics);
+
+        Assert.False(results[0].Passed);
+        Assert.Contains("timed out", results[0].Message);
+    }
+
+    [Fact]
+    public async Task ExitSuccessFailsWhenRunRecordedErrorAfterOutput()
+    {
+        var metrics = new RunMetrics
+        {
+            ErrorCount = 1,
+            AgentOutput = "partial output",
+            Events = [new AgentEvent("session.idle", 0, [])],
+        };
+
+        var results = await AssertionEvaluator.EvaluateAssertions(
+            [new Assertion(AssertionType.ExitSuccess)],
+            metrics.AgentOutput,
+            WorkDir,
+            metrics: metrics);
+
+        Assert.False(results[0].Passed);
+        Assert.Contains("recorded 1 error", results[0].Message);
+    }
+
+    [Fact]
+    public async Task ExitSuccessFailsAfterUnsuccessfulToolCompletionAndIdle()
+    {
+        var events = new List<AgentEvent>
+        {
+            new(
+                "tool.execution_complete",
+                0,
+                new Dictionary<string, JsonNode?>
+                {
+                    ["success"] = JsonValue.Create(false),
+                    ["result"] = JsonValue.Create("command failed"),
+                }),
+            new("session.idle", 1, []),
+        };
+        var metrics = MetricsCollector.CollectMetrics(
+            events, "partial output", 1000, "/tmp/work");
+
+        var results = await AssertionEvaluator.EvaluateAssertions(
+            [new Assertion(AssertionType.ExitSuccess)],
+            "partial output",
+            "/tmp/work",
+            metrics: metrics);
+
+        Assert.False(results[0].Passed);
+        Assert.Contains("recorded 1 error", results[0].Message);
+    }
+
+    [Fact]
+    public async Task ExitSuccessRequiresSessionIdleWhenMetricsProvided()
+    {
+        var metrics = new RunMetrics
+        {
+            AgentOutput = "partial output",
+            Events = [new AgentEvent("assistant.message", 0, [])],
+        };
+
+        var results = await AssertionEvaluator.EvaluateAssertions(
+            [new Assertion(AssertionType.ExitSuccess)],
+            metrics.AgentOutput,
+            WorkDir,
+            metrics: metrics);
+
+        Assert.False(results[0].Passed);
+        Assert.Contains("did not reach session.idle", results[0].Message);
+    }
+
+    [Fact]
+    public async Task ExitSuccessFailsClosedWhenEventsAreMissing()
+    {
+        var metrics = new RunMetrics
+        {
+            AgentOutput = "partial output",
+            Events = null!,
+        };
+
+        var results = await AssertionEvaluator.EvaluateAssertions(
+            [new Assertion(AssertionType.ExitSuccess)],
+            metrics.AgentOutput,
+            WorkDir,
+            metrics: metrics);
+
+        Assert.False(results[0].Passed);
+        Assert.Contains("did not reach session.idle", results[0].Message);
+    }
+
+    [Fact]
+    public async Task ExitSuccessPassesForCleanIdleRun()
+    {
+        var metrics = new RunMetrics
+        {
+            AgentOutput = "complete output",
+            Events = [new AgentEvent("session.idle", 0, [])],
+        };
+
+        var results = await AssertionEvaluator.EvaluateAssertions(
+            [new Assertion(AssertionType.ExitSuccess)],
+            metrics.AgentOutput,
+            WorkDir,
+            metrics: metrics);
+
+        Assert.True(results[0].Passed);
+    }
+
     [Fact]
     public async Task HandlesMultipleAssertions()
     {

eng/skill-validator/tests/Evaluate/BaselineStoreTests.cs

@@ -348,6 +348,69 @@ public class BaselineStoreTests
         }
     }
 
+    [Fact]
+    public void ComputeTargetSha_IncludesExplicitDirectorySourceContents()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"sv-explicit-dir-{Guid.NewGuid():N}");
+        var evalDir = Path.Combine(root, "tests", "demo", "agent.router");
+        var sourceDir = Path.Combine(evalDir, "fixtures", "project");
+        Directory.CreateDirectory(Path.Combine(sourceDir, "nested"));
+        var evalPath = Path.Combine(evalDir, "eval.yaml");
+        File.WriteAllText(evalPath, "stimuli: []");
+        var nestedFile = Path.Combine(sourceDir, "nested", "data.txt");
+        File.WriteAllText(nestedFile, "v1");
+        var scenario = new EvalScenario(
+            "s",
+            "inspect project",
+            new SetupConfig(Files: [new SetupFile("Project", "fixtures/project")]));
+        try
+        {
+            var before = BaselineStore.ComputeTargetSha(scenario, evalPath);
+            File.WriteAllText(nestedFile, "v2");
+            var after = BaselineStore.ComputeTargetSha(scenario, evalPath);
+
+            Assert.NotEqual(before, after);
+        }
+        finally
+        {
+            Directory.Delete(root, recursive: true);
+        }
+    }
+
+    [Fact]
+    public void ComputeTargetSha_DistinguishesReplacementDirectorySources()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"sv-replacement-dir-{Guid.NewGuid():N}");
+        var evalDir = Path.Combine(root, "tests", "demo", "agent.router");
+        var sourceA = Path.Combine(evalDir, "fixtures", "project-a");
+        var sourceB = Path.Combine(evalDir, "fixtures", "project-b");
+        Directory.CreateDirectory(sourceA);
+        Directory.CreateDirectory(sourceB);
+        var evalPath = Path.Combine(evalDir, "eval.yaml");
+        File.WriteAllText(evalPath, "stimuli: []");
+        File.WriteAllText(Path.Combine(sourceA, "data.txt"), "A");
+        File.WriteAllText(Path.Combine(sourceB, "data.txt"), "B");
+        try
+        {
+            var scenarioA = new EvalScenario(
+                "s",
+                "inspect project",
+                new SetupConfig(Files: [new SetupFile("Project", "fixtures/project-a")]));
+            var scenarioB = scenarioA with
+            {
+                Setup = new SetupConfig(Files: [new SetupFile("Project", "fixtures/project-b")]),
+            };
+
+            Assert.NotEqual(
+                BaselineStore.ComputeTargetSha(scenarioA, evalPath),
+                BaselineStore.ComputeTargetSha(scenarioB, evalPath));
+        }
+        finally
+        {
+            Directory.Delete(root, recursive: true);
+        }
+    }
+
     [Fact]
     public void Clone_ProducesIndependentCopy()
     {

eng/skill-validator/tests/Evaluate/ComparatorTests.cs

@@ -223,6 +223,38 @@ public class ComputeVerdictTests
         Assert.Contains("regressed", verdict.Reason);
     }
 
+    [Fact]
+    public void AgentVerdictUsesIsolatedArmForGateAndPluginAsDiagnostic()
+    {
+        var baseline = MakeRunResult(taskCompleted: true, tokenEstimate: 1000, overallScore: 3);
+        var isolated = MakeRunResult(taskCompleted: true, tokenEstimate: 500, overallScore: 5);
+        var plugin = MakeRunResult(taskCompleted: false, tokenEstimate: 2000, overallScore: 1);
+        var isolatedComparison = Comparator.CompareScenario("test", baseline, isolated);
+        var pluginComparison = Comparator.CompareScenario("test", baseline, plugin);
+        var comparison = new ScenarioComparison
+        {
+            ScenarioName = "test",
+            Baseline = baseline,
+            SkilledIsolated = isolated,
+            SkilledPlugin = plugin,
+            ImprovementScore = isolatedComparison.ImprovementScore,
+            IsolatedImprovementScore = isolatedComparison.ImprovementScore,
+            PluginImprovementScore = pluginComparison.ImprovementScore,
+            Breakdown = isolatedComparison.Breakdown,
+            IsolatedBreakdown = isolatedComparison.Breakdown,
+            PluginBreakdown = pluginComparison.Breakdown,
+            PerRunScores = [isolatedComparison.ImprovementScore],
+        };
+
+        var verdict = Comparator.ComputeAgentVerdict(MockSkill, [comparison], 0.1, true);
+
+        Assert.True(verdict.Passed);
+        Assert.Equal(isolatedComparison.ImprovementScore, verdict.OverallImprovementScore);
+        Assert.Equal(isolatedComparison.ImprovementScore, verdict.IsolatedScore);
+        Assert.Equal(pluginComparison.ImprovementScore, verdict.PluginScore);
+        Assert.True(verdict.NormalizedGain > 0);
+    }
+
     [Fact]
     public void CompareScenarioSetsPluginToNull()
     {

eng/skill-validator/tests/Evaluate/EvalDiscoveryTests.cs

@@ -87,6 +87,41 @@ public class EvalDiscoveryTests
         }
     }
 
+    [Fact]
+    public async Task RejectsReferencedMcpFileThroughSymlink()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"skill-mcp-link-{Guid.NewGuid():N}");
+        var pluginDir = Path.Combine(root, "plugin");
+        var skillDir = Path.Combine(pluginDir, "skills", "my-skill");
+        var outsideMcp = Path.Combine(root, "outside.mcp.json");
+        var linkedMcp = Path.Combine(pluginDir, ".mcp.json");
+        Directory.CreateDirectory(skillDir);
+        await File.WriteAllTextAsync(
+            Path.Combine(pluginDir, "plugin.json"),
+            """{"mcpServers":"./.mcp.json"}""",
+            TestContext.Current.CancellationToken);
+        await File.WriteAllTextAsync(
+            outsideMcp,
+            """{"mcpServers":{"external":{"command":"dotnet","args":["run"]}}}""",
+            TestContext.Current.CancellationToken);
+        if (!SymlinkTestHelper.TryCreateFile(linkedMcp, outsideMcp))
+        {
+            Directory.Delete(root, true);
+            return;
+        }
+
+        try
+        {
+            var result = await EvaluateCommand.FindPluginMcpServers(skillDir);
+
+            Assert.Null(result);
+        }
+        finally
+        {
+            Directory.Delete(root, true);
+        }
+    }
+
     [Fact]
     public async Task ResolveEvalPathFindsNestedTestDir()
     {

eng/skill-validator/tests/Evaluate/EvalSchemaTests.cs

@@ -241,6 +241,183 @@ public class ParseEvalConfigTests
         Assert.Contains("expected_exit_code", ex.Message);
         Assert.Contains("expected_std_output_contains", ex.Message);
     }
+
+    [Fact]
+    public void ParsesVallyAgentEvalForNativeExecution()
+    {
+        var yaml = """
+            name: agent.sample
+            defaults:
+              timeout: 20m
+            stimuli:
+              - name: Migrate the project
+                prompt: Apply the requested migration.
+                expect_activation: false
+                environment:
+                  files:
+                    - src: fixtures/project
+                      dest: Project
+                  commands:
+                    - git init -q
+                  skills:
+                    - ../../plugins/demo/skills/migrate
+                  agents:
+                    - helper-agent
+                constraints:
+                  expect_tools: [bash]
+                  reject_tools: [web]
+                  max_turns: 12
+                  max_tokens: 4000
+                graders:
+                  - type: file-contains
+                    config:
+                      path: "**/*.csproj"
+                      value: "4."
+                  - type: run-command
+                    config:
+                      command: dotnet test Project
+                      expected_exit_code: 0
+                      timeout: 5m
+                      stdout_contains: Passed!
+                      stdout_matches: Passed
+                  - type: prompt
+                rubric:
+                  - Completed the migration
+            """;
+
+        var config = EvalSchema.ParseEvalConfigFlexible(yaml);
+
+        Assert.NotNull(config);
+        var scenario = Assert.Single(config!.Scenarios);
+        Assert.Equal(1200, scenario.Timeout);
+        Assert.False(scenario.ExpectActivation);
+        Assert.Equal(["bash"], scenario.ExpectTools);
+        Assert.Equal(["web"], scenario.RejectTools);
+        Assert.Equal(12, scenario.MaxTurns);
+        Assert.Equal(4000, scenario.MaxTokens);
+        Assert.Equal("../../plugins/demo/skills/migrate", Assert.Single(scenario.Setup!.AdditionalRequiredSkills!));
+        Assert.Equal("helper-agent", Assert.Single(scenario.Setup.AdditionalRequiredAgents!));
+        var file = Assert.Single(scenario.Setup.Files!);
+        Assert.Equal("fixtures/project", file.Source);
+        Assert.Equal("Project", file.Path);
+        Assert.Equal("git init -q", Assert.Single(scenario.Setup.Commands!));
+        Assert.Equal(2, scenario.Assertions!.Count);
+        Assert.Equal(AssertionType.FileContains, scenario.Assertions[0].Type);
+        Assert.Equal(AssertionType.RunCommandAndAssert, scenario.Assertions[1].Type);
+        var command = scenario.Assertions[1].CommandArgs;
+        Assert.NotNull(command);
+        Assert.Equal(300, command!.Timeout);
+        Assert.Equal("Passed!", command.ExpectedStdOutContains);
+        Assert.Equal("Passed", command.ExpectedStdOutMatches);
+        if (OperatingSystem.IsWindows())
+        {
+            Assert.Equal("/d /s /c \"dotnet test Project\"", command.CommandArguments);
+            Assert.Null(command.ArgumentList);
+        }
+        else
+        {
+            Assert.Equal(["-c", "dotnet test Project"], command.ArgumentList!);
+            Assert.Null(command.CommandArguments);
+        }
+    }
+
+    [Fact]
+    public async Task VallyRunCommandPreservesNestedQuotes()
+    {
+        var shellCommand = OperatingSystem.IsWindows()
+            ? """powershell -NoLogo -NoProfile -Command "$value = 'quoted value'; if ($value -ne 'quoted value') { exit 1 }" """
+            : """sh -c "test \"quoted value\" = \"quoted value\"" """;
+        var yaml = $$"""
+            name: nested-quotes
+            stimuli:
+              - name: Execute nested quotes
+                prompt: Run the check.
+                graders:
+                  - type: run-command
+                    config:
+                      command: >-
+                        {{shellCommand}}
+                      expected_exit_code: 0
+            """;
+        var config = EvalSchema.ParseEvalConfigFlexible(yaml);
+        var assertion = Assert.Single(Assert.Single(config!.Scenarios).Assertions!);
+        var command = assertion.CommandArgs;
+        Assert.NotNull(command);
+        if (OperatingSystem.IsWindows())
+            Assert.Contains(shellCommand, command!.CommandArguments);
+        else
+            Assert.Equal(shellCommand, command!.ArgumentList![1]);
+
+        var workDir = Path.Combine(Path.GetTempPath(), $"nested-command-{Guid.NewGuid():N}");
+        Directory.CreateDirectory(workDir);
+        try
+        {
+            var result = Assert.Single(await AssertionEvaluator.EvaluateAssertions(
+                [assertion], "", workDir));
+            Assert.True(result.Passed, result.Message);
+        }
+        finally
+        {
+            Directory.Delete(workDir, true);
+        }
+    }
+
+    [Theory]
+    [InlineData("""powershell -NoLogo -NoProfile -Command "exit 7" """)]
+    [InlineData("""powershell -NoLogo -NoProfile -Command "throw 'must fail'" """)]
+    public async Task VallyRunCommandPreservesQuotedWindowsFailures(string shellCommand)
+    {
+        if (!OperatingSystem.IsWindows())
+            return;
+
+        var yaml = $$"""
+            name: quoted-windows-failure
+            stimuli:
+              - name: Execute quoted failure
+                prompt: Run the check.
+                graders:
+                  - type: run-command
+                    config:
+                      command: >-
+                        {{shellCommand}}
+            """;
+        var config = EvalSchema.ParseEvalConfigFlexible(yaml);
+        var assertion = Assert.Single(Assert.Single(config!.Scenarios).Assertions!);
+        var workDir = Path.Combine(Path.GetTempPath(), $"quoted-command-{Guid.NewGuid():N}");
+        Directory.CreateDirectory(workDir);
+        try
+        {
+            var result = Assert.Single(await AssertionEvaluator.EvaluateAssertions(
+                [assertion], "", workDir));
+            Assert.False(result.Passed, result.Message);
+        }
+        finally
+        {
+            Directory.Delete(workDir, true);
+        }
+    }
+
+    [Theory]
+    [InlineData("90", 90)]
+    [InlineData("1500ms", 2)]
+    [InlineData("2m", 120)]
+    [InlineData("1h", 3600)]
+    public void ParsesVallyDurations(string value, int expectedSeconds)
+    {
+        Assert.Equal(expectedSeconds, EvalSchema.ParseDurationSeconds(value));
+    }
+
+    [Theory]
+    [InlineData("2147483648s")]
+    [InlineData("9223372036854775807m")]
+    [InlineData("9223372036854775807h")]
+    public void RejectsVallyDurationsThatOverflowSeconds(string value)
+    {
+        var error = Assert.Throws<InvalidOperationException>(
+            () => EvalSchema.ParseDurationSeconds(value));
+
+        Assert.Contains("exceeds the supported maximum", error.Message);
+    }
 }
 
 public class ValidateEvalConfigTests

eng/skill-validator/tests/Evaluate/EvaluateCommandTests.cs

@@ -1,4 +1,5 @@
 using SkillValidator.Evaluate;
+using SkillValidator.Shared;
 
 namespace SkillValidator.Tests;
 
@@ -37,4 +38,588 @@ public class EvaluateCommandTests
 
         Assert.Equal(1, exitCode);
     }
+
+    [Fact]
+    public void CreateSkillPluginRunOptionsPreservesDeclaredAgentDependencies()
+    {
+        var scenario = new EvalScenario("scenario", "prompt");
+        var skill = new SkillInfo("target", "Target", "skill", "skill/SKILL.md", "# Target");
+        var evalSkill = new EvalSkillInfo(skill, "tests/demo/target/eval.yaml", null);
+        var dependency = new AgentInfo(
+            "helper",
+            "Helper",
+            "plugins/demo/agents/helper.agent.md",
+            "---\nname: helper\ndescription: Helper\n---\nHelp.",
+            "plugins/demo/agents/helper.agent.md");
+
+        var options = EvaluateCommand.CreateSkillPluginRunOptions(
+            scenario,
+            evalSkill,
+            new ValidatorConfig { Model = "gpt-4.1" },
+            "plugins/demo",
+            _ => { },
+            sessionsDir: null,
+            pluginSessionId: "plugin-session",
+            additionalAgents: [dependency]);
+
+        Assert.Same(dependency, Assert.Single(options.AdditionalAgents!));
+        Assert.Equal("plugins/demo", options.PluginRoot);
+        Assert.Equal("plugin-session", options.SessionId);
+    }
+
+    [Fact]
+    public void AgentPluginActivationIsDiagnosticOnly()
+    {
+        var run = new RunResult(
+            new RunMetrics { AgentOutput = "done", TaskCompleted = true, Events = [] },
+            new JudgeResult([], 5, "passed"));
+        var comparison = new ScenarioComparison
+        {
+            ScenarioName = "route work",
+            Baseline = run,
+            SkilledIsolated = run,
+            SkilledPlugin = run,
+            ImprovementScore = 0.5,
+            Breakdown = new MetricBreakdown(0, 0, 0, 0, 0, 0, 0),
+            SubagentActivationIsolated = new SubagentActivationInfo(["router"], 1),
+            SubagentActivationPlugin = new SubagentActivationInfo(["other-agent"], 1),
+        };
+        var verdict = new SkillVerdict
+        {
+            SkillName = "router",
+            SkillPath = "plugins/demo/agents/router.agent.md",
+            Passed = true,
+            Scenarios = [comparison],
+            OverallImprovementScore = 0.5,
+            Reason = "passed",
+        };
+
+        EvaluateCommand.ApplyAgentActivationGate(verdict, [comparison], "router", _ => { });
+
+        Assert.True(verdict.Passed);
+        Assert.False(verdict.SkillNotActivated);
+        Assert.Null(verdict.FailureKind);
+        Assert.Contains("AGENT NOT ACTIVATED (plugin)", verdict.Reason);
+    }
+
+    [Fact]
+    public void AgentIsolatedActivationRemainsAuthoritative()
+    {
+        var run = new RunResult(
+            new RunMetrics { AgentOutput = "done", TaskCompleted = true, Events = [] },
+            new JudgeResult([], 5, "passed"));
+        var comparison = new ScenarioComparison
+        {
+            ScenarioName = "route work",
+            Baseline = run,
+            SkilledIsolated = run,
+            SkilledPlugin = run,
+            ImprovementScore = 0.5,
+            Breakdown = new MetricBreakdown(0, 0, 0, 0, 0, 0, 0),
+            SubagentActivationIsolated = new SubagentActivationInfo(["other-agent"], 1),
+            SubagentActivationPlugin = new SubagentActivationInfo(["router"], 1),
+        };
+        var verdict = new SkillVerdict
+        {
+            SkillName = "router",
+            SkillPath = "plugins/demo/agents/router.agent.md",
+            Passed = true,
+            Scenarios = [comparison],
+            OverallImprovementScore = 0.5,
+            Reason = "passed",
+        };
+
+        EvaluateCommand.ApplyAgentActivationGate(verdict, [comparison], "router", _ => { });
+
+        Assert.False(verdict.Passed);
+        Assert.True(verdict.SkillNotActivated);
+        Assert.Equal(FailureKind.SkillNotActivated, verdict.FailureKind);
+        Assert.Contains("AGENT NOT ACTIVATED (isolated)", verdict.Reason);
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalAgentsIncludesTransitiveDeclaredDependencies()
+    {
+        var pluginRoot = Path.Combine(Path.GetTempPath(), $"agent-deps-{Guid.NewGuid():N}");
+        var agentsDir = Path.Combine(pluginRoot, "agents");
+        Directory.CreateDirectory(agentsDir);
+        try
+        {
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {
+                  "name": "demo",
+                  "version": "1.0.0",
+                  "description": "Demo",
+                  "agents": ["./agents/"]
+                }
+                """);
+            File.WriteAllText(Path.Combine(agentsDir, "coordinator.agent.md"), """
+                ---
+                name: coordinator
+                description: Coordinates work.
+                agents:
+                  - worker
+                ---
+                Coordinate.
+                """);
+            File.WriteAllText(Path.Combine(agentsDir, "worker.agent.md"), """
+                ---
+                name: worker
+                description: Does work.
+                ---
+                Work.
+                """);
+
+            var agents = await EvaluateCommand.ResolveAdditionalAgents(
+                ["coordinator"], pluginRoot);
+
+            Assert.Equal(
+                ["coordinator", "worker"],
+                agents!.Select(agent => agent.Name));
+        }
+        finally
+        {
+            Directory.Delete(pluginRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalAgentsAcceptsAgentFilePath()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"agent-file-dep-{Guid.NewGuid():N}");
+        var pluginRoot = Path.Combine(repoRoot, "plugins", "demo");
+        var agentsDir = Path.Combine(pluginRoot, "agents");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        Directory.CreateDirectory(agentsDir);
+        Directory.CreateDirectory(evalDir);
+        try
+        {
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {
+                  "name": "demo",
+                  "version": "1.0.0",
+                  "description": "Demo",
+                  "agents": ["./agents/"]
+                }
+                """);
+            File.WriteAllText(Path.Combine(agentsDir, "helper.agent.md"), """
+                ---
+                name: helper
+                description: Helper agent.
+                ---
+                Help.
+                """);
+            var evalPath = Path.Combine(evalDir, "eval.yaml");
+            File.WriteAllText(evalPath, "stimuli: []");
+
+            var agents = await EvaluateCommand.ResolveAdditionalAgents(
+                ["../../plugins/demo/agents/helper.agent.md"], pluginRoot, evalPath);
+
+            Assert.Equal("helper", Assert.Single(agents!).Name);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalSkillsAcceptsTrackedStyleCrossPluginPath()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"skill-deps-{Guid.NewGuid():N}");
+        var targetPlugin = Path.Combine(repoRoot, "plugins", "target");
+        var dependency = Path.Combine(repoRoot, "plugins", "shared", "skills", "helper");
+        var evalDir = Path.Combine(repoRoot, "tests", "target", "agent.router");
+        Directory.CreateDirectory(targetPlugin);
+        Directory.CreateDirectory(dependency);
+        Directory.CreateDirectory(evalDir);
+        try
+        {
+            File.WriteAllText(Path.Combine(targetPlugin, "plugin.json"), """
+                {
+                  "name": "target",
+                  "version": "1.0.0",
+                  "description": "Target",
+                  "skills": ["./skills/"]
+                }
+                """);
+            File.WriteAllText(Path.Combine(dependency, "SKILL.md"), """
+                ---
+                name: helper
+                description: Helper skill.
+                ---
+                Help.
+                """);
+            var evalPath = Path.Combine(evalDir, "eval.yaml");
+            File.WriteAllText(evalPath, "stimuli: []");
+
+            var skills = await EvaluateCommand.ResolveAdditionalSkills(
+                ["../../plugins/shared/skills/helper"], targetPlugin, evalPath);
+
+            Assert.Equal("helper", Assert.Single(skills!).Name);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalSkillsDoesNotAnchorPartialPluginsSegment()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"partial-plugins-segment-{Guid.NewGuid():N}");
+        var targetPlugin = Path.Combine(repoRoot, "plugins", "target");
+        var realDependency = Path.Combine(repoRoot, "plugins", "shared", "skills", "helper");
+        var evalDir = Path.Combine(repoRoot, "tests", "target", "agent.router");
+        Directory.CreateDirectory(targetPlugin);
+        Directory.CreateDirectory(realDependency);
+        Directory.CreateDirectory(evalDir);
+        try
+        {
+            File.WriteAllText(Path.Combine(targetPlugin, "plugin.json"), """
+                {"name":"target","version":"1.0.0","description":"Target","skills":["./skills/"]}
+                """);
+            File.WriteAllText(Path.Combine(realDependency, "SKILL.md"), """
+                ---
+                name: helper
+                description: Helper skill.
+                ---
+                Help.
+                """);
+            var evalPath = Path.Combine(evalDir, "eval.yaml");
+            File.WriteAllText(evalPath, "stimuli: []");
+
+            var error = await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalSkills(
+                    ["../../myplugins/shared/skills/helper"], targetPlugin, evalPath));
+
+            Assert.Contains("resolves outside the repository plugins directory", error.Message);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalSkillsRejectsLinkedNamedSkill()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"named-skill-link-{Guid.NewGuid():N}");
+        var pluginRoot = Path.Combine(repoRoot, "plugins", "target");
+        var skillsDir = Path.Combine(pluginRoot, "skills");
+        var outsideSkill = Path.Combine(repoRoot, "outside", "helper");
+        Directory.CreateDirectory(skillsDir);
+        Directory.CreateDirectory(outsideSkill);
+        File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+            {"name":"target","version":"1.0.0","description":"Target","skills":["./skills/"]}
+            """);
+        File.WriteAllText(Path.Combine(outsideSkill, "SKILL.md"), """
+            ---
+            name: helper
+            description: External helper.
+            ---
+            Help.
+            """);
+        if (!SymlinkTestHelper.TryCreateDirectory(
+            Path.Combine(skillsDir, "helper"),
+            outsideSkill))
+        {
+            Directory.Delete(repoRoot, true);
+            return;
+        }
+
+        try
+        {
+            var error = await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalSkills(["helper"], pluginRoot));
+
+            Assert.Contains("name 'helper'", error.Message);
+            Assert.Contains("could not be resolved", error.Message);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalSkillsExplainsMissingNameAndPathReferences()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"missing-skill-deps-{Guid.NewGuid():N}");
+        var pluginRoot = Path.Combine(repoRoot, "plugins", "demo");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        Directory.CreateDirectory(Path.Combine(pluginRoot, "skills"));
+        Directory.CreateDirectory(evalDir);
+        try
+        {
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {"name":"demo","version":"1.0.0","description":"Demo","skills":["./skills/"]}
+                """);
+            var evalPath = Path.Combine(evalDir, "eval.yaml");
+            File.WriteAllText(evalPath, "stimuli: []");
+
+            var nameError = await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalSkills(["missing"], pluginRoot, evalPath));
+            Assert.Contains("name 'missing'", nameError.Message);
+            Assert.Contains("bare skill name", nameError.Message);
+
+            var pathError = await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalSkills(
+                    ["../../plugins/demo/skills/missing"], pluginRoot, evalPath));
+            Assert.Contains("path '../../plugins/demo/skills/missing'", pathError.Message);
+            Assert.Contains("../../plugins/<plugin>/skills/<skill>", pathError.Message);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalAgentsExplainsMissingNameAndPathReferences()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"missing-agent-deps-{Guid.NewGuid():N}");
+        var pluginRoot = Path.Combine(repoRoot, "plugins", "demo");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        Directory.CreateDirectory(Path.Combine(pluginRoot, "agents"));
+        Directory.CreateDirectory(evalDir);
+        try
+        {
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {"name":"demo","version":"1.0.0","description":"Demo","agents":["./agents/"]}
+                """);
+            var evalPath = Path.Combine(evalDir, "eval.yaml");
+            File.WriteAllText(evalPath, "stimuli: []");
+
+            var nameError = await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalAgents(["missing"], pluginRoot, evalPath));
+            Assert.Contains("name 'missing'", nameError.Message);
+            Assert.Contains("bare agent name", nameError.Message);
+
+            var pathError = await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalAgents(
+                    ["../../plugins/demo/agents/missing.agent.md"], pluginRoot, evalPath));
+            Assert.Contains("path '../../plugins/demo/agents/missing.agent.md'", pathError.Message);
+            Assert.Contains("../../plugins/<plugin>/agents/<agent>.agent.md", pathError.Message);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalSkillsExplainsAmbiguousDirectoryPath()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"ambiguous-skill-deps-{Guid.NewGuid():N}");
+        var pluginRoot = Path.Combine(repoRoot, "plugins", "demo");
+        var skillsDir = Path.Combine(pluginRoot, "skills");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        Directory.CreateDirectory(Path.Combine(skillsDir, "first"));
+        Directory.CreateDirectory(Path.Combine(skillsDir, "second"));
+        Directory.CreateDirectory(evalDir);
+        try
+        {
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {"name":"demo","version":"1.0.0","description":"Demo","skills":["./skills/"]}
+                """);
+            File.WriteAllText(Path.Combine(skillsDir, "first", "SKILL.md"), """
+                ---
+                name: first
+                description: First skill.
+                ---
+                First.
+                """);
+            File.WriteAllText(Path.Combine(skillsDir, "second", "SKILL.md"), """
+                ---
+                name: second
+                description: Second skill.
+                ---
+                Second.
+                """);
+            var evalPath = Path.Combine(evalDir, "eval.yaml");
+            File.WriteAllText(evalPath, "stimuli: []");
+
+            var error = await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalSkills(
+                    ["../../plugins/demo/skills"], pluginRoot, evalPath));
+
+            Assert.Contains(Path.GetFullPath(skillsDir), error.Message);
+            Assert.Contains("'first'", error.Message);
+            Assert.Contains("'second'", error.Message);
+            Assert.Contains("Point to a specific skill directory", error.Message);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalAgentsExplainsAmbiguousDirectoryPath()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"ambiguous-agent-deps-{Guid.NewGuid():N}");
+        var pluginRoot = Path.Combine(repoRoot, "plugins", "demo");
+        var agentsDir = Path.Combine(pluginRoot, "agents");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        Directory.CreateDirectory(agentsDir);
+        Directory.CreateDirectory(evalDir);
+        try
+        {
+            File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+                {"name":"demo","version":"1.0.0","description":"Demo","agents":["./agents/"]}
+                """);
+            File.WriteAllText(Path.Combine(agentsDir, "first.agent.md"), """
+                ---
+                name: first
+                description: First agent.
+                ---
+                First.
+                """);
+            File.WriteAllText(Path.Combine(agentsDir, "second.agent.md"), """
+                ---
+                name: second
+                description: Second agent.
+                ---
+                Second.
+                """);
+            var evalPath = Path.Combine(evalDir, "eval.yaml");
+            File.WriteAllText(evalPath, "stimuli: []");
+
+            var error = await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalAgents(
+                    ["../../plugins/demo/agents"], pluginRoot, evalPath));
+
+            Assert.Contains(Path.GetFullPath(agentsDir), error.Message);
+            Assert.Contains("'first'", error.Message);
+            Assert.Contains("'second'", error.Message);
+            Assert.Contains("Point to a specific agent file", error.Message);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalSkillsRejectsLinkedDirectory()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"skill-dir-link-{Guid.NewGuid():N}");
+        var targetPlugin = Path.Combine(repoRoot, "plugins", "target");
+        var sharedPlugin = Path.Combine(repoRoot, "plugins", "shared");
+        var outsideSkill = Path.Combine(repoRoot, "outside", "helper");
+        var evalDir = Path.Combine(repoRoot, "tests", "target", "agent.router");
+        Directory.CreateDirectory(targetPlugin);
+        Directory.CreateDirectory(sharedPlugin);
+        Directory.CreateDirectory(outsideSkill);
+        Directory.CreateDirectory(evalDir);
+        File.WriteAllText(Path.Combine(targetPlugin, "plugin.json"), """
+            {"name":"target","version":"1.0.0","description":"Target","skills":["./skills/"]}
+            """);
+        File.WriteAllText(Path.Combine(outsideSkill, "SKILL.md"), """
+            ---
+            name: helper
+            description: External helper.
+            ---
+            Help.
+            """);
+        if (!SymlinkTestHelper.TryCreateDirectory(Path.Combine(sharedPlugin, "linked"), outsideSkill))
+        {
+            Directory.Delete(repoRoot, true);
+            return;
+        }
+        var evalPath = Path.Combine(evalDir, "eval.yaml");
+        File.WriteAllText(evalPath, "stimuli: []");
+        try
+        {
+            await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalSkills(
+                    ["../../plugins/shared/linked"], targetPlugin, evalPath));
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalSkillsRejectsLinkedSkillFile()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"skill-file-link-{Guid.NewGuid():N}");
+        var targetPlugin = Path.Combine(repoRoot, "plugins", "target");
+        var dependency = Path.Combine(repoRoot, "plugins", "shared", "skills", "helper");
+        var outsideFile = Path.Combine(repoRoot, "outside", "SKILL.md");
+        var evalDir = Path.Combine(repoRoot, "tests", "target", "agent.router");
+        Directory.CreateDirectory(targetPlugin);
+        Directory.CreateDirectory(dependency);
+        Directory.CreateDirectory(Path.GetDirectoryName(outsideFile)!);
+        Directory.CreateDirectory(evalDir);
+        File.WriteAllText(Path.Combine(targetPlugin, "plugin.json"), """
+            {"name":"target","version":"1.0.0","description":"Target","skills":["./skills/"]}
+            """);
+        File.WriteAllText(outsideFile, """
+            ---
+            name: helper
+            description: External helper.
+            ---
+            Help.
+            """);
+        if (!SymlinkTestHelper.TryCreateFile(Path.Combine(dependency, "SKILL.md"), outsideFile))
+        {
+            Directory.Delete(repoRoot, true);
+            return;
+        }
+        var evalPath = Path.Combine(evalDir, "eval.yaml");
+        File.WriteAllText(evalPath, "stimuli: []");
+        try
+        {
+            await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalSkills(
+                    ["../../plugins/shared/skills/helper"], targetPlugin, evalPath));
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public async Task ResolveAdditionalAgentsRejectsLinkedAgentFile()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"agent-dep-link-{Guid.NewGuid():N}");
+        var pluginRoot = Path.Combine(repoRoot, "plugins", "demo");
+        var agentsDir = Path.Combine(pluginRoot, "agents");
+        var outsideFile = Path.Combine(repoRoot, "outside", "helper.agent.md");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        Directory.CreateDirectory(agentsDir);
+        Directory.CreateDirectory(Path.GetDirectoryName(outsideFile)!);
+        Directory.CreateDirectory(evalDir);
+        File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+            {"name":"demo","version":"1.0.0","description":"Demo","agents":["./agents/"]}
+            """);
+        File.WriteAllText(outsideFile, """
+            ---
+            name: helper
+            description: External helper.
+            ---
+            Help.
+            """);
+        if (!SymlinkTestHelper.TryCreateFile(Path.Combine(agentsDir, "helper.agent.md"), outsideFile))
+        {
+            Directory.Delete(repoRoot, true);
+            return;
+        }
+        var evalPath = Path.Combine(evalDir, "eval.yaml");
+        File.WriteAllText(evalPath, "stimuli: []");
+        try
+        {
+            await Assert.ThrowsAsync<InvalidOperationException>(() =>
+                EvaluateCommand.ResolveAdditionalAgents(
+                    ["../../plugins/demo/agents/helper.agent.md"], pluginRoot, evalPath));
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
 }

eng/skill-validator/tests/Evaluate/MetricsTests.cs

@@ -425,6 +425,42 @@ public class CollectMetricsTests
         Assert.Equal(2, result.ErrorCount);
     }
 
+    [Theory]
+    [InlineData(false)]
+    [InlineData("False")]
+    public void CountsUnsuccessfulToolCompletionsAsErrors(object success)
+    {
+        var successNode = success switch
+        {
+            bool value => JsonValue.Create(value),
+            string value => JsonValue.Create(value),
+            _ => throw new InvalidOperationException(),
+        };
+        var events = new List<AgentEvent>
+        {
+            MakeEvent("tool.execution_complete", D(("success", successNode))),
+            MakeEvent("session.idle"),
+        };
+
+        var result = MetricsCollector.CollectMetrics(events, "partial output", 1000, "/tmp/work");
+
+        Assert.Equal(1, result.ErrorCount);
+    }
+
+    [Fact]
+    public void SuccessfulToolCompletionsDoNotCountAsErrors()
+    {
+        var events = new List<AgentEvent>
+        {
+            MakeEvent("tool.execution_complete", D(("success", JsonValue.Create(true)))),
+            MakeEvent("session.idle"),
+        };
+
+        var result = MetricsCollector.CollectMetrics(events, "done", 1000, "/tmp/work");
+
+        Assert.Equal(0, result.ErrorCount);
+    }
+
     [Fact]
     public void PreservesWallTimeAndWorkDir()
     {

eng/skill-validator/tests/Evaluate/RunnerTests.cs

@@ -1,5 +1,6 @@
 using System.Diagnostics;
 using System.Text.Json;
+using System.Text.Json.Nodes;
 using GitHub.Copilot;
 using SkillValidator.Evaluate;
 using SkillValidator.Shared;
@@ -264,6 +265,132 @@ public class BuildSessionConfigTests
         Assert.NotNull(config.Hooks.OnPreToolUse);
     }
 
+    [Fact]
+    public async Task ShellToolDefersToPermissionRequestPathInspection()
+    {
+        var config = await AgentRunner.BuildSessionConfig(MockSkill, null, "gpt-4.1", "C:\\tmp\\work");
+        var args = JsonDocument.Parse("""{"fullCommandText": "cat /etc/passwd"}""").RootElement;
+
+        var result = await config.Hooks!.OnPreToolUse!(
+            new PreToolUseHookInput { ToolName = "bash", ToolArgs = args },
+            null!);
+
+        Assert.Equal("ask", result!.PermissionDecision);
+    }
+
+    [Fact]
+    public async Task DeniesShellCommandWhenAnyPathIsOutsideAllowedDirectories()
+    {
+        var workDir = Path.GetFullPath(Path.Combine(Path.GetTempPath(), "work"));
+        var allowedPath = Path.Combine(workDir, "src", "Program.cs");
+        var config = await AgentRunner.BuildSessionConfig(null, null, "gpt-4.1", workDir);
+        var request = new PermissionRequestShell
+        {
+            CanOfferSessionApproval = false,
+            Commands = [],
+            FullCommandText = $"cat \"{allowedPath}\" /etc/passwd",
+            HasWriteFileRedirection = false,
+            Intention = "Read files",
+            PossiblePaths = [allowedPath, "/etc/passwd"],
+            PossibleUrls = [],
+        };
+
+        var decision = await config.OnPermissionRequest!(request, null!);
+
+        Assert.Equal("reject", decision.Kind);
+    }
+
+    [Fact]
+    public async Task ApprovesShellCommandWhenAllPathsAreAllowed()
+    {
+        var workDir = Path.GetFullPath(Path.Combine(Path.GetTempPath(), "work"));
+        var allowedPath = Path.Combine(workDir, "src", "Program.cs");
+        var config = await AgentRunner.BuildSessionConfig(null, null, "gpt-4.1", workDir);
+        var request = new PermissionRequestShell
+        {
+            CanOfferSessionApproval = false,
+            Commands = [],
+            FullCommandText = $"cat \"{allowedPath}\"",
+            HasWriteFileRedirection = false,
+            Intention = "Read a source file",
+            PossiblePaths = [allowedPath],
+            PossibleUrls = [],
+        };
+
+        var decision = await config.OnPermissionRequest!(request, null!);
+
+        Assert.Equal("approve-once", decision.Kind);
+    }
+
+    [Fact]
+    public async Task DeniesShellCommandWithUrlAndNoPaths()
+    {
+        var workDir = Path.GetFullPath(Path.Combine(Path.GetTempPath(), "work"));
+        var config = await AgentRunner.BuildSessionConfig(null, null, "gpt-4.1", workDir);
+        var request = new PermissionRequestShell
+        {
+            CanOfferSessionApproval = false,
+            Commands = [],
+            FullCommandText = "curl https://example.com/data",
+            HasWriteFileRedirection = false,
+            Intention = "Download data",
+            PossiblePaths = [],
+            PossibleUrls =
+            [
+                new PermissionRequestShellPossibleUrl
+                {
+                    Url = "https://example.com/data",
+                },
+            ],
+        };
+
+        var decision = await config.OnPermissionRequest!(request, null!);
+
+        Assert.Equal("reject", decision.Kind);
+    }
+
+    [Fact]
+    public async Task DeniesShellCommandWhenUrlMetadataIsMissing()
+    {
+        var workDir = Path.GetFullPath(Path.Combine(Path.GetTempPath(), "work"));
+        var config = await AgentRunner.BuildSessionConfig(null, null, "gpt-4.1", workDir);
+        var request = new PermissionRequestShell
+        {
+            CanOfferSessionApproval = false,
+            Commands = [],
+            FullCommandText = "curl https://example.com/data",
+            HasWriteFileRedirection = false,
+            Intention = "Download data",
+            PossiblePaths = [],
+            PossibleUrls = [],
+        };
+
+        var decision = await config.OnPermissionRequest!(request, null!);
+
+        Assert.Equal("reject", decision.Kind);
+    }
+
+    [Fact]
+    public async Task ApprovesLocalShellCommandWithoutPaths()
+    {
+        var workDir = Path.GetFullPath(Path.Combine(Path.GetTempPath(), "work"));
+        var config = await AgentRunner.BuildSessionConfig(null, null, "gpt-4.1", workDir);
+        var request = new PermissionRequestShell
+        {
+            CanOfferSessionApproval = false,
+            Commands = [],
+            FullCommandText = "dotnet test",
+            HasWriteFileRedirection = false,
+            Intention = "Run tests",
+            PossiblePaths = [],
+            PossibleUrls = [],
+        };
+
+        var decision = await config.OnPermissionRequest!(request, null!);
+
+        Assert.Equal("approve-once", decision.Kind);
+    }
+
     [Fact]
     public async Task SetsMcpServersWhenProvided()
     {
@@ -428,14 +555,17 @@ public class BuildSessionConfigTests
         {
             var config = await AgentRunner.BuildSessionConfig(MockSkill, tempDir, "gpt-4.1", "C:\\tmp\\work");
             Assert.Single(config.SkillDirectories!);
-            // Normalize trailing separators for comparison
-            var expected = Path.GetFullPath(Path.Combine(tempDir, "skills"));
-            var actual = config.SkillDirectories![0].TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar);
-            Assert.Equal(expected, actual);
+            var stagedRoot = config.SkillDirectories![0];
+            Assert.StartsWith(Path.GetTempPath(), stagedRoot);
+            Assert.NotEqual(
+                Path.GetFullPath(Path.Combine(tempDir, "skills")),
+                Path.TrimEndingDirectorySeparator(stagedRoot));
+            Assert.True(File.Exists(Path.Combine(stagedRoot, "my-skill", "SKILL.md")));
         }
         finally
         {
             Directory.Delete(tempDir, true);
+            await AgentRunner.CleanupWorkDirs();
         }
     }
 
@@ -447,6 +577,437 @@ public class BuildSessionConfigTests
         Assert.Single(config.SkillDirectories!);
         Assert.StartsWith(Path.GetTempPath(), config.SkillDirectories![0]);
     }
+
+    [Fact]
+    public async Task IsolatedAgentRegistersOnlyTargetAndDeclaredDependencies()
+    {
+        var target = new AgentInfo(
+            "target-agent",
+            "Target",
+            "target.agent.md",
+            "---\nname: target-agent\ndescription: Target\n---\nTarget prompt",
+            "target.agent.md");
+        var dependency = new AgentInfo(
+            "dependency-agent",
+            "Dependency",
+            "dependency.agent.md",
+            "---\nname: dependency-agent\ndescription: Dependency\n---\nDependency prompt",
+            "dependency.agent.md");
+
+        var config = await AgentRunner.BuildSessionConfig(
+            skill: null,
+            pluginRoot: null,
+            model: "gpt-4.1",
+            workDir: "C:\\tmp\\work",
+            agent: target,
+            additionalAgents: [dependency]);
+
+        Assert.Equal(
+            ["target-agent", "dependency-agent"],
+            config.CustomAgents!.Select(agent => agent.Name));
+    }
+
+    [Fact]
+    public async Task SetupWorkDirCopiesVallyDirectoryFixture()
+    {
+        var evalRoot = Path.Combine(Path.GetTempPath(), $"agent-fixture-{Guid.NewGuid():N}");
+        var fixtureDir = Path.Combine(evalRoot, "fixtures", "project");
+        Directory.CreateDirectory(fixtureDir);
+        File.WriteAllText(Path.Combine(fixtureDir, "Project.csproj"), "<Project />");
+        var evalPath = Path.Combine(evalRoot, "eval.yaml");
+        File.WriteAllText(evalPath, "stimuli: []");
+        try
+        {
+            var scenario = new EvalScenario(
+                "Copy fixture",
+                "Inspect it",
+                Setup: new SetupConfig(
+                    Files: [new SetupFile("Project", "fixtures/project")]));
+
+            var workDir = await AgentRunner.SetupWorkDir(scenario, null, evalPath);
+
+            Assert.True(File.Exists(Path.Combine(workDir, "Project", "Project.csproj")));
+        }
+        finally
+        {
+            Directory.Delete(evalRoot, true);
+            await AgentRunner.CleanupWorkDirs();
+        }
+    }
+
+    [Fact]
+    public void ResolveSourcePathAllowsSharedFixtureInsideRepository()
+    {
+        var repoRoot = Path.Combine(Path.GetTempPath(), $"shared-fixture-{Guid.NewGuid():N}");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        var sharedDir = Path.Combine(repoRoot, "tests", "demo", "shared", "fixtures");
+        Directory.CreateDirectory(evalDir);
+        Directory.CreateDirectory(sharedDir);
+        Directory.CreateDirectory(Path.Combine(repoRoot, "plugins"));
+        var evalPath = Path.Combine(evalDir, "eval.yaml");
+        var source = Path.Combine(sharedDir, "input.txt");
+        File.WriteAllText(evalPath, "stimuli: []");
+        File.WriteAllText(source, "input");
+        try
+        {
+            var resolved = AgentRunner.ResolveSourcePath(
+                "../shared/fixtures/input.txt", evalPath, skillPath: null);
+
+            Assert.Equal(Path.GetFullPath(source), resolved);
+        }
+        finally
+        {
+            Directory.Delete(repoRoot, true);
+        }
+    }
+
+    [Fact]
+    public void ResolveSourcePathRejectsFileSymlinkOutsideRepository()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"source-file-link-{Guid.NewGuid():N}");
+        var repoRoot = Path.Combine(root, "repo");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        var fixturesDir = Path.Combine(evalDir, "fixtures");
+        var outsideFile = Path.Combine(root, "secret.txt");
+        Directory.CreateDirectory(fixturesDir);
+        Directory.CreateDirectory(Path.Combine(repoRoot, "plugins"));
+        File.WriteAllText(Path.Combine(evalDir, "eval.yaml"), "stimuli: []");
+        File.WriteAllText(outsideFile, "secret");
+        if (!SymlinkTestHelper.TryCreateFile(Path.Combine(fixturesDir, "secret.txt"), outsideFile))
+        {
+            Directory.Delete(root, true);
+            return;
+        }
+        try
+        {
+            var resolved = AgentRunner.ResolveSourcePath(
+                "fixtures/secret.txt", Path.Combine(evalDir, "eval.yaml"), skillPath: null);
+
+            Assert.Null(resolved);
+        }
+        finally
+        {
+            Directory.Delete(root, true);
+        }
+    }
+
+    [Fact]
+    public void ResolveSourcePathRejectsDirectorySymlinkComponentOutsideRepository()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"source-dir-link-{Guid.NewGuid():N}");
+        var repoRoot = Path.Combine(root, "repo");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        var fixturesDir = Path.Combine(evalDir, "fixtures");
+        var outsideDir = Path.Combine(root, "outside");
+        Directory.CreateDirectory(fixturesDir);
+        Directory.CreateDirectory(outsideDir);
+        Directory.CreateDirectory(Path.Combine(repoRoot, "plugins"));
+        File.WriteAllText(Path.Combine(evalDir, "eval.yaml"), "stimuli: []");
+        File.WriteAllText(Path.Combine(outsideDir, "secret.txt"), "secret");
+        if (!SymlinkTestHelper.TryCreateDirectory(Path.Combine(fixturesDir, "linked"), outsideDir))
+        {
+            Directory.Delete(root, true);
+            return;
+        }
+        try
+        {
+            var resolved = AgentRunner.ResolveSourcePath(
+                "fixtures/linked/secret.txt", Path.Combine(evalDir, "eval.yaml"), skillPath: null);
+
+            Assert.Null(resolved);
+        }
+        finally
+        {
+            Directory.Delete(root, true);
+        }
+    }
+
+    [Fact]
+    public async Task SetupWorkDirSkipsExplicitDirectorySymlinkSource()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"setup-dir-link-{Guid.NewGuid():N}");
+        var repoRoot = Path.Combine(root, "repo");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        var fixturesDir = Path.Combine(evalDir, "fixtures");
+        var outsideDir = Path.Combine(root, "outside");
+        Directory.CreateDirectory(fixturesDir);
+        Directory.CreateDirectory(outsideDir);
+        Directory.CreateDirectory(Path.Combine(repoRoot, "plugins"));
+        var evalPath = Path.Combine(evalDir, "eval.yaml");
+        File.WriteAllText(evalPath, "stimuli: []");
+        File.WriteAllText(Path.Combine(outsideDir, "secret.txt"), "secret");
+        if (!SymlinkTestHelper.TryCreateDirectory(Path.Combine(fixturesDir, "linked"), outsideDir))
+        {
+            Directory.Delete(root, true);
+            return;
+        }
+        try
+        {
+            var scenario = new EvalScenario(
+                "Copy fixture",
+                "Inspect it",
+                Setup: new SetupConfig(
+                    Files: [new SetupFile("Fixture", "fixtures/linked")]));
+
+            var workDir = await AgentRunner.SetupWorkDir(scenario, null, evalPath);
+
+            Assert.False(File.Exists(Path.Combine(workDir, "Fixture", "secret.txt")));
+        }
+        finally
+        {
+            Directory.Delete(root, true);
+            await AgentRunner.CleanupWorkDirs();
+        }
+    }
+
+    [Fact]
+    public async Task SetupWorkDirSkipsTopLevelSymlinkWhenCopyingTestFiles()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"setup-top-link-{Guid.NewGuid():N}");
+        var repoRoot = Path.Combine(root, "repo");
+        var evalDir = Path.Combine(repoRoot, "tests", "demo", "agent.router");
+        var outsideFile = Path.Combine(root, "secret.txt");
+        Directory.CreateDirectory(evalDir);
+        Directory.CreateDirectory(Path.Combine(repoRoot, "plugins"));
+        var evalPath = Path.Combine(evalDir, "eval.yaml");
+        File.WriteAllText(evalPath, "stimuli: []");
+        File.WriteAllText(outsideFile, "secret");
+        if (!SymlinkTestHelper.TryCreateFile(Path.Combine(evalDir, "secret-link.txt"), outsideFile))
+        {
+            Directory.Delete(root, true);
+            return;
+        }
+        try
+        {
+            var scenario = new EvalScenario(
+                "Copy fixtures",
+                "Inspect them",
+                Setup: new SetupConfig(CopyTestFiles: true));
+
+            var workDir = await AgentRunner.SetupWorkDir(scenario, null, evalPath);
+
+            Assert.False(File.Exists(Path.Combine(workDir, "secret-link.txt")));
+        }
+        finally
+        {
+            Directory.Delete(root, true);
+            await AgentRunner.CleanupWorkDirs();
+        }
+    }
+
+    [Fact]
+    public async Task PluginAgentRunRegistersCompleteProductionSurface()
+    {
+        var pluginRoot = Path.Combine(Path.GetTempPath(), $"agent-plugin-{Guid.NewGuid():N}");
+        var skillsDir = Path.Combine(pluginRoot, "skills", "helper-skill");
+        var agentsDir = Path.Combine(pluginRoot, "agents");
+        Directory.CreateDirectory(skillsDir);
+        Directory.CreateDirectory(agentsDir);
+        File.WriteAllText(Path.Combine(skillsDir, "SKILL.md"), """
+            ---
+            name: helper-skill
+            description: Helps.
+            ---
+            Help.
+            """);
+        File.WriteAllText(Path.Combine(agentsDir, "target.agent.md"), """
+            ---
+            name: target
+            description: Target.
+            ---
+            Target.
+            """);
+        File.WriteAllText(Path.Combine(agentsDir, "peer.agent.md"), """
+            ---
+            name: peer
+            description: Peer.
+            ---
+            Peer.
+            """);
+        File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+            {
+              "name": "demo",
+              "version": "1.0.0",
+              "description": "Demo",
+              "skills": ["./skills/"],
+              "agents": ["./agents/"]
+            }
+            """);
+        try
+        {
+            var target = Assert.Single(
+                await AgentDiscovery.DiscoverAgentsInPlugin(pluginRoot),
+                agent => agent.Name == "target");
+
+            var config = await AgentRunner.BuildSessionConfig(
+                skill: null,
+                pluginRoot: pluginRoot,
+                model: "gpt-4.1",
+                workDir: "C:\\tmp\\work",
+                agent: target);
+
+            Assert.Equal(
+                ["peer", "target"],
+                config.CustomAgents!.Select(agent => agent.Name).Order());
+            var stagedRoot = config.SkillDirectories!.Single();
+            Assert.StartsWith(Path.GetTempPath(), stagedRoot);
+            Assert.True(File.Exists(Path.Combine(stagedRoot, "helper-skill", "SKILL.md")));
+        }
+        finally
+        {
+            Directory.Delete(pluginRoot, true);
+            await AgentRunner.CleanupWorkDirs();
+        }
+    }
+
+    [Fact]
+    public async Task PluginModeStagesOnlySafeSkillTrees()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"plugin-skill-link-{Guid.NewGuid():N}");
+        var pluginRoot = Path.Combine(root, "plugins", "demo");
+        var skillsRoot = Path.Combine(pluginRoot, "skills");
+        var safeSkill = Path.Combine(skillsRoot, "safe");
+        var outsideSkill = Path.Combine(root, "outside", "linked");
+        Directory.CreateDirectory(safeSkill);
+        Directory.CreateDirectory(outsideSkill);
+        File.WriteAllText(Path.Combine(safeSkill, "SKILL.md"), """
+            ---
+            name: safe
+            description: Safe skill.
+            ---
+            Safe.
+            """);
+        File.WriteAllText(Path.Combine(outsideSkill, "SKILL.md"), """
+            ---
+            name: linked
+            description: External skill.
+            ---
+            External.
+            """);
+        File.WriteAllText(Path.Combine(outsideSkill, "secret.txt"), "secret");
+        File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+            {"name":"demo","version":"1.0.0","description":"Demo","skills":["./skills/"]}
+            """);
+        var linkedSkill = Path.Combine(skillsRoot, "linked");
+        if (!SymlinkTestHelper.TryCreateDirectory(linkedSkill, outsideSkill))
+        {
+            Directory.Delete(root, true);
+            return;
+        }
+
+        try
+        {
+            var config = await AgentRunner.BuildSessionConfig(
+                MockSkill,
+                pluginRoot,
+                "gpt-4.1",
+                Path.Combine(root, "work"));
+
+            var stagedRoot = Assert.Single(config.SkillDirectories!);
+            Assert.True(File.Exists(Path.Combine(stagedRoot, "safe", "SKILL.md")));
+            Assert.False(Directory.Exists(Path.Combine(stagedRoot, "linked")));
+            Assert.False(AgentRunner.CheckPermission(
+                Path.Combine(linkedSkill, "secret.txt"),
+                Path.Combine(root, "work"),
+                skillPath: null,
+                log: null,
+                pluginRoot: pluginRoot));
+        }
+        finally
+        {
+            Directory.Delete(root, true);
+            await AgentRunner.CleanupWorkDirs();
+        }
+    }
+
+    [Fact]
+    public async Task PluginSkillRunRegistersOnlyDeclaredAgentDependencies()
+    {
+        var pluginRoot = Path.Combine(Path.GetTempPath(), $"skill-plugin-{Guid.NewGuid():N}");
+        var skillDir = Path.Combine(pluginRoot, "skills", "target-skill");
+        var agentsDir = Path.Combine(pluginRoot, "agents");
+        Directory.CreateDirectory(skillDir);
+        Directory.CreateDirectory(agentsDir);
+        File.WriteAllText(Path.Combine(skillDir, "SKILL.md"), """
+            ---
+            name: target-skill
+            description: Target skill.
+            ---
+            Target.
+            """);
+        File.WriteAllText(Path.Combine(agentsDir, "declared.agent.md"), """
+            ---
+            name: declared
+            description: Declared dependency.
+            ---
+            Declared.
+            """);
+        File.WriteAllText(Path.Combine(agentsDir, "unrelated.agent.md"), """
+            ---
+            name: unrelated
+            description: Unrelated plugin agent.
+            ---
+            Unrelated.
+            """);
+        File.WriteAllText(Path.Combine(pluginRoot, "plugin.json"), """
+            {
+              "name": "demo",
+              "version": "1.0.0",
+              "description": "Demo",
+              "skills": ["./skills/"],
+              "agents": ["./agents/"]
+            }
+            """);
+        try
+        {
+            var targetSkill = Assert.Single(
+                await SkillDiscovery.DiscoverSkills(Path.Combine(pluginRoot, "skills")),
+                skill => skill.Name == "target-skill");
+            var declaredAgent = Assert.Single(
+                await AgentDiscovery.DiscoverAgentsInPlugin(pluginRoot),
+                agent => agent.Name == "declared");
+
+            var config = await AgentRunner.BuildSessionConfig(
+                skill: targetSkill,
+                pluginRoot: pluginRoot,
+                model: "gpt-4.1",
+                workDir: "C:\\tmp\\work",
+                additionalAgents: [declaredAgent]);
+
+            Assert.Equal("declared", Assert.Single(config.CustomAgents!).Name);
+        }
+        finally
+        {
+            Directory.Delete(pluginRoot, true);
+        }
+    }
+}
+
+public class RunEventBufferTests
+{
+    [Fact]
+    public void ConcurrentRecordsPreserveEventsAndOutput()
+    {
+        const int eventCount = 10_000;
+        var buffer = new RunEventBuffer();
+
+        Parallel.For(0, eventCount, index =>
+            buffer.Record("assistant.message_delta", (agentEvent, output) =>
+            {
+                agentEvent.Data["index"] = JsonValue.Create(index);
+                output.Append('x');
+            }));
+
+        var (events, output) = buffer.Snapshot();
+
+        Assert.Equal(eventCount, events.Count);
+        Assert.Equal(eventCount, output.Length);
+        Assert.Equal(
+            eventCount,
+            events.Select(agentEvent => agentEvent.Data["index"]!.GetValue<int>())
+                .Distinct()
+                .Count());
+    }
 }
 
 public class ExtractPathFromToolArgsTests
@@ -471,15 +1032,15 @@ public class ExtractPathFromToolArgsTests
     }
 
     [Fact]
-    public void ExtractsFullCommandTextKey()
+    public void IgnoresFullCommandText()
     {
         var args = JsonDocument.Parse("""{"fullCommandText": "dotnet build"}""").RootElement;
         var result = AgentRunner.ExtractPathFromToolArgs(MakeInput(args));
-        Assert.Equal("dotnet build", result);
+        Assert.Null(result);
     }
 
     [Fact]
-    public void PrefersPathOverFileNameAndFullCommandText()
+    public void PrefersPathOverFileName()
     {
         var args = JsonDocument.Parse("""{"fullCommandText": "cmd", "fileName": "f.cs", "path": "/p"}""").RootElement;
         var result = AgentRunner.ExtractPathFromToolArgs(MakeInput(args));

eng/skill-validator/tests/Shared/PathSafetyTests.cs

@@ -0,0 +1,33 @@
+using SkillValidator.Shared;
+
+namespace SkillValidator.Tests;
+
+public class PathSafetyTests
+{
+    [Fact]
+    public void ContainsReparsePointRejectsLinkedAllowedRoot()
+    {
+        var root = Path.Combine(Path.GetTempPath(), $"linked-allowed-root-{Guid.NewGuid():N}");
+        var target = Path.Combine(root, "target");
+        var linkedRoot = Path.Combine(root, "linked-root");
+        Directory.CreateDirectory(target);
+        File.WriteAllText(Path.Combine(target, "child.txt"), "content");
+        if (!SymlinkTestHelper.TryCreateDirectory(linkedRoot, target))
+        {
+            Directory.Delete(root, true);
+            return;
+        }
+
+        try
+        {
+            Assert.True(PathSafety.ContainsReparsePoint(linkedRoot, linkedRoot));
+            Assert.True(PathSafety.ContainsReparsePoint(
+                linkedRoot,
+                Path.Combine(linkedRoot, "child.txt")));
+        }
+        finally
+        {
+            Directory.Delete(root, true);
+        }
+    }
+}

eng/skill-validator/tests/SymlinkTestHelper.cs

@@ -0,0 +1,38 @@
+namespace SkillValidator.Tests;
+
+internal static class SymlinkTestHelper
+{
+    internal static bool TryCreateFile(string path, string target)
+    {
+        try
+        {
+            File.CreateSymbolicLink(path, target);
+            return true;
+        }
+        catch (Exception ex) when (
+            ex is IOException
+            or UnauthorizedAccessException
+            or PlatformNotSupportedException
+            or NotSupportedException)
+        {
+            return false;
+        }
+    }
+
+    internal static bool TryCreateDirectory(string path, string target)
+    {
+        try
+        {
+            Directory.CreateSymbolicLink(path, target);
+            return true;
+        }
+        catch (Exception ex) when (
+            ex is IOException
+            or UnauthorizedAccessException
+            or PlatformNotSupportedException
+            or NotSupportedException)
+        {
+            return false;
+        }
+    }
+}

eng/vally-adapter/InvestigatingResults.md

@@ -1,12 +1,12 @@
 # Investigating Evaluation Results (Vally)
 
-This guide is for AI agents (and humans) investigating non-passing, invalid, or warning-bearing skill evaluation results produced by the **Vally** harness via `eng/vally-adapter/adapt.mjs`. It documents the `results.json` schema, how to reach the raw Vally output, common result patterns, and recommended fixes.
+This guide is for AI agents (and humans) investigating non-passing, invalid, or warning-bearing skill and custom-agent evaluation results. Skills are produced by the **Vally** harness via `eng/vally-adapter/adapt.mjs`; custom agents use the native Copilot SDK lane and `eng/vally-adapter/adapt-agent-results.mjs` because Vally 0.14 cannot register custom agents. Both lanes emit the same result schema.
 
 For the end-to-end architecture, decision policy, metric definitions, and
 historical examples, start with the
 [Skill evaluation infrastructure overview](./README.md).
 
-Evaluations run through Vally (`@microsoft/vally-cli`): every skill's `tests/<plugin>/<skill>/eval.yaml` is run in up to three variants — **baseline** (no skills), **skilled** (only the skill under test), and **plugin** (the whole plugin loaded). The workflow records the exact expected-eval manifest before execution. The adapter then runs `vally compare` (a debiased, position-swapped head-to-head judgment of skilled vs baseline) and writes one `results.json` per expected skill, including an explicit invalid result when required evidence is missing.
+Every target runs in up to three variants — **baseline** (no target), **isolated** (only the target plus declared dependencies), and **plugin** (the production plugin surface). Skill evals run through Vally (`@microsoft/vally-cli`). Agent evals run through `skill-validator evaluate`, which registers `CustomAgents` directly and retains target activation, nested delegation, invoked skills, tool calls, completion, tokens, and wall time. Both adapters write one `results.json` per expected target, including an explicit invalid result when required evidence is missing.
 
 > Note: the linter (`skill-validator check`) is a **separate** workflow (`skill-check.yml`) and is unrelated to these eval results.
 
@@ -51,7 +51,7 @@ When updating the SDK, reassess the guard and run
 
 `eng/vally-adapter/consolidate.mjs` renders the comment and the fuller step summary. The PR comment starts with:
 
-- the number of unique skills, execution models, and model/skill results;
+- the number of unique targets, execution models, and model/target results;
 - the exact evaluated commit and judge model;
 - expected / observed / written result accounting, with missing, unexpected,
   invalid, recovered, and unresolved counts; and
@@ -79,7 +79,7 @@ reliability from stimulus-vote gate evidence.
 
 `--format full` (the workflow summary) keeps every result and adds `Δ Pref`,
 isolated/plugin quality, and baseline quality. These are triage metrics. They
-are not the gate. The `p` value applies to one model/skill result; the renderer
+are not the gate. The `p` value applies to one model/target result; the renderer
 does not apply a matrix-wide multiple-comparison correction.
 
 ### Reading the evaluation dashboard
@@ -118,12 +118,13 @@ Each file has a top-level object:
 
 | Field | Description |
 |-------|-------------|
-| `schemaVersion` | Adapter schema version. Version 2 adds explicit states; version 3 makes stimulus votes authoritative and separates repeated-run evidence; version 4 separates dormancy activation contracts from preference-eligible evidence |
+| `schemaVersion` | Adapter schema version. Version 2 adds explicit states; version 3 makes stimulus votes authoritative and separates repeated-run evidence; version 4 separates dormancy activation contracts from preference-eligible evidence; version 5 identifies the target with `skillKind` and adds native-agent activation/delegation evidence |
+| `skillKind` | `skill` or `agent`; custom-agent results are never represented as invocable skills |
 | `evalFile` / `expectedEval` | Normalized eval path and whether it was in the pre-run manifest |
 | `model` | Model used for agent runs |
 | `judgeModel` | Model used by `vally compare` |
 | `timestamp` | When results were written (UTC) |
-| `verdicts[]` | Per-skill results (one entry, since the adapter writes one file per skill) |
+| `verdicts[]` | Per-target results (one entry, since each adapter writes one file per skill or agent) |
 
 ### Verdict structure
 
@@ -131,14 +132,14 @@ A verdict carries **both** the head-to-head preference and absolute per-role dat
 
 | Field | Description |
 |-------|-------------|
-| `skillName` / `skillPath` | The skill under test |
+| `skillName` / `skillPath` | Compatibility field names containing the evaluated target name and source path; `skillKind` disambiguates skills and agents |
 | `state` | One of `VALID_PASS`, `VALID_REGRESSION`, `VALID_NO_CHANGE`, or `INVALID_INCONCLUSIVE` |
 | `stateReason` | Machine-readable `{ code, phase }`. Use this field for automation; do not parse `reason` |
 | `passed` | **The gate.** `true` only when `conclusive`, at least 5 preference-eligible distinct stimuli were counted, `signTest.pValue <= 0.05`, `netWin >= 0.20`, and `activationContract.passed == true` |
 | `netWin` | `(wins − losses) / preference-eligible stimulus votes` — the effect size the gate reads. Magnitude-free, so an identical eligible W/T/L record always yields an identical preference verdict |
 | `practicalSignificance` | `{ netWin, minimum, passed }`. The absolute directional effect must reach 20%; this blocks sparse records such as `5W/95T/0L` |
 | `signTest` | `{ wins, ties, losses, discordant, direction, pValue, alpha }` — exact one-sided binomial tail over discordant stimulus votes. **This is what decides.** Ties cannot support a win, so they hold `discordant` down |
-| `regressed` / `preferenceRegressed` | Compatibility and explicit fields for a credible LLM preference loss. In the current schema version 4 this maps to `VALID_NO_CHANGE`, not `VALID_REGRESSION`, because ordinal LLM preference is not objective completion evidence. Renderers apply the same report-only meaning to legacy records that have `regressed: true` but no `state` |
+| `regressed` / `preferenceRegressed` | Compatibility and explicit fields for a credible LLM preference loss. In the current schema this maps to `VALID_NO_CHANGE`, not `VALID_REGRESSION`, because ordinal LLM preference is not objective completion evidence. Renderers apply the same report-only meaning to legacy records that have `regressed: true` but no `state` |
 | `conclusive` | `false` when the comparison did not complete: errored runs, unmatched trajectories, or a summary that disagrees with its own `stimuli[].trials`. Integrity remains fail-closed across eligible and excluded stimuli |
 | `underpowered` | `true` when a completed, `conclusive: true` comparison counted fewer than `minCredibleStimuli` preference-eligible distinct stimuli. An independently proven `activation_contract_failed` state takes headline precedence while this field preserves the preference-power limitation |
 | `minCredibleStimuli` | The distinct-stimulus floor in force (5). See `eng/eval-quality/README.md` for why |
@@ -170,17 +171,22 @@ Each scenario merges the compare preference for that stimulus with the absolute
 | `expectActivation` | Whether the target should activate; `false` marks an expected-dormancy stimulus |
 | `preferenceGateEligible` / `preferenceGateExclusionReason` | Whether this scenario contributes a preference vote. Explicit dormancy is `false` / `activation_contract_only` |
 | `timedOut` | Whether the skilled run hit its timeout |
+| `agentActivationIsolated` / `agentActivationPlugin` | Agent targets only: exact target activation plus invoked/delegated agent names and event counts |
 | `skillActivationIsolated` | Isolated activation telemetry: `activated`, `activatedRuns`, `continuedRuns`, `activationOnlyCompletions`, `failedActivationOnlyCompletions`, and `unclassifiedRuns`. `continuedRuns` requires an ordered non-skill tool call after skill activation. An activation-only completion is a normally completed run with no such post-activation call; the failed count includes only runs whose graders did not pass |
 | `skillActivationPlugin` | The same telemetry for the whole-plugin run. `activated` means some plugin skill activity was observed; the current adapter does not retain the emitting skill identity (present only when a plugin variant ran) |
 | `baseline` | `{ judgeResult: { overallScore }, metrics }` — the skill-free control (`overallScore` is 0–5) |
 | `skilledIsolated` | Same shape, for the isolated skilled run |
 | `skilledPlugin` | Same shape, for the whole-plugin run (may be absent) |
 
-`metrics` on each role: `{ wallTimeMs, tokenEstimate, inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens }`.
+`metrics` on each role includes executor usage
+`{ wallTimeMs, tokenEstimate, inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens }`
+and judge usage
+`{ judgeInputTokens, judgeOutputTokens, judgeCacheReadTokens, judgeCacheWriteTokens }`.
 
-### Schema version 4 compatibility
+### Schema version 5 compatibility
 
-Schema version 4 changes the meaning of the existing top-level preference
+Schema version 5 adds `skillKind: agent` and agent activation/delegation fields.
+It retains the version 4 meaning of the existing top-level preference
 aliases (`wins`, `ties`, `losses`, `winRate`, `stimulusVoteCount`, and
 `trialCount`) from all stimulus votes to preference-eligible stimulus votes.
 Consumers that need the old all-stimulus view must read
@@ -223,6 +229,9 @@ The adapter's `results.json` is a summary. The uploaded artifact also contains t
 
 - `_experiment/<timestamp>/<variant>/results.jsonl` — one `trial-result` record per stimulus per variant, each with the full `trajectory` (`endReason`, `metrics.tokenUsage`, `metrics.skillActivationCount`, `toolCallCount`) and `gradeResult.score` (0–1).
 - `_experiment/<timestamp>/executor-session-logs/**/{metadata.json,events.jsonl}` — the per-session event stream (prompts, tool calls, agent output). `metadata.json` carries `variant`, `stimulusName`, `evalName`/`evalFilePath`, `model`, and `status`. This is what powers the AGENTVIZ replay link in the PR comment.
+- `_agent-evaluation/<timestamp>/{sessions.db,sessions/**/events.jsonl}` — native
+  custom-agent runs, including target-agent invocation, nested delegation, skill
+  invocation, tool calls, and usage events.
 
 To see exactly what the agent did for a failing scenario, open its `events.jsonl` (match on `variant` + `stimulusName` in the sibling `metadata.json`).
 

eng/vally-adapter/README.md

@@ -44,7 +44,7 @@ The design follows these official Vally concepts:
 | --- | --- | --- |
 | Define a test case | `stimuli` in `eval.yaml` | Require unique names, valid fixtures, and at least five preference-eligible distinct stimuli for new gated evals; keep explicit dormancy as activation-contract evidence |
 | Repeat a test | `defaults.runs` | Treat repeats as reliability evidence, not new independent task breadth |
-| Run the agent | Copilot SDK executor | Run baseline, isolated-skill, and full-plugin variants at one exact commit |
+| Run the target | Copilot SDK executor | Run baseline, isolated-target, and full-plugin variants at one exact commit; Vally runs skills and the native SDK lane runs custom agents |
 | Grade output | Static and LLM graders | Preserve grader evidence, but do not treat a mixed aggregate as objective completion |
 | Compare arms | Paired comparison judge | Check stable slot identity and retry only failed comparison slots |
 | Report scores | Scores, confidence intervals, pass metrics | Use scores for diagnosis, not as the improvement gate |
@@ -74,9 +74,9 @@ flowchart TD
 
     subgraph R["One matrix job"]
         F --> G["Download exact validator artifact"]
-        G --> H["Run Vally baseline variant"]
-        G --> I["Run Vally isolated-skill variant"]
-        G --> J["Run Vally full-plugin variant"]
+        G --> H["Run baseline variant"]
+        G --> I["Run isolated skill or agent variant"]
+        G --> J["Run full-plugin variant"]
         H --> K["Vally compare: baseline vs isolated skill"]
         I --> K
         J --> M
@@ -88,7 +88,7 @@ flowchart TD
 
     N2 --> O["Reconcile expected, observed, and written results"]
     O --> P["Consolidate all model/shard outputs"]
-    P --> Q["Publish Skill Evaluation Results PR comment"]
+    P --> Q["Publish Skill and Agent Evaluation Results PR comment"]
     P --> S["Upload raw evidence and rendered report artifacts"]
 ```
 
@@ -98,7 +98,8 @@ The implementation is split across these main components:
 | --- | --- |
 | [`.github/workflows/evaluation.yml`](../../.github/workflows/evaluation.yml) | Authorizes a request, binds it to an exact commit, discovers work, starts the reusable workflow, consolidates results, and publishes the PR comment |
 | [`.github/workflows/evaluation-run.yml`](../../.github/workflows/evaluation-run.yml) | Builds the trusted validator artifact, runs the model/shard matrix, invokes Vally, applies fault injection in tests, and uploads artifacts |
-| [`adapt.mjs`](./adapt.mjs) | Validates Vally comparison data, retries failed judge slots, computes schema-version-4 evidence, and assigns repository verdicts |
+| [`adapt.mjs`](./adapt.mjs) | Validates Vally skill comparison data, retries failed judge slots, computes schema-version-5 evidence, and assigns repository verdicts |
+| [`adapt-agent-results.mjs`](./adapt-agent-results.mjs) | Converts native SDK custom-agent runs into the same schema-version-5 evidence, including target activation and delegated-agent/skill telemetry |
 | [`consolidate.mjs`](./consolidate.mjs) | Combines model/shard result sets and produces the decision-first PR comment |
 | [`check_eval_quality.py`](../eval-quality/check_eval_quality.py) | Blocks structurally invalid or newly underpowered eval instruments before they run |
 
@@ -176,18 +177,25 @@ Partial artifacts remain available for diagnosis, but they are labeled
 incomplete and are never consolidated into quality evidence. A leg that found
 evals also fails if its primary result artifact is missing.
 
-### 3. Run three Vally variants
+### 3. Run three variants
 
-- **Baseline:** the tested skill is unavailable.
-- **Skilled:** only the tested skill is available.
-- **Plugin:** the full plugin is available, which can expose routing conflicts
-  and interactions with sibling skills.
+- **Baseline:** the tested skill or custom agent is unavailable.
+- **Isolated:** only the target skill, or the target custom agent plus its
+  declared skill/agent dependencies, is available.
+- **Plugin:** the full production plugin skill and custom-agent surface is
+  available, which can expose routing conflicts and interactions with siblings.
 
-The paired preference gate compares baseline only to the isolated-skill
+The paired preference gate compares baseline only to the isolated-target
 variant. The plugin variant contributes absolute activation and quality
 telemetry. It can expose routing conflicts and sibling-skill interference, but
 it does not receive a separate sign-test or practical-effect verdict.
 
+Vally 0.14 exposes `environment.skills` but no custom-agent registration field,
+and its Copilot executor only sets `skillDirectories`. Agent evals therefore use
+the existing .NET SDK runner, which passes `CustomAgents` directly and leaves
+the default parent agent selected so routing and delegation are measured rather
+than forced. The adapter normalizes those results before consolidation.
+
 ### 4. Preserve raw Vally evidence
 
 Each Vally run produces trajectories, grader output, metrics, and trial
@@ -275,7 +283,7 @@ twelve independent test cases.
 
 ### 9. Apply the repository decision rule
 
-For one model and skill's baseline-versus-isolated comparison:
+For one model and target's baseline-versus-isolated comparison:
 
 1. Require complete and well-formed result identity.
 2. Require every explicit dormancy activation contract to pass. This objective
@@ -377,7 +385,7 @@ measurement-health layer is valid.
 | Preference-eligible stimuli | How many independent in-scope task cases vote in the gate? | 4 in-scope stimuli + 1 dormancy contract = 4 votes, not 5 | Fewer than five preference cases cannot pass the exact 5% test. |
 | Stimulus W/T/L | On how many tasks did skilled beat, tie, or lose to baseline? | `5W / 1T / 1L` | This is the primary task-level effect summary. |
 | Discordant votes | How many stimulus votes were wins or losses? | `5W / 95T / 0L` has 5 discordant votes | Ties do not enter the sign-test numerator or denominator. |
-| One-sided exact p-value | Is the positive W/L direction unlikely under a 50/50 null? | `5W / 0L` gives `p = 0.03125` | Applies to one model/skill result. It is not corrected across the full matrix. |
+| One-sided exact p-value | Is the positive W/L direction unlikely under a 50/50 null? | `5W / 0L` gives `p = 0.03125` | Applies to one model/target result. It is not corrected across the full matrix. |
 | Aggregate net win | Is the improvement broad enough across all stimuli? | `(5 - 0) / 100 = 5%` | Must be at least 20% to pass, even when the p-value passes. |
 | Mean score and confidence interval | Did absolute grader scores move, and how uncertain is the mean? | Both arms can score highly while the paired preference is inconclusive | Triage only. It is not the pass statistic. |
 | Model identity | Which executor model produced the trajectories? | Claude Sonnet and GPT can disagree | Never pool different executor models into one vote. |

eng/vally-adapter/adapt-agent-results.mjs

@@ -0,0 +1,568 @@
+#!/usr/bin/env node
+
+/**
+ * Convert the native SDK custom-agent evaluator output into the current
+ * Vally-adapter result schema. Vally 0.14 cannot register custom agents, so
+ * agent evals use skill-validator's Copilot SDK runner for execution and this
+ * adapter keeps them in the same statistical/reporting pipeline as skills.
+ */
+
+import {
+  existsSync,
+  mkdirSync,
+  readdirSync,
+  readFileSync,
+  writeFileSync,
+} from "node:fs";
+import { join, relative, resolve } from "node:path";
+import { parseArgs } from "node:util";
+
+import {
+  comparisonToVerdict,
+  loadExpectedEvalFiles,
+  normalizeEvalFile,
+  readNonActivationStimuli,
+  VERDICT_STATES,
+} from "./adapt.mjs";
+
+const { values: opts } = parseArgs({
+  options: {
+    "results-file": { type: "string" },
+    "output-root": { type: "string", default: "eval-results" },
+    "expected-evals": { type: "string" },
+    "repo-root": { type: "string", default: "." },
+    model: { type: "string" },
+    "judge-model": { type: "string" },
+    help: { type: "boolean", default: false },
+  },
+  strict: true,
+});
+
+if (opts.help || !opts["results-file"]) {
+  console.log(`Usage:
+  node adapt-agent-results.mjs --results-file <legacy-results.json> [options]
+
+Options:
+  --output-root <dir>       Output root for per-agent results.json files.
+  --expected-evals <file>   Newline-delimited or JSON-array expected eval manifest.
+  --repo-root <dir>         Repository root used to read eval specs.
+  --model <model>           Override the recorded executor model.
+  --judge-model <model>     Override the recorded judge model.
+  --help                    Show this help.`);
+  process.exit(opts.help ? 0 : 1);
+}
+
+function agentIdentity(evalFile) {
+  const normalized = normalizeEvalFile(evalFile);
+  const parts = normalized.split("/").filter(Boolean);
+  if (parts.length < 4 || parts[0] !== "tests" || parts.at(-1) !== "eval.yaml") {
+    throw new Error(`Agent eval path must be under tests/<plugin>/**/eval.yaml: ${evalFile}`);
+  }
+  const plugin = parts[1];
+  const evalName = parts.at(-2);
+  const agentName = evalName.startsWith("agent.")
+    ? evalName.slice("agent.".length)
+    : evalName;
+  if (!agentName) {
+    throw new Error(`Agent eval path has no agent directory name: ${evalFile}`);
+  }
+  const skill = `agent.${agentName}`;
+  return {
+    plugin,
+    agentName,
+    skill,
+    skillPath: `plugins/${plugin}/agents/${agentName}.agent.md`,
+  };
+}
+
+function findAgentEvalFile(repoRoot, plugin, agentName) {
+  const testsRoot = join(repoRoot, "tests", plugin);
+  const candidates = [
+    join(testsRoot, `agent.${agentName}`, "eval.yaml"),
+    join(testsRoot, agentName, "eval.yaml"),
+  ];
+  if (existsSync(testsRoot)) {
+    for (const entry of readdirSync(testsRoot, { withFileTypes: true })
+      .filter((item) => item.isDirectory())
+      .sort((left, right) => left.name.localeCompare(right.name))) {
+      candidates.push(
+        join(testsRoot, entry.name, `agent.${agentName}`, "eval.yaml"),
+        join(testsRoot, entry.name, agentName, "eval.yaml"),
+      );
+    }
+  }
+  const found = candidates.find(existsSync);
+  return found
+    ? normalizeEvalFile(relative(repoRoot, found))
+    : `tests/${plugin}/agent.${agentName}/eval.yaml`;
+}
+
+function evalFileFromLegacyVerdict(verdict, repoRoot) {
+  const normalized = normalizeEvalFile(verdict.skillPath);
+  const match = /(?:^|\/)plugins\/([^/]+)\/.+\.agent\.md$/.exec(normalized);
+  if (!match) {
+    throw new Error(`Agent result has an invalid skillPath: ${verdict.skillPath}`);
+  }
+  const agentName = String(verdict.skillName ?? "").replace(/^agent\./, "");
+  if (!agentName) {
+    throw new Error("Agent result is missing skillName");
+  }
+  return findAgentEvalFile(repoRoot, match[1], agentName);
+}
+
+function agentSourcePath(verdict, repoRoot) {
+  if (!verdict?.skillPath)
+    return null;
+  const absolute = resolve(repoRoot, verdict.skillPath);
+  const repoRelative = relative(resolve(repoRoot), absolute);
+  if (repoRelative && repoRelative !== ".." && !repoRelative.startsWith(`..${process.platform === "win32" ? "\\" : "/"}`)) {
+    return normalizeEvalFile(repoRelative);
+  }
+  return normalizeEvalFile(verdict.skillPath);
+}
+
+function directionFromPairwise(pairwise) {
+  const winner = String(pairwise?.overallWinner ?? "").toLowerCase();
+  if (winner === "skill" || winner === "agent" || winner === "treatment") return 1;
+  if (winner === "baseline") return -1;
+  return 0;
+}
+
+function magnitudeFromPairwise(pairwise, direction) {
+  const raw = pairwise?.overallMagnitude;
+  const text = String(raw ?? "").toLowerCase();
+  const equal = raw === 2 || text === "equal";
+  const much = raw === 0 || raw === 4 || text.includes("much");
+  if (direction === 0 || equal) return { magnitude: "equal", score: 0 };
+  const magnitude = much
+    ? direction > 0 ? "much-better" : "much-worse"
+    : direction > 0 ? "slightly-better" : "slightly-worse";
+  return { magnitude, score: direction * (much ? 1 : 0.4) };
+}
+
+function targetAgentActivated(activation, agentName) {
+  return (activation?.invokedAgents ?? []).some(
+    (name) => String(name).toLowerCase() === agentName.toLowerCase(),
+  );
+}
+
+function fakeRecord(run, activated) {
+  const metrics = run?.metrics ?? {};
+  const judgeScore = run?.judgeResult?.overallScore;
+  return {
+    gradeResult: {
+      score: typeof judgeScore === "number" ? Math.max(0, Math.min(1, judgeScore / 5)) : null,
+    },
+    trajectory: {
+      endReason: metrics.timedOut ? "agent_timeout" : "completed",
+      metrics: {
+        wallTimeMs: metrics.wallTimeMs ?? 0,
+        tokenUsage: {
+          totalTokens: metrics.tokenEstimate ?? 0,
+          inputTokens: metrics.inputTokens ?? 0,
+          outputTokens: metrics.outputTokens ?? 0,
+          cacheReadTokens: metrics.cacheReadTokens ?? 0,
+          cacheWriteTokens: metrics.cacheWriteTokens ?? 0,
+        },
+        skillActivationCount: activated ? 1 : 0,
+      },
+    },
+  };
+}
+
+function dashboardRun(run) {
+  if (!run) return null;
+  const metrics = run.metrics ?? {};
+  return {
+    judgeResult: run.judgeResult ?? { overallScore: null },
+    metrics: {
+      wallTimeMs: metrics.wallTimeMs ?? 0,
+      tokenEstimate: metrics.tokenEstimate ?? 0,
+      inputTokens: metrics.inputTokens ?? 0,
+      outputTokens: metrics.outputTokens ?? 0,
+      cacheReadTokens: metrics.cacheReadTokens ?? 0,
+      cacheWriteTokens: metrics.cacheWriteTokens ?? 0,
+      judgeInputTokens: metrics.judgeInputTokens ?? 0,
+      judgeOutputTokens: metrics.judgeOutputTokens ?? 0,
+      judgeCacheReadTokens: metrics.judgeCacheReadTokens ?? 0,
+      judgeCacheWriteTokens: metrics.judgeCacheWriteTokens ?? 0,
+      toolCallCount: metrics.toolCallCount ?? 0,
+      toolCallBreakdown: metrics.toolCallBreakdown ?? {},
+      taskCompleted: metrics.taskCompleted ?? false,
+      errorCount: metrics.errorCount ?? 0,
+      timedOut: metrics.timedOut ?? false,
+    },
+  };
+}
+
+function activationEvidence(activation, agentName) {
+  const invokedAgents = [...new Set((activation?.invokedAgents ?? []).map(String))];
+  const delegatedAgents = invokedAgents.filter(
+    (name) => name.toLowerCase() !== agentName.toLowerCase(),
+  );
+  return {
+    activated: targetAgentActivated(activation, agentName),
+    targetAgent: agentName,
+    invokedAgents,
+    eventCount: activation?.subagentEventCount ?? 0,
+    delegatedAgents,
+    delegated: delegatedAgents.length > 0,
+  };
+}
+
+function scenarioTimedOut(scenario) {
+  return Boolean(
+    scenario.timedOut
+    || scenario.baseline?.metrics?.timedOut
+    || scenario.skilledIsolated?.metrics?.timedOut
+    || scenario.skilledPlugin?.metrics?.timedOut,
+  );
+}
+
+function legacyToVerdict(legacyVerdict, evalFile, repoRoot) {
+  const identity = agentIdentity(evalFile);
+  identity.skillPath = agentSourcePath(legacyVerdict, repoRoot) ?? identity.skillPath;
+  if ((legacyVerdict.scenarios ?? []).length === 0) {
+    const failureKind = legacyVerdict.failureKind ?? "native_evaluator_failure";
+    const message = legacyVerdict.reason
+      ?? `Native agent evaluator failed with ${failureKind} before producing scenarios`;
+    const verdict = invalidAgentVerdict(
+      identity,
+      `native_${failureKind}`,
+      message,
+    );
+    verdict.evaluationLane = "native-agent-sdk";
+    return verdict;
+  }
+  const baselineByStim = new Map();
+  const skilledByStim = new Map();
+  const pluginByStim = new Map();
+  const reportStimuli = [];
+
+  for (const scenario of legacyVerdict.scenarios ?? []) {
+    const targetIsolated = targetAgentActivated(
+      scenario.subagentActivationIsolated,
+      identity.agentName,
+    );
+    const targetPlugin = targetAgentActivated(
+      scenario.subagentActivationPlugin,
+      identity.agentName,
+    );
+    baselineByStim.set(scenario.scenarioName, [fakeRecord(scenario.baseline, false)]);
+    skilledByStim.set(scenario.scenarioName, [
+      fakeRecord(scenario.skilledIsolated, targetIsolated),
+    ]);
+    pluginByStim.set(scenario.scenarioName, [
+      fakeRecord(scenario.skilledPlugin, targetPlugin),
+    ]);
+
+    const direction = directionFromPairwise(scenario.pairwiseResult);
+    const { magnitude, score } = magnitudeFromPairwise(
+      scenario.pairwiseResult,
+      direction,
+    );
+    const missingRequiredArms = [
+      ["baseline", scenario.baseline],
+      ["isolated", scenario.skilledIsolated],
+      ["plugin", scenario.skilledPlugin],
+    ].filter(([, run]) => !run).map(([name]) => name);
+    const requiredErrorCount = [
+      scenario.baseline,
+      scenario.skilledIsolated,
+      scenario.skilledPlugin,
+    ].reduce((sum, run) => sum + (run?.metrics?.errorCount ?? 0), 0);
+    const requiredTimedOut = scenarioTimedOut(scenario);
+    const executionError = scenario.executionError
+      ?? (missingRequiredArms.length > 0
+        ? `Missing required agent evaluation arm(s): ${missingRequiredArms.join(", ")}`
+        : null)
+      ?? (requiredTimedOut ? "Required agent evaluation arm timed out" : null)
+      ?? (requiredErrorCount > 0
+        ? `Required agent evaluation arm(s) reported ${requiredErrorCount} executor error(s)`
+        : null)
+      ?? ((scenario.failedRunCount ?? 0) > 0
+        ? `${scenario.failedRunCount} run(s) failed`
+        : null)
+      ?? (!scenario.pairwiseResult ? "Pairwise judge did not produce a result" : null);
+    reportStimuli.push({
+      stimulusName: scenario.scenarioName,
+      meanScore: executionError ? 0 : score,
+      trials: [{
+        trialIndex: 0,
+        winner: executionError || magnitude === "equal"
+          ? "tie"
+          : direction > 0 ? "treatment" : direction < 0 ? "baseline" : "tie",
+        magnitude: executionError ? "equal" : magnitude,
+        score: executionError ? 0 : score,
+        evidence: executionError
+          ?? scenario.pairwiseResult?.overallReasoning
+          ?? "",
+        baselinePassed: scenario.baseline?.metrics?.taskCompleted ?? null,
+        treatmentPassed: scenario.skilledIsolated?.metrics?.taskCompleted ?? null,
+        errored: Boolean(executionError),
+      }],
+    });
+  }
+
+  const counted = reportStimuli.flatMap((stimulus) =>
+    stimulus.trials.filter((trial) => !trial.errored));
+  const wins = counted.filter((trial) => trial.winner === "treatment").length;
+  const losses = counted.filter((trial) => trial.winner === "baseline").length;
+  const ties = counted.length - wins - losses;
+  const report = {
+    summary: {
+      trialCount: counted.length,
+      erroredCount: reportStimuli.length - counted.length,
+      meanScore: counted.length
+        ? counted.reduce((sum, trial) => sum + trial.score, 0) / counted.length
+        : 0,
+      ciLow: legacyVerdict.confidenceInterval?.low ?? null,
+      ciHigh: legacyVerdict.confidenceInterval?.high ?? null,
+      wins,
+      ties,
+      losses,
+      winRate: counted.length ? wins / counted.length : 0,
+      mcnemar: null,
+      metricDeltas: null,
+    },
+    stimuli: reportStimuli,
+    unmatchedBaseline: [],
+    unmatchedTreatment: [],
+  };
+  const nonActivation = readNonActivationStimuli(evalFile, repoRoot);
+  const verdict = comparisonToVerdict(
+    report,
+    identity,
+    { baselineByStim, skilledByStim, pluginByStim, hasPlugin: true },
+    nonActivation,
+    "agent",
+  );
+  verdict.evaluationLane = "native-agent-sdk";
+  verdict.overfittingResult = legacyVerdict.overfittingResult ?? null;
+  const nativeCompletionRegressed =
+    legacyVerdict.failureKind === "completion_regression";
+  const nativeActivationFailed = legacyVerdict.skillNotActivated === true
+    || legacyVerdict.failureKind === "skill_not_activated";
+  if (nativeCompletionRegressed) {
+    verdict.state = VERDICT_STATES.VALID_REGRESSION;
+    verdict.stateReason = {
+      code: "native_completion_regression",
+      phase: "completion",
+    };
+    verdict.passed = false;
+    verdict.regressed = true;
+    verdict.reason = `${verdict.reason} — native evaluator reported an objective task-completion regression`;
+  } else if (nativeActivationFailed) {
+    verdict.passed = false;
+    if (verdict.state === VERDICT_STATES.VALID_PASS) {
+      verdict.state = VERDICT_STATES.VALID_NO_CHANGE;
+      verdict.stateReason = {
+        code: "target_agent_not_activated",
+        phase: "activation",
+      };
+    }
+    verdict.reason = `${verdict.reason} — native evaluator reported that the target agent did not activate`;
+  }
+
+  const legacyByScenario = new Map(
+    (legacyVerdict.scenarios ?? []).map((scenario) => [scenario.scenarioName, scenario]),
+  );
+  for (const scenario of verdict.scenarios) {
+    const legacy = legacyByScenario.get(scenario.scenarioName);
+    if (!legacy) continue;
+    scenario.agentActivationIsolated = activationEvidence(
+      legacy.subagentActivationIsolated,
+      identity.agentName,
+    );
+    scenario.agentActivationPlugin = activationEvidence(
+      legacy.subagentActivationPlugin,
+      identity.agentName,
+    );
+    scenario.skillActivationIsolated = legacy.skillActivationIsolated ?? null;
+    scenario.skillActivationPlugin = legacy.skillActivationPlugin ?? null;
+    scenario.baseline = dashboardRun(legacy.baseline);
+    scenario.skilledIsolated = dashboardRun(legacy.skilledIsolated);
+    scenario.skilledPlugin = dashboardRun(legacy.skilledPlugin);
+    scenario.timedOut = scenarioTimedOut(legacy);
+  }
+
+  return verdict;
+}
+
+function invalidAgentVerdict(identity, code, message) {
+  return {
+    skillName: identity.skill,
+    skillPath: identity.skillPath,
+    skillKind: "agent",
+    evaluationLane: "native-agent-sdk",
+    state: VERDICT_STATES.INVALID_INCONCLUSIVE,
+    stateReason: { code, phase: "agent_adapter" },
+    conclusive: false,
+    underpowered: false,
+    passed: false,
+    regressed: false,
+    preferenceRegressed: false,
+    netWin: 0,
+    signTest: {
+      wins: 0,
+      ties: 0,
+      losses: 0,
+      discordant: 0,
+      direction: "none",
+      pValue: 1,
+      alpha: 0.05,
+    },
+    wins: 0,
+    ties: 0,
+    losses: 0,
+    stimulusVoteCount: 0,
+    trialCount: 0,
+    scenarios: [],
+    errors: [{ code, phase: "agent_adapter", message }],
+    recoveredErrors: [],
+    reason: message,
+  };
+}
+
+function writeResult(outputRoot, evalFile, identity, verdict, model, judgeModel, expectedEval) {
+  const outputDir = join(outputRoot, identity.plugin, identity.skill);
+  mkdirSync(outputDir, { recursive: true });
+  writeFileSync(
+    join(outputDir, "results.json"),
+    JSON.stringify({
+      schemaVersion: 5,
+      evalFile,
+      model,
+      judgeModel,
+      timestamp: new Date().toISOString(),
+      expectedEval,
+      evaluationLane: "native-agent-sdk",
+      verdicts: [verdict],
+    }, null, 2),
+  );
+}
+
+function main() {
+  const sourcePath = resolve(opts["results-file"]);
+  const outputRoot = resolve(opts["output-root"]);
+  const repoRoot = resolve(opts["repo-root"]);
+  const expectedEvals = loadExpectedEvalFiles(opts["expected-evals"]);
+  const expectedManifestProvided = Boolean(opts["expected-evals"]);
+  if (!existsSync(sourcePath)) {
+    throw new Error(`Native agent results file not found: ${sourcePath}`);
+  }
+
+  const source = JSON.parse(readFileSync(sourcePath, "utf8"));
+  const model = opts.model ?? source.model ?? "unknown";
+  const judgeModel = opts["judge-model"] ?? source.judgeModel ?? "unknown";
+  const legacyVerdicts = new Map(
+    (source.verdicts ?? []).map((verdict) => [
+      String(verdict.skillName ?? "").replace(/^agent\./, ""),
+      verdict,
+    ]),
+  );
+  const expectedSet = new Set(expectedEvals);
+  const observedEvals = [
+    ...new Set((source.verdicts ?? []).map(
+      (verdict) => evalFileFromLegacyVerdict(verdict, repoRoot),
+    )),
+  ].sort();
+  const targetEvals = [
+    ...new Set([...expectedEvals, ...observedEvals]),
+  ].sort();
+
+  const invalidEvals = [];
+  const missingEvals = [];
+  const unexpectedEvals = [];
+  let written = 0;
+  for (const evalFile of targetEvals) {
+    const identity = agentIdentity(evalFile);
+    const expectedEval = !expectedManifestProvided || expectedSet.has(evalFile);
+    const legacy = legacyVerdicts.get(identity.agentName);
+    let verdict;
+    if (!legacy) {
+      const message = `Native agent evaluator produced no verdict for ${identity.agentName}`;
+      verdict = invalidAgentVerdict(identity, "missing_agent_verdict", message);
+      missingEvals.push(evalFile);
+      invalidEvals.push(evalFile);
+    } else {
+      try {
+        verdict = legacyToVerdict(legacy, evalFile, repoRoot);
+        if (verdict.state === VERDICT_STATES.INVALID_INCONCLUSIVE
+            && verdict.stateReason?.code !== "underpowered") {
+          invalidEvals.push(evalFile);
+        }
+      } catch (error) {
+        const message = error instanceof Error ? error.message : String(error);
+        verdict = invalidAgentVerdict(identity, "agent_result_adaptation_failed", message);
+        invalidEvals.push(evalFile);
+      }
+    }
+    if (!expectedEval) {
+      unexpectedEvals.push(evalFile);
+      verdict.state = VERDICT_STATES.INVALID_INCONCLUSIVE;
+      verdict.stateReason = { code: "unexpected_eval", phase: "agent_adapter" };
+      verdict.conclusive = false;
+      verdict.passed = false;
+      verdict.regressed = false;
+      verdict.preferenceRegressed = false;
+      verdict.errors ??= [];
+      verdict.errors.push({
+        phase: "agent_adapter",
+        kind: "permanent",
+        code: "unexpected_eval",
+        message: `${evalFile} was observed but was not in the expected-eval manifest`,
+      });
+      verdict.reason = `${verdict.reason}; observed eval was not in the expected-eval manifest`;
+      invalidEvals.push(evalFile);
+    }
+    writeResult(
+      outputRoot,
+      evalFile,
+      identity,
+      verdict,
+      model,
+      judgeModel,
+      expectedEval,
+    );
+    written++;
+  }
+
+  const uniqueInvalidEvals = [...new Set(invalidEvals)];
+  const measurementInvalidEvals = [...new Set([...missingEvals, ...uniqueInvalidEvals])];
+  mkdirSync(outputRoot, { recursive: true });
+  writeFileSync(
+    join(outputRoot, "adapter-summary.json"),
+    JSON.stringify({
+      schemaVersion: 1,
+      evaluationLane: "native-agent-sdk",
+      expectedManifestProvided,
+      expectedEvalCount: expectedEvals.length,
+      observedEvalCount: observedEvals.length,
+      writtenResultCount: written,
+      missingEvalCount: missingEvals.length,
+      unexpectedEvalCount: unexpectedEvals.length,
+      invalidEvalCount: uniqueInvalidEvals.length,
+      measurementInvalidEvalCount: measurementInvalidEvals.length,
+      missingEvals,
+      invalidEvals: uniqueInvalidEvals,
+      measurementInvalidEvals,
+      unexpectedEvals,
+    }, null, 2),
+  );
+}
+
+try {
+  main();
+} catch (error) {
+  console.error(`Error: ${error instanceof Error ? error.message : String(error)}`);
+  process.exitCode = 1;
+}
+
+export {
+  activationEvidence,
+  agentIdentity,
+  directionFromPairwise,
+  legacyToVerdict,
+  magnitudeFromPairwise,
+};

eng/vally-adapter/adapt-agent-results.test.mjs

@@ -0,0 +1,573 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+import {
+  mkdirSync,
+  mkdtempSync,
+  readFileSync,
+  rmSync,
+  writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { spawnSync } from "node:child_process";
+
+const script = join(dirname(fileURLToPath(import.meta.url)), "adapt-agent-results.mjs");
+const dashboardScript = join(
+  dirname(fileURLToPath(import.meta.url)),
+  "..",
+  "dashboard",
+  "generate-benchmark-data.ps1",
+);
+
+function runResult(score, taskCompleted = true) {
+  return {
+    metrics: {
+      wallTimeMs: 1200,
+      tokenEstimate: 300,
+      inputTokens: 200,
+      outputTokens: 100,
+      cacheReadTokens: 20,
+      cacheWriteTokens: 10,
+      judgeInputTokens: 80,
+      judgeOutputTokens: 40,
+      judgeCacheReadTokens: 8,
+      judgeCacheWriteTokens: 4,
+      toolCallCount: 2,
+      toolCallBreakdown: { bash: 1, skill: 1 },
+      taskCompleted,
+      errorCount: 0,
+      timedOut: false,
+    },
+    judgeResult: {
+      overallScore: score,
+      overallReasoning: "quality evidence",
+      rubricScores: [],
+    },
+  };
+}
+
+function writeAgentEval(root, scenarioCount = 5) {
+  const evalDir = join(root, "tests", "demo", "agent.router");
+  mkdirSync(evalDir, { recursive: true });
+  const stimuli = Array.from({ length: scenarioCount }, (_, index) => `
+  - name: Scenario ${index + 1}
+    prompt: Route this request.
+    rubric:
+      - Completed the task`);
+  const evalFile = "tests/demo/agent.router/eval.yaml";
+  writeFileSync(join(root, evalFile), `name: agent.router
+defaults:
+  timeout: 5m
+stimuli:${stimuli.join("")}
+`);
+  writeFileSync(join(root, "expected.txt"), `${evalFile}\n`);
+  return evalFile;
+}
+
+function winningScenario(index) {
+  return {
+    scenarioName: `Scenario ${index}`,
+    baseline: runResult(2),
+    skilledIsolated: runResult(4),
+    skilledPlugin: runResult(4.5),
+    pairwiseResult: {
+      overallWinner: "skill",
+      overallMagnitude: 1,
+      overallReasoning: "The registered agent completed more of the task.",
+    },
+    subagentActivationIsolated: {
+      invokedAgents: ["router"],
+      subagentEventCount: 1,
+    },
+    subagentActivationPlugin: {
+      invokedAgents: ["router"],
+      subagentEventCount: 1,
+    },
+    timedOut: false,
+    failedRunCount: 0,
+  };
+}
+
+function runAdapter(root, verdict) {
+  writeFileSync(join(root, "legacy.json"), JSON.stringify({
+    model: "executor",
+    judgeModel: "judge",
+    verdicts: [verdict],
+  }));
+  const output = join(root, "out");
+  const result = spawnSync(process.execPath, [
+    script,
+    "--results-file", join(root, "legacy.json"),
+    "--output-root", output,
+    "--expected-evals", join(root, "expected.txt"),
+    "--repo-root", root,
+  ], { encoding: "utf8" });
+  return { output, result };
+}
+
+test("converts native agent results into schema-version-5 agent evidence", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-"));
+  try {
+    const evalDir = join(root, "tests", "demo", "agent.router");
+    mkdirSync(evalDir, { recursive: true });
+    const stimuli = Array.from({ length: 5 }, (_, index) => `
+  - name: Scenario ${index + 1}
+    prompt: Route this request.
+    rubric:
+      - Completed the task`);
+    writeFileSync(join(evalDir, "eval.yaml"), `name: agent.router
+defaults:
+  timeout: 5m
+stimuli:${stimuli.join("")}
+`);
+    const expected = "tests/demo/agent.router/eval.yaml";
+    writeFileSync(join(root, "expected.txt"), `${expected}\n`);
+
+    const scenario = (index) => ({
+      scenarioName: `Scenario ${index}`,
+      baseline: runResult(2),
+      skilledIsolated: runResult(4),
+      skilledPlugin: runResult(4.5),
+      pairwiseResult: {
+        overallWinner: "skill",
+        overallMagnitude: 1,
+        overallReasoning: "The registered agent completed more of the task.",
+      },
+      subagentActivationIsolated: {
+        invokedAgents: ["router", "helper"],
+        subagentEventCount: 4,
+      },
+      subagentActivationPlugin: {
+        invokedAgents: ["router", "helper", "plugin-peer"],
+        subagentEventCount: 6,
+      },
+      skillActivationIsolated: {
+        activated: true,
+        detectedSkills: ["routing-skill"],
+        extraTools: [],
+        skillEventCount: 1,
+      },
+      skillActivationPlugin: {
+        activated: true,
+        detectedSkills: ["routing-skill", "plugin-skill"],
+        extraTools: [],
+        skillEventCount: 2,
+      },
+      timedOut: false,
+      failedRunCount: 0,
+    });
+    writeFileSync(join(root, "legacy.json"), JSON.stringify({
+      model: "executor",
+      judgeModel: "judge",
+      verdicts: [{
+        skillName: "router",
+        skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+        skillKind: "agent",
+        scenarios: [1, 2, 3, 4, 5].map(scenario),
+      }],
+    }));
+
+    const output = join(root, "out");
+    const result = spawnSync(process.execPath, [
+      script,
+      "--results-file", join(root, "legacy.json"),
+      "--output-root", output,
+      "--expected-evals", join(root, "expected.txt"),
+      "--repo-root", root,
+    ], { encoding: "utf8" });
+
+    assert.equal(result.status, 0, result.stderr);
+    const adapted = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    );
+    const verdict = adapted.verdicts[0];
+    assert.equal(adapted.schemaVersion, 5);
+    assert.equal(adapted.evaluationLane, "native-agent-sdk");
+    assert.equal(verdict.skillKind, "agent");
+    assert.equal(verdict.state, "VALID_PASS");
+    assert.equal(verdict.signTest.wins, 5);
+    assert.equal(verdict.scenarios[0].agentActivationIsolated.activated, true);
+    assert.deepEqual(
+      verdict.scenarios[0].agentActivationIsolated.delegatedAgents,
+      ["helper"],
+    );
+    assert.deepEqual(
+      verdict.scenarios[0].skillActivationPlugin.detectedSkills,
+      ["routing-skill", "plugin-skill"],
+    );
+    assert.equal(verdict.scenarios[0].skilledIsolated.metrics.toolCallCount, 2);
+    assert.equal(verdict.scenarios[0].skilledIsolated.metrics.taskCompleted, true);
+    assert.equal(verdict.scenarios[0].skilledIsolated.metrics.judgeInputTokens, 80);
+    assert.equal(verdict.scenarios[0].skilledIsolated.metrics.judgeOutputTokens, 40);
+    assert.equal(verdict.scenarios[0].skilledIsolated.metrics.judgeCacheReadTokens, 8);
+    assert.equal(verdict.scenarios[0].skilledIsolated.metrics.judgeCacheWriteTokens, 4);
+
+    const dashboardOutput = join(root, "dashboard");
+    const dashboardResult = spawnSync("pwsh", [
+      "-NoLogo",
+      "-NoProfile",
+      "-NonInteractive",
+      "-File", dashboardScript,
+      "-ResultsFile", join(output, "demo", "agent.router", "results.json"),
+      "-PluginName", "demo",
+      "-OutputDir", dashboardOutput,
+      "-SkipBenchmarkData",
+    ], { encoding: "utf8" });
+    assert.equal(
+      dashboardResult.status,
+      0,
+      dashboardResult.stdout + dashboardResult.stderr,
+    );
+    const tokenUsage = JSON.parse(
+      readFileSync(join(dashboardOutput, "token-usage.json"), "utf8"),
+    );
+    assert.ok(tokenUsage.entries.length > 0);
+    assert.equal(tokenUsage.entries[0].judgeTokensIn, 80);
+    assert.equal(tokenUsage.entries[0].judgeTokensOut, 40);
+    assert.equal(tokenUsage.entries[0].judgeCacheRead, 8);
+    assert.equal(tokenUsage.entries[0].judgeCacheWrite, 4);
+
+    const summary = JSON.parse(
+      readFileSync(join(output, "adapter-summary.json"), "utf8"),
+    );
+    assert.equal(summary.expectedEvalCount, 1);
+    assert.equal(summary.writtenResultCount, 1);
+    assert.equal(summary.measurementInvalidEvalCount, 0);
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("derives the eval file from an absolute native agent path", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-path-"));
+  try {
+    writeAgentEval(root);
+    writeFileSync(join(root, "legacy.json"), JSON.stringify({
+      model: "executor",
+      judgeModel: "judge",
+      verdicts: [{
+        skillName: "router",
+        skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+        skillKind: "agent",
+        passed: true,
+        scenarios: [1, 2, 3, 4, 5].map(winningScenario),
+      }],
+    }));
+    const output = join(root, "out");
+    const result = spawnSync(process.execPath, [
+      script,
+      "--results-file", join(root, "legacy.json"),
+      "--output-root", output,
+      "--repo-root", root,
+    ], { encoding: "utf8" });
+
+    assert.equal(result.status, 0, result.stderr);
+    const adapted = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    );
+    assert.equal(adapted.evalFile, "tests/demo/agent.router/eval.yaml");
+    assert.equal(adapted.expectedEval, true);
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("preserves numeric and string equal pairwise judgments as ties", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-equal-"));
+  try {
+    writeAgentEval(root);
+    const scenarios = [1, 2, 3, 4, 5].map((index) => {
+      const scenario = winningScenario(index);
+      scenario.pairwiseResult.overallMagnitude = index % 2 === 0 ? "Equal" : 2;
+      return scenario;
+    });
+    const { output, result } = runAdapter(root, {
+      skillName: "router",
+      skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+      skillKind: "agent",
+      passed: true,
+      scenarios,
+    });
+
+    assert.equal(result.status, 0, result.stderr);
+    const verdict = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    ).verdicts[0];
+    assert.equal(verdict.signTest.wins, 0);
+    assert.equal(verdict.signTest.losses, 0);
+    assert.equal(verdict.signTest.ties, 5);
+    assert.equal(verdict.scenarios[0].trials[0].winner, "tie");
+    assert.equal(verdict.scenarios[0].trials[0].magnitude, "equal");
+    assert.equal(verdict.scenarios[0].trials[0].score, 0);
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("preserves a declared nonstandard agent source path", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-custom-path-"));
+  try {
+    writeAgentEval(root);
+    writeFileSync(join(root, "legacy.json"), JSON.stringify({
+      model: "executor",
+      judgeModel: "judge",
+      verdicts: [{
+        skillName: "router",
+        skillPath: join(root, "plugins", "demo", "custom-agents", "router.agent.md"),
+        skillKind: "agent",
+        passed: true,
+        scenarios: [1, 2, 3, 4, 5].map(winningScenario),
+      }],
+    }));
+    const output = join(root, "out");
+    const result = spawnSync(process.execPath, [
+      script,
+      "--results-file", join(root, "legacy.json"),
+      "--output-root", output,
+      "--repo-root", root,
+    ], { encoding: "utf8" });
+
+    assert.equal(result.status, 0, result.stderr);
+    const adapted = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    );
+    assert.equal(
+      adapted.verdicts[0].skillPath,
+      "plugins/demo/custom-agents/router.agent.md",
+    );
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("adapts an agent eval from a nested test layout", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-nested-eval-"));
+  try {
+    const nestedEval = "tests/demo/nested/agent.router/eval.yaml";
+    const nestedDir = join(root, "tests", "demo", "nested", "agent.router");
+    mkdirSync(nestedDir, { recursive: true });
+    const stimuli = Array.from({ length: 5 }, (_, index) => `
+  - name: Scenario ${index + 1}
+    prompt: Route this request.
+    rubric:
+      - Completed the task`);
+    writeFileSync(join(root, nestedEval), `name: agent.router
+defaults:
+  timeout: 5m
+stimuli:${stimuli.join("")}
+`);
+    writeFileSync(join(root, "legacy.json"), JSON.stringify({
+      model: "executor",
+      judgeModel: "judge",
+      verdicts: [{
+        skillName: "router",
+        skillPath: join(root, "plugins", "demo", "custom-agents", "router.agent.md"),
+        skillKind: "agent",
+        passed: true,
+        scenarios: [1, 2, 3, 4, 5].map(winningScenario),
+      }],
+    }));
+    const output = join(root, "out");
+    const result = spawnSync(process.execPath, [
+      script,
+      "--results-file", join(root, "legacy.json"),
+      "--output-root", output,
+      "--repo-root", root,
+    ], { encoding: "utf8" });
+
+    assert.equal(result.status, 0, result.stderr);
+    const adapted = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    );
+    assert.equal(adapted.evalFile, nestedEval);
+    assert.equal(adapted.verdicts[0].skillName, "agent.router");
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("treats a native no-scenario failure as measurement-invalid", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-no-scenarios-"));
+  try {
+    writeAgentEval(root);
+    const { output, result } = runAdapter(root, {
+      skillName: "router",
+      skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+      skillKind: "agent",
+      passed: false,
+      failureKind: "spec_conformance_failure",
+      reason: "Prompt mentions target name",
+      scenarios: [],
+    });
+
+    assert.equal(result.status, 0, result.stderr);
+    const verdict = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    ).verdicts[0];
+    assert.equal(verdict.state, "INVALID_INCONCLUSIVE");
+    assert.equal(verdict.stateReason.code, "native_spec_conformance_failure");
+    assert.match(verdict.reason, /Prompt mentions target name/);
+    const summary = JSON.parse(
+      readFileSync(join(output, "adapter-summary.json"), "utf8"),
+    );
+    assert.equal(summary.invalidEvalCount, 1);
+    assert.equal(summary.measurementInvalidEvalCount, 1);
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("marks observed agents outside the manifest as unexpected", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-unexpected-"));
+  try {
+    const evalFile = writeAgentEval(root);
+    writeFileSync(join(root, "expected.txt"), "tests/demo/agent.other/eval.yaml\n");
+    const { output, result } = runAdapter(root, {
+      skillName: "router",
+      skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+      skillKind: "agent",
+      passed: true,
+      scenarios: [1, 2, 3, 4, 5].map(winningScenario),
+    });
+
+    assert.equal(result.status, 0, result.stderr);
+    const adapted = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    );
+    assert.equal(adapted.evalFile, evalFile);
+    assert.equal(adapted.expectedEval, false);
+    assert.equal(adapted.verdicts[0].state, "INVALID_INCONCLUSIVE");
+    assert.equal(adapted.verdicts[0].stateReason.code, "unexpected_eval");
+    const summary = JSON.parse(
+      readFileSync(join(output, "adapter-summary.json"), "utf8"),
+    );
+    assert.deepEqual(summary.unexpectedEvals, [evalFile]);
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("fails closed when the plugin arm times out", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-timeout-"));
+  try {
+    writeAgentEval(root);
+    const scenarios = [1, 2, 3, 4, 5].map(winningScenario);
+    scenarios[0].skilledPlugin.metrics.timedOut = true;
+    const { output, result } = runAdapter(root, {
+      skillName: "router",
+      skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+      skillKind: "agent",
+      passed: true,
+      scenarios,
+    });
+
+    assert.equal(result.status, 0, result.stderr);
+    const verdict = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    ).verdicts[0];
+    assert.equal(verdict.state, "INVALID_INCONCLUSIVE");
+    assert.equal(verdict.signTest.wins, 4);
+    assert.equal(verdict.scenarios[0].timedOut, true);
+    assert.equal(verdict.scenarios[0].trials[0].errored, true);
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("fails closed when a required arm records an executor error", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-error-"));
+  try {
+    writeAgentEval(root);
+    const scenarios = [1, 2, 3, 4, 5].map(winningScenario);
+    scenarios[0].skilledPlugin.metrics.errorCount = 1;
+    const { output, result } = runAdapter(root, {
+      skillName: "router",
+      skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+      skillKind: "agent",
+      passed: true,
+      scenarios,
+    });
+
+    assert.equal(result.status, 0, result.stderr);
+    const verdict = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    ).verdicts[0];
+    assert.equal(verdict.state, "INVALID_INCONCLUSIVE");
+    assert.equal(verdict.signTest.wins, 4);
+    assert.equal(verdict.scenarios[0].trials[0].errored, true);
+    assert.match(
+      verdict.scenarios[0].trials[0].evidence,
+      /reported 1 executor error/,
+    );
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("preserves a native target-agent activation failure", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-activation-"));
+  try {
+    writeAgentEval(root);
+    const scenarios = [1, 2, 3, 4, 5].map((index) => {
+      const scenario = winningScenario(index);
+      scenario.subagentActivationIsolated.invokedAgents = ["helper"];
+      scenario.subagentActivationPlugin.invokedAgents = ["helper"];
+      return scenario;
+    });
+
+    const { output, result } = runAdapter(root, {
+      skillName: "router",
+      skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+      skillKind: "agent",
+      passed: false,
+      failureKind: "skill_not_activated",
+      skillNotActivated: true,
+      scenarios,
+    });
+
+    assert.equal(result.status, 0, result.stderr);
+    const verdict = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    ).verdicts[0];
+    assert.equal(verdict.signTest.wins, 5);
+    assert.equal(verdict.state, "VALID_NO_CHANGE");
+    assert.equal(verdict.stateReason.code, "target_agent_not_activated");
+    assert.equal(verdict.passed, false);
+    assert.match(verdict.reason, /target agent did not activate/);
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});
+
+test("preserves a native completion regression over a preference win", () => {
+  const root = mkdtempSync(join(tmpdir(), "agent-adapter-completion-"));
+  try {
+    writeAgentEval(root);
+    const scenarios = [1, 2, 3, 4, 5].map(winningScenario);
+    scenarios[0].baseline.metrics.taskCompleted = true;
+    scenarios[0].skilledIsolated.metrics.taskCompleted = false;
+    const { output, result } = runAdapter(root, {
+      skillName: "router",
+      skillPath: join(root, "plugins", "demo", "agents", "router.agent.md"),
+      skillKind: "agent",
+      passed: false,
+      failureKind: "completion_regression",
+      scenarios,
+    });
+
+    assert.equal(result.status, 0, result.stderr);
+    const verdict = JSON.parse(
+      readFileSync(join(output, "demo", "agent.router", "results.json"), "utf8"),
+    ).verdicts[0];
+    assert.equal(verdict.signTest.wins, 5);
+    assert.equal(verdict.state, "VALID_REGRESSION");
+    assert.equal(verdict.stateReason.code, "native_completion_regression");
+    assert.equal(verdict.passed, false);
+    assert.equal(verdict.regressed, true);
+    assert.equal(verdict.preferenceRegressed, false);
+    assert.match(verdict.reason, /objective task-completion regression/);
+  } finally {
+    rmSync(root, { recursive: true, force: true });
+  }
+});

eng/vally-adapter/adapt.mjs

@@ -1008,7 +1008,7 @@ function pct(x) {
   return `${(x * 100).toFixed(1)}%`;
 }
 
-function comparisonToVerdict(report, identity, roles, nonActivationStims) {
+function comparisonToVerdict(report, identity, roles, nonActivationStims, targetKind = "skill") {
   const s = report.summary;
   const unmatchedBaseline = report.unmatchedBaseline ?? [];
   const unmatchedTreatment = report.unmatchedTreatment ?? [];
@@ -1173,11 +1173,17 @@ function comparisonToVerdict(report, identity, roles, nonActivationStims) {
       baseline: roleToDashboard(baseline),
       skilledIsolated: roleToDashboard(skilled),
     };
+    if (targetKind === "agent") {
+      scenario.agentActivationIsolated = { activated: Boolean(skilled?.activated) };
+    }
     if (hasPlugin) {
       scenario.skillActivationPlugin = {
         activated: Boolean(plugin?.activated),
         ...(plugin?.postActivation ?? {}),
       };
+      if (targetKind === "agent") {
+        scenario.agentActivationPlugin = { activated: Boolean(plugin?.activated) };
+      }
       scenario.skilledPlugin = roleToDashboard(plugin);
     }
     return scenario;
@@ -1248,8 +1254,14 @@ function comparisonToVerdict(report, identity, roles, nonActivationStims) {
     .map((scenario) => ({
       scenarioName: scenario.scenarioName,
       expected: "dormant",
-      observed: scenario.skillActivationIsolated?.activated ? "activated" : "dormant",
-      satisfied: !scenario.skillActivationIsolated?.activated,
+      observed: (targetKind === "agent"
+        ? scenario.agentActivationIsolated?.activated
+        : scenario.skillActivationIsolated?.activated)
+        ? "activated"
+        : "dormant",
+      satisfied: !(targetKind === "agent"
+        ? scenario.agentActivationIsolated?.activated
+        : scenario.skillActivationIsolated?.activated),
     }));
   const activationContractFailures = activationContractScenarios.filter(
     (scenario) => !scenario.satisfied,
@@ -1267,7 +1279,7 @@ function comparisonToVerdict(report, identity, roles, nonActivationStims) {
   const activationContract = {
     evaluated: true,
     requiredForPass: true,
-    source: "isolated_target_skill_activation",
+    source: `isolated_target_${targetKind}_activation`,
     reason:
       "Explicit dormancy expectations are evaluated independently of preference",
     count: activationContractScenarios.length,
@@ -1318,7 +1330,7 @@ function comparisonToVerdict(report, identity, roles, nonActivationStims) {
                       `${discordant} discordant preference vote(s). The sign test conditions on non-tie ` +
                       `stimulus votes and cannot reach ${SIGN_TEST_ALPHA} below ${MIN_CREDIBLE_STIMULI}, so ` +
                       `no record could have passed here — this is not a measured null. Either the ` +
-                      `skill is inert on these scenarios (make them discriminate) or the eval ` +
+                      `${targetKind} is inert on these scenarios (make them discriminate) or the eval ` +
                       `needs more distinct stimuli to clear the ties`
                     : `not credible (sign test p=${pValue.toFixed(3)} > ${SIGN_TEST_ALPHA})`;
 
@@ -1399,6 +1411,7 @@ function comparisonToVerdict(report, identity, roles, nonActivationStims) {
   return {
     skillName: identity.skill,
     skillPath: identity.skillPath,
+    skillKind: targetKind,
     state,
     stateReason,
     conclusive,
@@ -1500,7 +1513,7 @@ function verdictSummaryLine(v) {
   return `${icon} ${v.skillName}: ${v.reason}${scenarios ? "\n" + scenarios : ""}`;
 }
 
-function invalidVerdict(identity, cause, message, accounting = {}) {
+function invalidVerdict(identity, cause, message, accounting = {}, targetKind = "skill") {
   const error = {
     phase: cause.phase,
     kind: cause.kind ?? "permanent",
@@ -1510,6 +1523,7 @@ function invalidVerdict(identity, cause, message, accounting = {}) {
   return {
     skillName: identity.skill,
     skillPath: identity.skillPath,
+    skillKind: targetKind,
     state: VERDICT_STATES.INVALID_INCONCLUSIVE,
     stateReason: { code: cause.code, phase: cause.phase },
     conclusive: false,
@@ -1617,7 +1631,7 @@ function invalidVerdict(identity, cause, message, accounting = {}) {
 
 function writeVerdictResults(outputRoot, evalFile, identity, verdict, expectedEval) {
   const results = {
-    schemaVersion: 4,
+    schemaVersion: 5,
     evalFile,
     model: opts.model,
     judgeModel: opts["judge-model"],

eng/vally-adapter/adapt.test.mjs

@@ -226,6 +226,7 @@ test("retries a transient comparison error once", () => {
     assert.equal(verdict.underpowered, false);
     assert.equal(verdict.passed, true);
     assert.equal(verdict.state, VERDICT_STATES.VALID_PASS);
+    assert.equal(verdict.skillKind, "skill");
     assert.equal(verdict.recoveredErrors.length, 5);
     assert.match(processOutput(result), /without replacing successful judgments/);
   });

eng/vally-adapter/consolidate.mjs

@@ -28,7 +28,7 @@ if (opts.help || (opts.format !== "full" && opts.format !== "simple")) {
   console.log(`Usage:
   node consolidate.mjs --format <full|simple> [--output <file>] [--root <dir>] [--commit <sha>] [<results.json>...]
 
-Consolidates per-skill results.json into a markdown summary table.
+Consolidates per-target results.json into a markdown summary table.
 
 Options:
   --format <full|simple>  full: all metrics and details (workflow summary).
@@ -178,6 +178,17 @@ function fmtOverfit(verdict) {
   return `${icon}${score}`;
 }
 
+function targetActivation(verdict, scenario, arm) {
+  if (verdict.skillKind === "agent") {
+    return arm === "isolated"
+      ? scenario?.agentActivationIsolated
+      : scenario?.agentActivationPlugin;
+  }
+  return arm === "isolated"
+    ? scenario?.skillActivationIsolated
+    : scenario?.skillActivationPlugin;
+}
+
 function activationStats(verdict) {
   const expected = (verdict.scenarios ?? []).filter(
     (scenario) => scenario?.expectActivation !== false,
@@ -185,11 +196,15 @@ function activationStats(verdict) {
   if (expected.length === 0) return null;
   const total = expected.length;
   const isolated = expected.filter(
-    (scenario) => scenario?.skillActivationIsolated?.activated,
+    (scenario) => targetActivation(verdict, scenario, "isolated")?.activated,
   ).length;
-  const hasPlugin = expected.some((scenario) => scenario?.skillActivationPlugin != null);
+  const hasPlugin = expected.some(
+    (scenario) => targetActivation(verdict, scenario, "plugin") != null,
+  );
   const plugin = hasPlugin
-    ? expected.filter((scenario) => scenario?.skillActivationPlugin?.activated).length
+    ? expected.filter(
+        (scenario) => targetActivation(verdict, scenario, "plugin")?.activated,
+      ).length
     : null;
   return {
     total,
@@ -245,23 +260,24 @@ function scenarioStats(scenario) {
   };
 }
 
-function isWeakOrWarningScenario(scenario) {
+function isWeakOrWarningScenario(verdict, scenario) {
   const { netWin } = scenarioStats(scenario);
+  const isolatedActivation = targetActivation(verdict, scenario, "isolated");
+  const pluginActivation = targetActivation(verdict, scenario, "plugin");
   return netWin <= 0
     || scenario?.timedOut === true
     || (scenario?.skillActivationIsolated?.failedActivationOnlyCompletions ?? 0) > 0
     || (scenario?.skillActivationPlugin?.failedActivationOnlyCompletions ?? 0) > 0
     || (scenario?.expectActivation === false
-      && scenario?.skillActivationIsolated?.activated === true)
+      && isolatedActivation?.activated === true)
     || (scenario?.expectActivation !== false
-      && (!scenario?.skillActivationIsolated?.activated
-        || (scenario?.skillActivationPlugin != null
-          && !scenario.skillActivationPlugin.activated)));
+      && (!isolatedActivation?.activated
+        || (pluginActivation != null && !pluginActivation.activated)));
 }
 
 function scenarioTable(verdict, weakOnly = false) {
   const scenarios = (verdict.scenarios ?? []).filter(
-    (scenario) => !weakOnly || isWeakOrWarningScenario(scenario),
+    (scenario) => !weakOnly || isWeakOrWarningScenario(verdict, scenario),
   );
   if (scenarios.length === 0) return [];
   const rows = [
@@ -288,13 +304,13 @@ function representativeEvidence(verdict) {
   const scenarios = [...(verdict.scenarios ?? [])].sort((left, right) => {
     const priority = (scenario) => {
       if (scenario.preferenceGateEligible !== false) return 0;
-      if (scenario.skillActivationIsolated?.activated === true) return 1;
+      if (targetActivation(verdict, scenario, "isolated")?.activated === true) return 1;
       return 2;
     };
     return priority(left) - priority(right);
   });
   for (const scenario of scenarios) {
-    if (!isWeakOrWarningScenario(scenario)) continue;
+    if (!isWeakOrWarningScenario(verdict, scenario)) continue;
     const trials = (scenario.trials ?? []).filter((trial) => !trial.errored);
     const trial = trials.find((candidate) => trialDirection(candidate) < 0)
       ?? trials.find((candidate) => trialDirection(candidate) === 0);
@@ -476,7 +492,9 @@ const noChangeCount = verdicts.length
   - regressedCount
   - activationContractFailureCount
   - preferenceRegressedCount;
-const skillCount = new Set(verdicts.map((verdict) => verdict.skillName)).size;
+const targetCount = new Set(
+  verdicts.map((verdict) => `${verdict.skillKind ?? "skill"}:${verdict.skillName}`),
+).size;
 const models = [...new Set(verdicts.map((verdict) => verdict.model))];
 const judges = [...new Set(verdicts.map((verdict) => verdict.judgeModel))];
 const objectiveGateEnabled = regressedCount > 0
@@ -484,7 +502,7 @@ const objectiveGateEnabled = regressedCount > 0
 const isFull = opts.format === "full";
 
 const compactHeader = [
-  "Skill",
+  "Target",
   "Model",
   "Verdict",
   "Gate evidence",
@@ -493,7 +511,7 @@ const compactHeader = [
   "Next action",
 ];
 const fullHeader = [
-  "Skill",
+  "Target",
   "Model",
   "Verdict",
   "Gate evidence",
@@ -506,11 +524,11 @@ const fullHeader = [
   "Next action",
 ];
 const header = isFull ? fullHeader : compactHeader;
-const lines = ["## 📊 Skill Evaluation Results", ""];
+const lines = ["## 📊 Skill and Agent Evaluation Results", ""];
 
 lines.push(
-  `${countNoun(verdicts.length, "model/skill result")} across `
-  + `${countNoun(skillCount, "skill")} and ${countNoun(models.length, "model")} — `
+  `${countNoun(verdicts.length, "model/target result")} across `
+  + `${countNoun(targetCount, "target")} and ${countNoun(models.length, "model")} — `
   + `✅ **${passedCount} improved**, ➖ **${noChangeCount} not proven improved**, `
   + `⚠️ **${underpoweredCount + invalidCount} invalid or underpowered**, `
   + `⛔ **${countNoun(activationContractFailureCount, "activation contract failure")}**, `
@@ -569,7 +587,7 @@ lines.push(
 lines.push("");
 
 if (verdicts.length === 0) {
-  lines.push("_No skill verdicts were produced._");
+  lines.push("_No target verdicts were produced._");
 } else {
   lines.push(`| ${header.join(" | ")} |`);
   lines.push(`|${header.map(() => "---").join("|")}|`);
@@ -605,10 +623,10 @@ if (verdicts.length === 0) {
   lines.push("");
   lines.push("- **✅ Improved** — the result passed both the statistical gate and the 20% practical net-win floor.");
   lines.push("- **➖ Not proven improved** — the result is valid but did not pass both gates. This is not automatically a regression.");
-  lines.push("- **⚠️ Invalid / underpowered** — the gate withheld a quality verdict. Fix the measurement before judging the skill.");
-  lines.push("- **⛔ Activation contract failed** — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.");
+  lines.push("- **⚠️ Invalid / underpowered** — the gate withheld a quality verdict. Fix the measurement before judging the target.");
+  lines.push("- **⛔ Activation contract failed** — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.");
   lines.push("- **📉 Preference loss** — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.");
-  lines.push("- **Gate evidence** — `n` preference-eligible distinct-stimulus votes, W/T/L stimulus votes, `d` discordant votes, exact one-sided `p`, net win, and the count of separately retained dormancy stimuli. The `p` value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.");
+  lines.push("- **Gate evidence** — `n` preference-eligible distinct-stimulus votes, W/T/L stimulus votes, `d` discordant votes, exact one-sided `p`, net win, and the count of separately retained dormancy stimuli. The `p` value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.");
   lines.push("- **Overfit** — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.");
   lines.push("- **Warnings** — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.");
   if (isFull) {
@@ -731,7 +749,7 @@ const markdown = lines.join("\n");
 if (opts.output) {
   writeFileSync(opts.output, markdown);
   console.error(
-    `Wrote ${opts.format} summary (${countNoun(verdicts.length, "model/skill result")}) to ${opts.output}`,
+    `Wrote ${opts.format} summary (${countNoun(verdicts.length, "model/target result")}) to ${opts.output}`,
   );
 } else {
   process.stdout.write(`${markdown}\n`);

eng/vally-adapter/consolidate.test.mjs

@@ -138,6 +138,57 @@ test("keeps routine passing details out of the PR comment but in Full Results",
   assert.match(full, /<code>VALID_PASS<\/code>/);
 });
 
+test("uses target-agent activation evidence for agent verdicts", () => {
+  const markdown = render([{
+    skillName: "agent.router",
+    skillKind: "agent",
+    state: "VALID_PASS",
+    passed: true,
+    conclusive: true,
+    reason: "credible preference improvement",
+    scenarios: [{
+      scenarioName: "route request",
+      expectActivation: true,
+      agentActivationIsolated: { activated: true },
+      agentActivationPlugin: { activated: false },
+    }],
+  }], { format: "full" });
+
+  assert.match(markdown, /Activation: isolated 1\/1; plugin 0\/1/);
+  assert.doesNotMatch(markdown, /Activation: isolated 0\/1/);
+});
+
+test("selects weak scenarios using only target-agent activation", () => {
+  const markdown = render([{
+    skillName: "agent.router",
+    skillKind: "agent",
+    state: "VALID_PASS",
+    passed: true,
+    conclusive: true,
+    reason: "credible preference improvement",
+    scenarios: [
+      {
+        scenarioName: "missing target agent",
+        expectActivation: true,
+        netWin: 1,
+        skillActivationIsolated: { activated: true },
+        agentActivationIsolated: { activated: false },
+      },
+      {
+        scenarioName: "correctly dormant target agent",
+        expectActivation: false,
+        netWin: 1,
+        skillActivationIsolated: { activated: true },
+        agentActivationIsolated: { activated: false },
+      },
+    ],
+  }]);
+
+  assert.match(markdown, /\*\*Weak or warning scenarios:\*\*/);
+  assert.match(markdown, /missing target agent/);
+  assert.doesNotMatch(markdown, /correctly dormant target agent/);
+});
+
 test("preserves execution model identity and aggregates measurement health", () => {
   const verdict = {
     skillName: "same-skill",
@@ -172,7 +223,7 @@ test("preserves execution model identity and aggregates measurement health", ()
     commit: "abc123",
   });
 
-  assert.match(markdown, /2 model\/skill results across 1 skill and 2 models/);
+  assert.match(markdown, /2 model\/target results across 1 target and 2 models/);
   assert.match(markdown, /\| same-skill \| model-a \| ✅ Improved \|/);
   assert.match(markdown, /\| same-skill \| model-b \| ✅ Improved \|/);
   assert.match(markdown, /evaluated commit `abc123`; judge `judge-a`/);
@@ -266,7 +317,7 @@ test("keeps Overfit visible and gives actionable evidence for a non-pass", () =>
     },
   ]);
 
-  assert.match(markdown, /\| Skill \| Model \| Verdict \| Gate evidence \| Overfit \| Warnings \| Next action \|/);
+  assert.match(markdown, /\| Target \| Model \| Verdict \| Gate evidence \| Overfit \| Warnings \| Next action \|/);
   assert.match(markdown, /n=5; 4W\/1T\/0L; d=4; p=0.063; net \+40.0%/);
   assert.match(markdown, /🟡 0.51/);
   assert.match(markdown, /Activation: isolated 0\/1/);

plugins/dotnet-msbuild/agents/msbuild.agent.md

@@ -3,6 +3,9 @@ name: msbuild
 description: "Expert agent for MSBuild and .NET build troubleshooting, optimization, and project file quality. Routes to specialized agents for performance analysis and code review. Verifies MSBuild domain relevance before deep-diving. Specializes in build configuration, error diagnosis, binary log analysis, and resolving common build issues."
 user-invokable: true
 disable-model-invocation: false
+agents:
+  - build-perf
+  - msbuild-code-review
 license: MIT
 ---
 

plugins/dotnet-test/agents/testability-migration.agent.md

@@ -7,6 +7,8 @@ description: >-
   Use when asked to make code testable, remove static coupling, migrate to
   TimeProvider, adopt IFileSystem, or improve testability of a legacy codebase.
 name: testability-migration
+agents:
+  - code-testing-generator
 handoffs:
   - label: Generate Tests for Migrated Code
     agent: code-testing-generator

tests/dotnet-msbuild/agent.msbuild/eval.yaml

@@ -1,7 +1,7 @@
 name: agent.msbuild
-description: Evaluates the dotnet-msbuild/agent.msbuild skill
+description: Evaluates the dotnet-msbuild/msbuild custom agent
 type: capability
-config:
+defaults:
   timeout: 3m
 stimuli:
   - name: Triage a build failure and route to appropriate analysis
@@ -12,8 +12,8 @@ stimuli:
         - src: .
           dest: .
       skills:
-        - binlog-failure-analysis
-        - binlog-generation
+        - ../../plugins/dotnet-msbuild/skills/binlog-failure-analysis
+        - ../../plugins/dotnet-msbuild/skills/binlog-generation
     graders:
       - type: output-contains
         config:
@@ -40,3 +40,79 @@ stimuli:
       - Recommended using Directory.Build.props for shared properties like TargetFramework
       - Explained how Directory.Build.props works (automatic import by MSBuild for all projects in the directory tree)
       - Suggested moving duplicated PackageReferences or common properties into the shared file
+  - name: Route a slow build to performance analysis
+    prompt: |
+      This small .NET project builds successfully, but our real solution has
+      become much slower over time. Explain how you would establish a baseline,
+      capture the right evidence, and identify the expensive targets, tasks, and
+      analyzers before changing the build.
+    environment:
+      files:
+        - src: .
+          dest: .
+      skills:
+        - ../../plugins/dotnet-msbuild/skills/binlog-generation
+        - ../../plugins/dotnet-msbuild/skills/build-perf-baseline
+        - ../../plugins/dotnet-msbuild/skills/build-perf-diagnostics
+    graders:
+      - type: output-matches
+        config:
+          pattern: (baseline|cold build|warm build|no-op build)
+      - type: output-matches
+        config:
+          pattern: (binlog|binary log)
+      - type: output-matches
+        config:
+          pattern: (target|task|analyzer)
+      - type: prompt
+    rubric:
+      - Proposed measuring comparable cold, warm, and no-op builds before optimizing
+      - Recommended collecting a binary log and inspecting expensive targets, tasks, or analyzers
+      - Avoided guessing at optimizations before gathering evidence
+  - name: Diagnose broken incremental build behavior
+    prompt: |
+      Rebuilding without source changes still reruns an expensive custom target.
+      Show me how to diagnose why the target is not being skipped and what input,
+      output, or timestamp mistakes to look for in the project files.
+    environment:
+      files:
+        - src: .
+          dest: .
+      skills:
+        - ../../plugins/dotnet-msbuild/skills/incremental-build
+    graders:
+      - type: output-matches
+        config:
+          pattern: (Inputs|Outputs|incremental|up.to.date|skip)
+      - type: output-matches
+        config:
+          pattern: (timestamp|changed|missing|mapping)
+      - type: prompt
+    rubric:
+      - Explained that MSBuild target skipping depends on correct Inputs and Outputs relationships
+      - Identified concrete timestamp, missing-output, or one-to-one mapping checks
+      - Proposed a repeatable no-change rebuild to verify the fix
+  - name: Review a project file for maintainability risks
+    prompt: |
+      Review this .NET project file for build configuration maintainability and correctness
+      risks. Prioritize concrete issues, explain their impact, and suggest
+      targeted improvements without redesigning unrelated application code.
+    environment:
+      files:
+        - src: .
+          dest: .
+      skills:
+        - ../../plugins/dotnet-msbuild/skills/msbuild-antipatterns
+        - ../../plugins/dotnet-msbuild/skills/msbuild-modernization
+    graders:
+      - type: output-matches
+        config:
+          pattern: (PackageReference|TargetFramework|property|item|condition|target)
+      - type: output-matches
+        config:
+          pattern: (risk|issue|recommend|improv)
+      - type: prompt
+    rubric:
+      - Grounded findings in the supplied project file rather than giving generic MSBuild advice
+      - Prioritized correctness and maintainability issues by impact
+      - Kept recommendations scoped to build configuration

tests/dotnet-test-migration/agent.test-migration/eval.yaml

@@ -1,7 +1,7 @@
 name: agent.test-migration
 description: Evaluates the dotnet-test/agent.test-migration orchestrator agent
 type: capability
-config:
+defaults:
   timeout: 5m
 stimuli:
   - name: Detect MSTest v2 on VSTest and recommend migration path
@@ -98,6 +98,81 @@ stimuli:
       - Addressed or warned about the Assert.AreEqual object overload removal
       - Did not attempt additional migrations (v3 to v4, VSTest to MTP) that were not requested
 
+  - name: Execute targeted xUnit v2 to v3 migration
+    prompt: |
+      Upgrade this xUnit.net v2 test project to xUnit.net v3. Apply only the
+      framework-version migration, preserve the current test platform, and make
+      the project build after the change.
+    environment:
+      files:
+        - src: fixtures/xunit-v2/CalcTests.csproj
+          dest: CalcTests/CalcTests.csproj
+        - src: fixtures/xunit-v2/CalculatorTests.cs
+          dest: CalcTests/CalculatorTests.cs
+      skills:
+        - ../../plugins/dotnet-test/skills/platform-detection
+        - ../../plugins/dotnet-test-migration/skills/migrate-xunit-to-xunit-v3
+    graders:
+      - type: file-contains
+        config:
+          path: "**/CalcTests.csproj"
+          value: xunit.v3
+      - type: file-not-contains
+        config:
+          path: "**/CalcTests.csproj"
+          value: "2.9.3"
+      - type: run-command
+        config:
+          command: dotnet build CalcTests/CalcTests.csproj
+          expected_exit_code: 0
+          timeout: 5m
+      - type: prompt
+    rubric:
+      - Migrated the project from xUnit.net v2 to the v3 package model
+      - Preserved the existing test platform rather than adding an unrelated platform migration
+      - Updated project settings required by xUnit.net v3 and left the project buildable
+
+  - name: Plan a staged MSTest v2 to v4 and MTP migration
+    prompt: |
+      I want this MSTest 2.2.10 project to end up on MSTest v4 and
+      Microsoft.Testing.Platform, but I need a safe staged plan before any
+      edits. Inspect the project, order the migration steps, identify validation
+      gates between them, and do not change files yet.
+    environment:
+      files:
+        - src: fixtures/mstest-v2-vstest/OrderTests.csproj
+          dest: OrderTests/OrderTests.csproj
+        - src: fixtures/mstest-v2-vstest/OrderServiceTests.cs
+          dest: OrderTests/OrderServiceTests.cs
+      skills:
+        - ../../plugins/dotnet-test/skills/platform-detection
+        - ../../plugins/dotnet-test-migration/skills/migrate-mstest-v1v2-to-v3
+        - ../../plugins/dotnet-test-migration/skills/migrate-mstest-v3-to-v4
+        - ../../plugins/dotnet-test-migration/skills/migrate-vstest-to-mtp
+    graders:
+      - type: output-matches
+        config:
+          pattern: (v2.*v3|2\.2\.10.*3)
+      - type: output-matches
+        config:
+          pattern: (v3.*v4|3.*4)
+      - type: output-matches
+        config:
+          pattern: (Microsoft\.Testing\.Platform|MTP)
+      - type: output-matches
+        config:
+          pattern: (build|test|verify|validation)
+      - type: run-command
+        config:
+          command: sh -c "grep -Rqs '2.2.10' OrderTests"
+          expected_exit_code: 0
+          timeout: 1m
+      - type: prompt
+    rubric:
+      - Ordered the migration as MSTest v2 to v3, then v3 to v4, then the platform change
+      - Included a build and test validation gate between migration stages
+      - Kept the request plan-only and did not modify the supplied project
+
   - name: Decline request to write new tests
     prompt: |
       I need to add unit tests for my Calculator class. Can you generate

tests/dotnet-test/agent.code-testing-generator/eval.yaml

@@ -0,0 +1,199 @@
+name: agent.code-testing-generator
+description: Evaluates the dotnet-test/code-testing-generator custom-agent pipeline
+type: capability
+executionShard: generation
+defaults:
+  timeout: 60m
+  runs: 1
+stimuli:
+  - name: Generate a project-wide pytest suite across modules
+    prompt: |
+      Generate a comprehensive pytest suite for the project under
+      fixtures/python-multimodule/. Cover the public behavior of analytics.stats,
+      analytics.window.RateWindow, and textkit.slug, including validation,
+      boundaries, rollover, and truncation. Run pytest and map every requested
+      behavior to named test evidence.
+    environment:
+      files:
+        - src: ../code-testing-agent/fixtures/python-multimodule
+          dest: fixtures/python-multimodule
+      commands:
+        - git init -q
+      skills:
+        - ../../plugins/dotnet-test/skills/code-testing-agent
+        - ../../plugins/dotnet-test/skills/code-testing-extensions
+        - ../../plugins/dotnet-test/skills/test-gap-analysis
+        - ../../plugins/dotnet-test/skills/assertion-quality
+    graders:
+      - type: run-command
+        config:
+          command: sh -c "cd fixtures/python-multimodule && python3 -m pip install --quiet pytest && python3 -m pytest -q"
+          expected_exit_code: 0
+          timeout: 5m
+      - type: output-matches
+        config:
+          pattern: \|\s*Requirement\s*\|\s*Evidence\s*\|
+      - type: run-command
+        config:
+          command: state_dir="$(git rev-parse --path-format=absolute --git-path testagent)" && test -f "$state_dir/research.md" && test -f "$state_dir/plan.md" && test -f "$state_dir/status.md"
+          expected_exit_code: 0
+          timeout: 1m
+      - type: prompt
+    rubric:
+      - Generated passing tests for all three modules and their distinct boundaries
+      - Used the broad research, plan, implementation, and quality-review pipeline
+      - Mapped every requested behavior to concrete test evidence
+
+  - name: Preserve a classic MSTest project while adding broad coverage
+    prompt: |
+      Add the missing project-wide unit tests for the classic net472 library
+      under fixtures/classic-mstest/. Preserve MSTest 3.5.2, Moq 4.2,
+      packages.config, explicit compile items, and the existing test file.
+      Cover DiscountService and TieredDiscountPolicy validation and boundaries,
+      then report exact evidence for each requirement.
+    environment:
+      files:
+        - src: ../code-testing-agent/fixtures/classic-mstest
+          dest: fixtures/classic-mstest
+      commands:
+        - rm -f fixtures/classic-mstest/tests/DiscountServiceBoundaryTests.cs fixtures/classic-mstest/tests/TieredDiscountPolicyTests.cs && cp fixtures/classic-mstest/tests/Discounts.Tests.csproj.pristine fixtures/classic-mstest/tests/Discounts.Tests.csproj && rm fixtures/classic-mstest/tests/Discounts.Tests.csproj.pristine
+        - git init -q
+      skills:
+        - ../../plugins/dotnet-test/skills/code-testing-agent
+        - ../../plugins/dotnet-test/skills/code-testing-extensions
+        - ../../plugins/dotnet-test/skills/test-gap-analysis
+        - ../../plugins/dotnet-test/skills/assertion-quality
+    graders:
+      - type: file-exists
+        config:
+          path: fixtures/classic-mstest/tests/DiscountServiceBoundaryTests.cs
+      - type: file-exists
+        config:
+          path: fixtures/classic-mstest/tests/TieredDiscountPolicyTests.cs
+      - type: run-command
+        config:
+          command: sh -c "grep -q 'Compile Include=\"DiscountServiceBoundaryTests.cs\"' fixtures/classic-mstest/tests/Discounts.Tests.csproj && grep -q 'Compile Include=\"TieredDiscountPolicyTests.cs\"' fixtures/classic-mstest/tests/Discounts.Tests.csproj"
+          expected_exit_code: 0
+          timeout: 1m
+      - type: output-matches
+        config:
+          pattern: \|\s*Requirement\s*\|\s*Evidence\s*\|
+      - type: prompt
+    rubric:
+      - Added and registered tests for both requested production types
+      - Preserved the classic project format and pinned dependency stack
+      - Used MSTest APIs compatible with version 3.5.2
+
+  - name: Generate collaborating Go package tests
+    prompt: |
+      Generate comprehensive Go tests for the module under
+      fixtures/go-multipackage/. Cover money.Discount, shipping.Rate, and
+      order.Total, including validation, exact boundaries, collaborator errors,
+      and short-circuit behavior. Use fakes where appropriate and make
+      `go test ./...` pass.
+    environment:
+      files:
+        - src: ../code-testing-agent/fixtures/go-multipackage
+          dest: fixtures/go-multipackage
+      commands:
+        - git init -q
+      skills:
+        - ../../plugins/dotnet-test/skills/code-testing-agent
+        - ../../plugins/dotnet-test/skills/code-testing-extensions
+        - ../../plugins/dotnet-test/skills/test-gap-analysis
+        - ../../plugins/dotnet-test/skills/assertion-quality
+    graders:
+      - type: run-command
+        config:
+          command: sh -c "cd fixtures/go-multipackage && go test ./..."
+          expected_exit_code: 0
+          timeout: 5m
+      - type: run-command
+        config:
+          command: sh -c "test -f fixtures/go-multipackage/money/discount_test.go && test -f fixtures/go-multipackage/shipping/rate_test.go && test -f fixtures/go-multipackage/order/total_test.go"
+          expected_exit_code: 0
+          timeout: 1m
+      - type: output-matches
+        config:
+          pattern: \|\s*Requirement\s*\|\s*Evidence\s*\|
+      - type: prompt
+    rubric:
+      - Generated passing tests in all three packages
+      - Proved exact boundaries and collaborator error propagation
+      - Used concrete assertions and mapped each requirement to evidence
+
+  - name: Generate layered Vitest coverage for an async cart
+    prompt: |
+      Generate a comprehensive Vitest suite for the TypeScript shopping-cart
+      library under fixtures/typescript-vitest-cart/. Cover pricing, tax,
+      shipping, inventory, and Cart behavior, including async refresh,
+      collaborator failures, snapshot isolation, and configured coverage
+      thresholds. Use fakes instead of real I/O.
+    environment:
+      files:
+        - src: ../code-testing-agent/fixtures/typescript-vitest-cart
+          dest: fixtures/typescript-vitest-cart
+      commands:
+        - git init -q
+      skills:
+        - ../../plugins/dotnet-test/skills/code-testing-agent
+        - ../../plugins/dotnet-test/skills/code-testing-extensions
+        - ../../plugins/dotnet-test/skills/test-gap-analysis
+        - ../../plugins/dotnet-test/skills/assertion-quality
+    graders:
+      - type: run-command
+        config:
+          command: sh -c "cd fixtures/typescript-vitest-cart && npm ci --silent && npm run test:coverage"
+          expected_exit_code: 0
+          timeout: 10m
+      - type: run-command
+        config:
+          command: sh -c "grep -R -q 'InventoryError' fixtures/typescript-vitest-cart/tests && grep -R -q 'checkout' fixtures/typescript-vitest-cart/tests"
+          expected_exit_code: 0
+          timeout: 1m
+      - type: output-matches
+        config:
+          pattern: \|\s*Requirement\s*\|\s*Evidence\s*\|
+      - type: prompt
+    rubric:
+      - Generated passing tests for every production module without real I/O
+      - Covered the requested async, composition, failure, and isolation behaviors
+      - Cleared the configured coverage thresholds and mapped requirements to evidence
+
+  - name: Generate project-wide xUnit tests for a .NET library
+    prompt: |
+      Generate the complete xUnit v3 suite for the .NET 10 library under
+      fixtures/sdk-xunit-orders/. Cover OrderPricing.Total and ReservationWindow,
+      including validation, exact decimal values, and every before/at boundary.
+      Keep production code unchanged and run the existing test project.
+    environment:
+      files:
+        - src: ../code-testing-agent/fixtures/sdk-xunit-orders
+          dest: fixtures/sdk-xunit-orders
+      commands:
+        - find fixtures/sdk-xunit-orders/tests -type f -name '*.cs' -delete
+        - git init -q
+      skills:
+        - ../../plugins/dotnet-test/skills/code-testing-agent
+        - ../../plugins/dotnet-test/skills/code-testing-extensions
+        - ../../plugins/dotnet-test/skills/test-gap-analysis
+        - ../../plugins/dotnet-test/skills/assertion-quality
+    graders:
+      - type: run-command
+        config:
+          command: sh -c "dotnet test fixtures/sdk-xunit-orders/tests/Orders.Tests.csproj"
+          expected_exit_code: 0
+          timeout: 10m
+      - type: run-command
+        config:
+          command: sh -c "grep -R -q 'OrderPricing' fixtures/sdk-xunit-orders/tests --include='*.cs' && grep -R -q 'ReservationWindow' fixtures/sdk-xunit-orders/tests --include='*.cs'"
+          expected_exit_code: 0
+          timeout: 1m
+      - type: output-matches
+        config:
+          pattern: \|\s*Requirement\s*\|\s*Evidence\s*\|
+      - type: prompt
+    rubric:
+      - Generated passing xUnit v3 tests for both production types
+      - Covered every requested validation, pricing, and time boundary
+      - Preserved production code and mapped each behavior to exact test evidence

tests/dotnet-test/agent.test-quality-auditor/eval.yaml

@@ -1,7 +1,7 @@
 name: agent.test-quality-auditor
 description: Evaluates the dotnet-test/agent.test-quality-auditor orchestrator agent
 type: capability
-config:
+defaults:
   timeout: 5m
 stimuli:
   - name: Comprehensive test quality audit of weak test suite
@@ -102,6 +102,64 @@ stimuli:
       - Flagged tests with no assertions or meaningless assertions
       - Pointed out missing assertion types (exception assertions, collection/state checks, structural verification)
 
+  - name: Identify behavior gaps that existing tests would miss
+    prompt: |
+      Review the ShoppingCart production code and its current tests. Identify
+      concrete behavior changes or boundary bugs that the suite would fail to
+      catch, and prioritize the missing tests that would close those gaps.
+    environment:
+      files:
+        - src: fixtures/weak-tests/WeakTests.csproj
+          dest: WeakTests/WeakTests.csproj
+        - src: fixtures/weak-tests/ShoppingCart.cs
+          dest: WeakTests/ShoppingCart.cs
+        - src: fixtures/weak-tests/ShoppingCartTests.cs
+          dest: WeakTests/ShoppingCartTests.cs
+      skills:
+        - ../../plugins/dotnet-test/skills/test-gap-analysis
+    graders:
+      - type: output-matches
+        config:
+          pattern: (RemoveItem|GetTotalWithDiscount|boundary|negative|empty)
+      - type: output-matches
+        config:
+          pattern: (mutation|would.*pass|would.*miss|gap|not.*detect)
+      - type: exit-success
+      - type: prompt
+    rubric:
+      - Identified concrete production behaviors that the current tests do not protect
+      - Explained at least one plausible bug that would survive the current suite
+      - Prioritized missing tests by risk rather than proposing generic coverage increases
+
+  - name: Diagnose test smells and propose a repair order
+    prompt: |
+      Assess the maintainability and reliability of ShoppingCartTests. Call out
+      the most important test smells, explain why each one is risky, and give a
+      practical order for repairing the suite without changing production code.
+    environment:
+      files:
+        - src: fixtures/weak-tests/WeakTests.csproj
+          dest: WeakTests/WeakTests.csproj
+        - src: fixtures/weak-tests/ShoppingCart.cs
+          dest: WeakTests/ShoppingCart.cs
+        - src: fixtures/weak-tests/ShoppingCartTests.cs
+          dest: WeakTests/ShoppingCartTests.cs
+      skills:
+        - ../../plugins/dotnet-test/skills/test-smell-detection
+    graders:
+      - type: output-matches
+        config:
+          pattern: (smell|brittle|flaky|maintain|reliab)
+      - type: output-matches
+        config:
+          pattern: (priority|first|critical|high)
+      - type: exit-success
+      - type: prompt
+    rubric:
+      - Identified multiple concrete maintainability or reliability problems in the supplied tests
+      - Explained the behavioral risk of the findings instead of listing labels only
+      - Proposed a prioritized repair sequence that keeps production code unchanged
+
   - name: Decline request to generate new tests
     prompt: |
       I need you to write comprehensive unit tests for the ShoppingCart class.

tests/dotnet-test/agent.testability-migration/eval.yaml

@@ -106,3 +106,75 @@ stimuli:
       - Added deterministic tests proving StartedAt, the 14-day ExpiresAt value, active-before-expiry, and expired-at-boundary behavior
       - Used a controlled TimeProvider rather than wall-clock timing
       - Ran the tests successfully instead of stopping after migration or build
+
+  - name: "Replace filesystem statics without touching unrelated dependencies"
+    prompt: |
+      Replace only the File and Directory calls in SubscriptionManager with an
+      injectable filesystem abstraction. Wire the production implementation and
+      keep DateTime, Environment, and Console usage unchanged. Build the project
+      after the migration.
+    environment:
+      files:
+        - src: ./fixtures/full-pipeline
+          dest: FullPipeline
+      skills:
+        - ../../plugins/dotnet-test/skills/generate-testability-wrappers
+        - ../../plugins/dotnet-test/skills/migrate-static-to-wrapper
+    graders:
+      - type: output-matches
+        config:
+          pattern: (IFileSystem|System\.IO\.Abstractions|file system|filesystem)
+      - type: run-command
+        config:
+          command: >-
+            python -c "import pathlib,re,sys; text=next(pathlib.Path('.').rglob('SubscriptionManager.cs')).read_text(); sys.exit(1 if re.search(r'(?<![\w.])(?:System\.IO\.)?(?:File|Directory)\.', text) else 0)"
+      - type: file-contains
+        config:
+          path: "**/SubscriptionManager.cs"
+          value: DateTime.UtcNow
+      - type: run-command
+        config:
+          command: dotnet build FullPipeline
+          expected_exit_code: 0
+          timeout: 5m
+      - type: prompt
+    rubric:
+      - Replaced the filesystem statics with one injectable abstraction and wired its production implementation
+      - Preserved DateTime, Environment, and Console calls because they were outside the requested scope
+      - Left the project in a buildable state
+
+  - name: "Inventory static dependencies without modifying the project"
+    prompt: |
+      Analyze SubscriptionManager for dependencies that make deterministic unit
+      testing difficult. Group the findings by dependency category, identify the
+      affected members, and rank the migration order. This is analysis only; do
+      not edit any files.
+    environment:
+      files:
+        - src: ./fixtures/full-pipeline
+          dest: FullPipeline
+      skills:
+        - ../../plugins/dotnet-test/skills/detect-static-dependencies
+    graders:
+      - type: output-matches
+        config:
+          pattern: (DateTime\.UtcNow|time)
+      - type: output-matches
+        config:
+          pattern: (File\.|Directory\.|filesystem|file system)
+      - type: output-matches
+        config:
+          pattern: (Environment\.|Console\.)
+      - type: output-matches
+        config:
+          pattern: (priority|rank|order|first)
+      - type: run-command
+        config:
+          command: sh -c "grep -Rqs 'DateTime.UtcNow' FullPipeline && grep -Rqs 'File\\.' FullPipeline"
+          expected_exit_code: 0
+          timeout: 1m
+      - type: prompt
+    rubric:
+      - Identified time, filesystem, environment, and console dependencies with their affected members
+      - Ranked the migration sequence based on testing value and implementation risk
+      - Kept the analysis read-only and did not alter the supplied project