skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fix eval investigation docs: download command, schema accuracy, example script (#449)

Browse Source
This commit is contained in:
Dan Moseley
2026-03-25 15:24:33 -06:00
committed by GitHub
parent 7286c31118
commit 604c46c8c6
4 changed files with 34 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/evaluation.yml
+1 -1
View File
@@ -673,7 +673,7 @@ jobs:
echo ""
echo "> **To investigate failures**, paste this to your AI coding agent:"
echo ">"
echo "> _Download eval artifacts with \`gh run download ${RUN_ID} --repo ${{ github.repository }} --dir /tmp/eval-results\`, then fetch https://raw.githubusercontent.com/${{ github.repository }}/${{ needs.gate.outputs.head_sha }}/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._"
echo "> _Download eval artifacts with \`gh run download ${RUN_ID} --repo ${{ github.repository }} --pattern \"skill-validator-results-*\" --dir ./eval-results\`, then fetch https://raw.githubusercontent.com/${{ github.repository }}/${{ needs.gate.outputs.head_sha }}/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._"
fi
} > consolidated-comment.md
.github/workflows/skill-validator.yml
+1 -1
View File
@@ -38,7 +38,7 @@ jobs:
$head = "${{ github.event.pull_request.head.sha }}"
$mergeBase = git merge-base $base $head
$changed = git diff --name-only $mergeBase $head |
Where-Object { $_ -match '^(eng/skill-validator/|\.github/workflows/skill-validator\.yml$)' } |
Where-Object { $_ -match '^(eng/skill-validator/|\.github/workflows/skill-validator\.yml$)' -and $_ -ne 'eng/skill-validator/InvestigatingResults.md' } |
Select-Object -First 1
if ($changed) {
echo "has_changes=true" >> $env:GITHUB_OUTPUT
AGENTS.md
+2
View File
@@ -15,3 +15,5 @@ This validates skill frontmatter and recompiles knowledge lock files. Always com
## Skill-Validator
Don't care much about backwards-compatibility for this tool. Consumers understand that the shape is constantly changing.
When modifying the evaluation pipeline (`evaluation.yml`), results JSON schema (`Models.cs`), or the skill-validator evaluation logic, review and update `eng/skill-validator/InvestigatingResults.md` to keep the failure investigation guidance, schema documentation, and example scripts in sync.
eng/skill-validator/InvestigatingResults.md
+30 -12
View File
@@ -10,7 +10,7 @@ If you need to run the investigation manually, follow the [Quick start](#quick-s
## Quick start
1. **Download the results artifact:** `gh run download <run-id> --repo dotnet/skills --dir <path>`
1. **Download the results artifact:** `gh run download <run-id> --repo dotnet/skills --pattern "skill-validator-results-*" --dir <path>`
2. **Read `summary.md` first** for a quick overview of which scenarios passed/failed
3. **Read `results.json`** for the full metrics, agent output, assertions, and judge reasoning
4. **Identify the failure pattern** using the categories below — most failures match multiple patterns; fix them in priority order (timeouts first, then activation, then quality/rubric issues)
@@ -23,10 +23,12 @@ If you need to run the investigation manually, follow the [Quick start](#quick-s
Extract the workflow run ID from the **Full results** link in the PR eval comment (e.g., `https://github.com/dotnet/skills/actions/runs/23520818616` → `23520818616`), then:
```bash
gh run download <run-id> --repo dotnet/skills --dir /tmp/eval-results
gh run download <run-id> --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results
```
This downloads all artifacts into subdirectories, each containing `results.json` and `summary.md`.
This downloads all result artifacts into subdirectories, each containing `results.json` and `summary.md`.
> **Note:** The `--pattern` flag is important — without it, `gh` will attempt to download all workflow artifacts including non-zip files (e.g., `.tar.gz`), which causes an extraction error and a non-zero exit code even though the eval results download successfully.
### Via browser
@@ -77,7 +79,9 @@ Each scenario includes two required runs (baseline + isolated). It may also incl
| `isolatedBreakdown` | Per-metric contribution to the score (see below) |
| `pluginBreakdown` | Per-metric contribution to the score (see below); optional and only populated when a plugin run is present |
| `pairwiseResult` | Judge's rubric-by-rubric comparison |
| `perRunScores` | Individual run scores (shows variance) |
| `perRunScores` | Per-run improvement scores as a flat array of numbers (one per run); when a plugin run is present, each value is `min(isolated, plugin)` for that run; when no plugin run is present (`skilledPlugin` is null), each value is the isolated improvement score for that run |
> **Note:** Scenarios do not have a `passed` field. To determine pass/fail for an individual scenario, check whether `improvementScore >= 0` (this field is already the effective score — the min of isolated and plugin when both exist). The `passed` field exists only at the verdict level (per-skill).
### Breakdown fields
@@ -112,6 +116,8 @@ Each of `baseline`, `skilledIsolated`, and `skilledPlugin` contains a `metrics`
| `assertionResults[]` | Per-assertion pass/fail with messages |
| `agentOutput` | The agent's final text output |
> **Note:** The quality scores shown in the summary table (e.g., "4.0/5") come from `baseline.judgeResult.overallScore`, `skilledIsolated.judgeResult.overallScore`, etc. — they are on the run result object, not inside `metrics`. When parsing `results.json`, look for `judgeResult.overallScore` alongside `metrics` on each run.
## Common failure patterns
### 1. Timeout with empty output
@@ -244,17 +250,29 @@ def analyze(path):
with open(path) as f:
data = json.load(f)
for verdict in data['verdicts']:
print(f"=== {verdict['skillName']} (passed={verdict['passed']}) ===")
for scenario in verdict['scenarios']:
name = scenario['scenarioName']
bl = scenario['baseline']['metrics']
sk = scenario['skilledIsolated']['metrics']
print(f"--- {name} ---")
print(f" Baseline: timedOut={bl['timedOut']}, output={len(bl.get('agentOutput',''))} chars")
print(f" Skilled: timedOut={sk['timedOut']}, output={len(sk.get('agentOutput',''))} chars")
print(f" Improvement: {scenario.get('isolatedImprovementScore', 0):.1%}")
for a in bl.get('assertionResults', []):
bl_metrics = scenario['baseline']['metrics']
sk_metrics = scenario['skilledIsolated']['metrics']
bl_quality = scenario['baseline'].get('judgeResult', {}).get('overallScore', '?')
sk_quality = scenario['skilledIsolated'].get('judgeResult', {}).get('overallScore', '?')
improvement = scenario.get('improvementScore', 0)
print(f"\n--- {name} ---")
print(f" Quality: baseline={bl_quality}/5, skilled={sk_quality}/5")
print(f" Baseline: timedOut={bl_metrics['timedOut']}, tokens={bl_metrics.get('tokenEstimate', 0)}")
print(f" Skilled: timedOut={sk_metrics['timedOut']}, tokens={sk_metrics.get('tokenEstimate', 0)}")
print(f" Improvement: {improvement:.1%}")
# perRunScores is a flat list of numbers (one per run)
per_run = scenario.get('perRunScores', [])
if per_run:
formatted = ', '.join(f'{s:.2f}' for s in per_run)
print(f" Per-run scores: [{formatted}]")
for a in sk_metrics.get('assertionResults', []):
status = 'PASS' if a['passed'] else 'FAIL'
print(f" Baseline assertion [{status}]: {a['message']}")
print(f" Assertion [{status}]: {a['message']}")
analyze('results.json')
```