mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
[Infra] Add devops agentic workflow (#141)
* Add initial plan * add token strategy * Add devops agentic workflow PoC * Fix PR Comments * Replace hand-written .yml wrappers with gh aw compile - Move workflow .md files from eng/agentic-workflows/ to .github/workflows/ - Delete hand-written .yml wrappers (devops-health-check.yml, devops-health-investigate.yml) - Generate .lock.yml files via gh aw compile - Move compiled knowledge files to .github/aw/shared/ (required by gh-aw import security) - Fix permissions: issues: write -> issues: read (strict mode uses safe-outputs) - Update README with new file structure and setup instructions Addresses PR #141 Comment 5 (ViktorHofer) * Use fuzzy daily schedule, update README for gh-aw standard structure * Recompile lock.yml with latest changes * Add skills health * Remove redundant file
This commit is contained in:
Jan Krivanek
@@ -1,2 +1,4 @@
|
||||
# Normalize all text files to LF line endings
|
||||
* text=auto eol=lf
|
||||
|
||||
.github/workflows/*.lock.yml linguist-generated=true merge=ours
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"entries": {
|
||||
"actions/github-script@v8": {
|
||||
"repo": "actions/github-script",
|
||||
"version": "v8",
|
||||
"sha": "ed597411d8f924073f98dfc5c65a23a2325f34cd"
|
||||
},
|
||||
"github/gh-aw/actions/setup@v0.45.4": {
|
||||
"repo": "github/gh-aw/actions/setup",
|
||||
"version": "v0.45.4",
|
||||
"sha": "ac090214a48a1938f7abafe132460b66752261af"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,409 @@
|
||||
<!-- AUTO-GENERATED — DO NOT EDIT -->
|
||||
<!-- Source: devops-health-check.md knowledge compilation -->
|
||||
|
||||
# DevOps Health Check — Compiled Knowledge
|
||||
|
||||
This document contains the health check catalog, fingerprinting rules, output templates, and operational guidance for the DevOps Daily Health Check agentic workflow.
|
||||
|
||||
---
|
||||
|
||||
## 1. Fingerprinting Rules
|
||||
|
||||
Every health finding MUST be assigned a deterministic **fingerprint** — a string ID derived from the finding's category and key attributes (but NOT timestamps, run IDs, or other ephemeral data). The same real-world issue MUST produce the same fingerprint on every run.
|
||||
|
||||
### 1.1 Pipeline Fingerprints
|
||||
|
||||
```
|
||||
fingerprint = "pipeline:{workflow_name}:{job_name}:{failed_step}:{conclusion}"
|
||||
```
|
||||
|
||||
- Normalize `workflow_name` by lowercasing and replacing spaces with hyphens
|
||||
- Normalize `job_name` and `failed_step` the same way
|
||||
- Same workflow + job + step + conclusion = same finding (even across different run IDs)
|
||||
- A workflow that fails in a _different_ step is a _different_ finding
|
||||
- For timeouts/cancellations: `pipeline:{workflow_name}:{job_name}:timeout`
|
||||
|
||||
**Examples:**
|
||||
| Finding | Fingerprint |
|
||||
|---------|-------------|
|
||||
| Evaluation workflow, evaluate job, "Run skill-validator" step failed | `pipeline:evaluation:evaluate:run-skill-validator:failure` |
|
||||
| Evaluation workflow, evaluate job, "Build validator" step failed | `pipeline:evaluation:evaluate:build-validator:failure` |
|
||||
| validate-skills workflow, validate job timed out | `pipeline:validate-skills:validate:timeout` |
|
||||
|
||||
### 1.2 Quality Fingerprints
|
||||
|
||||
```
|
||||
fingerprint = "quality:{skill_name}:{scenario_name}:{signal}"
|
||||
where signal ∈ { "{flag_name}", "regressed", "no-uplift", "high-variance" }
|
||||
```
|
||||
|
||||
- Extract `skill_name` and `scenario_name` from bench entry `name` field
|
||||
- Format: `"{skill}/{scenario} - {metric}"` → parse text before ` - ` and split on `/`
|
||||
- `{flag_name}` = any non-standard boolean property found on a bench entry (e.g., `notActivated`, `timedOut`, `testOverfitted`, or any future flag added to `generate-benchmark-data.ps1`)
|
||||
- Anomaly flags are **dynamically discovered**: any property beyond `name`/`unit`/`value` on a bench entry is treated as an anomaly flag
|
||||
|
||||
**Examples:**
|
||||
| Finding | Fingerprint |
|
||||
|---------|-------------|
|
||||
| dump-collect/basic-dump has notActivated flag | `quality:dump-collect:basic-dump:notActivated` |
|
||||
| csharp-scripts/basic-script quality dropped 2.3 points | `quality:csharp-scripts:basic-script:regressed` |
|
||||
| dotnet-pinvoke/marshal-array skilled ≤ vanilla | `quality:dotnet-pinvoke:marshal-array:no-uplift` |
|
||||
| analyzing-dotnet-performance/memory-leak high stddev | `quality:analyzing-dotnet-performance:memory-leak:high-variance` |
|
||||
|
||||
### 1.3 Coverage Fingerprints
|
||||
|
||||
```
|
||||
fingerprint = "coverage:{skill_name}:no-tests"
|
||||
```
|
||||
|
||||
### 1.4 Benchmark Staleness Fingerprints
|
||||
|
||||
```
|
||||
fingerprint = "quality:benchmark-stale:{component_name}"
|
||||
```
|
||||
|
||||
### 1.5 PR Fingerprints
|
||||
|
||||
```
|
||||
fingerprint = "pr:{pr_number}:{signal}"
|
||||
where signal ∈ { "stale", "no-review", "failing-checks", "stale-draft" }
|
||||
```
|
||||
|
||||
- `pr_number` is the integer PR number (not the node ID)
|
||||
- A PR that was "stale" last run and is still stale → EXISTING
|
||||
- A PR that was "stale" but got merged/closed → RESOLVED
|
||||
|
||||
### 1.6 Infrastructure Fingerprints
|
||||
|
||||
```
|
||||
fingerprint = "infra:{config_key}"
|
||||
where config_key ∈ { "no-codeowners", "no-dependabot", "relaxed-skill-validation",
|
||||
"verdict-warn-only", "pages-deployment-failed",
|
||||
"unpinned-action:{action_name}" }
|
||||
```
|
||||
|
||||
### 1.7 Resource Fingerprints
|
||||
|
||||
```
|
||||
fingerprint = "resource:{metric}:{threshold_breach}"
|
||||
```
|
||||
|
||||
- `resource:eval-duration:warning` — eval avg > 50 min
|
||||
- `resource:eval-duration:critical` — eval avg > 55 min
|
||||
- `resource:cost-increase` — weekly compute hours up >20%
|
||||
|
||||
---
|
||||
|
||||
## 2. Diff Algorithm
|
||||
|
||||
```
|
||||
previous_fps = cache_memory_load("health-check-fingerprints") ?? {}
|
||||
current_fps = {}
|
||||
|
||||
for each finding in all_collected_findings:
|
||||
fp = compute_fingerprint(finding)
|
||||
current_fps[fp] = finding
|
||||
|
||||
new_findings = { fp: f for fp, f in current_fps if fp NOT IN previous_fps }
|
||||
existing_findings = { fp: f for fp, f in current_fps if fp IN previous_fps }
|
||||
resolved_findings = { fp: f for fp, f in previous_fps if fp NOT IN current_fps }
|
||||
|
||||
# Update occurrence tracking
|
||||
for fp in existing_findings:
|
||||
existing_findings[fp].occurrences = previous_fps[fp].occurrences + 1
|
||||
existing_findings[fp].first_seen = previous_fps[fp].first_seen
|
||||
|
||||
for fp in new_findings:
|
||||
new_findings[fp].occurrences = 1
|
||||
new_findings[fp].first_seen = today
|
||||
|
||||
cache_memory_save("health-check-fingerprints", current_fps)
|
||||
cache_memory_save("health-check-history", append(
|
||||
load("health-check-history"),
|
||||
{ date: today, new_count, existing_count, resolved_count, by_severity }
|
||||
))
|
||||
```
|
||||
|
||||
### 2.1 Sorting Within Diff Categories
|
||||
|
||||
Within each category (NEW, EXISTING, RESOLVED):
|
||||
1. **Primary**: Severity descending — 🔴 Critical → 🟡 Warning → 🔵 Info
|
||||
2. **Secondary**: Category — pipeline → quality → pr → infra → resource
|
||||
3. **Tertiary**: Alphabetical by title
|
||||
|
||||
---
|
||||
|
||||
## 3. Severity Rules Reference
|
||||
|
||||
### Pipeline
|
||||
|
||||
| Check | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| P1 | `evaluation` workflow failed | 🔴 Critical |
|
||||
| P1 | Other workflow failed | 🟡 Warning |
|
||||
| P1 | Matches `known-noise` pattern | 🔵 Info (demoted) |
|
||||
| P2 | Any cancelled/timed-out run | 🟡 Warning |
|
||||
| P3 | Eval avg duration > 55 min | 🔴 Critical |
|
||||
| P3 | Eval avg duration > 50 min | 🟡 Warning |
|
||||
|
||||
### Quality
|
||||
|
||||
| Check | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| Q1 | `notActivated` flag on bench entry | 🔴 Critical |
|
||||
| Q1 | Any other anomaly flag | 🟡 Warning |
|
||||
| Q2 | Quality drop > 2.0 points vs 7-day avg | 🔴 Critical |
|
||||
| Q2 | Quality drop > 1.0 points vs 7-day avg | 🟡 Warning |
|
||||
| Q3 | Skilled ≤ Vanilla quality | 🟡 Warning |
|
||||
| Q4 | Quality stddev > 1.5 over 7 days | 🟡 Warning |
|
||||
| Q5 | Skill has no eval tests | 🟡 Warning |
|
||||
| Q6 | Benchmark data > 24h old | 🟡 Warning |
|
||||
|
||||
### PR
|
||||
|
||||
| Check | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| R1 | Open > 7 days, no review | 🟡 Warning |
|
||||
| R2 | Open > 14 days (any review) | 🟡 Warning |
|
||||
| R3 | All checks failing | 🟡 Warning |
|
||||
| R4 | Draft, no activity > 7 days | 🔵 Info |
|
||||
|
||||
### Infrastructure
|
||||
|
||||
| Check | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| I1 | No CODEOWNERS file | 🟡 Warning |
|
||||
| I2 | No Dependabot config | 🟡 Warning |
|
||||
| I3 | `fail-on-warning: false` in validate-skills | 🟡 Warning |
|
||||
| I4 | `--verdict-warn-only` in evaluation | 🔵 Info |
|
||||
| I5 | Pages deployment failed | 🔴 Critical |
|
||||
| I6 | Unpinned third-party action | 🔵 Info |
|
||||
|
||||
### Resource
|
||||
|
||||
| Check | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| U3 | Weekly compute up >20% | 🟡 Warning |
|
||||
|
||||
---
|
||||
|
||||
## 4. Benchmark Data Format
|
||||
|
||||
Dashboard data files on `gh-pages` at `data/{component}.json` have this structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"lastUpdate": 1740000000000,
|
||||
"repoUrl": "",
|
||||
"entries": {
|
||||
"Quality": [
|
||||
{
|
||||
"commit": { "id": "abc1234", "message": "...", "timestamp": "..." },
|
||||
"date": 1740000000000,
|
||||
"tool": "customBiggerIsBetter",
|
||||
"model": "claude-opus-4.6",
|
||||
"benches": [
|
||||
{
|
||||
"name": "skillName/scenarioName - Skilled Quality",
|
||||
"unit": "Score (0-10)",
|
||||
"value": 7.8
|
||||
},
|
||||
{
|
||||
"name": "skillName/scenarioName - Vanilla Quality",
|
||||
"unit": "Score (0-10)",
|
||||
"value": 5.2
|
||||
},
|
||||
{
|
||||
"name": "skillName/scenarioName - Skilled Quality",
|
||||
"unit": "Score (0-10)",
|
||||
"value": 0.0,
|
||||
"notActivated": true
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"Efficiency": [
|
||||
{
|
||||
"commit": { ... },
|
||||
"date": 1740000000000,
|
||||
"tool": "customSmallerIsBetter",
|
||||
"model": "claude-opus-4.6",
|
||||
"benches": [
|
||||
{
|
||||
"name": "skillName/scenarioName - Skilled Time",
|
||||
"unit": "seconds",
|
||||
"value": 45.2,
|
||||
"timedOut": true
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Key Points for Parsing:
|
||||
- `date` is Unix epoch in **milliseconds**
|
||||
- Bench entry `name` format: `"{skill}/{scenario} - {metric}"`
|
||||
- Standard fields: `name`, `unit`, `value` — anything else is an anomaly flag
|
||||
- Both `Quality` and `Efficiency` arrays carry the same anomaly flags
|
||||
- Quality scores range 0-10 (mapped from 0-5 judge scale)
|
||||
- The latest entry is the last element in the array (`entries.Quality[-1]`)
|
||||
- For 7-day rolling averages, filter entries by `date` field (not array index)
|
||||
|
||||
### Parsing Skill/Scenario from Bench Name:
|
||||
|
||||
```
|
||||
bench.name = "dump-collect/basic-dump - Skilled Quality"
|
||||
parts = bench.name.split(" - ")
|
||||
# parts[0] = "dump-collect/basic-dump"
|
||||
# parts[1] = "Skilled Quality"
|
||||
|
||||
skill_scenario = parts[0].split("/")
|
||||
# skill_scenario[0] = "dump-collect" (skill name)
|
||||
# skill_scenario[1] = "basic-dump" (scenario name)
|
||||
```
|
||||
|
||||
### Detecting Anomaly Flags:
|
||||
|
||||
For each bench entry, check all properties. Any property key that is NOT `name`, `unit`, or `value` is an anomaly flag:
|
||||
|
||||
```
|
||||
standard_fields = {"name", "unit", "value"}
|
||||
flags = { key: value for key, value in bench_entry if key not in standard_fields and value == true }
|
||||
```
|
||||
|
||||
This approach automatically discovers new flag types added in the future.
|
||||
|
||||
---
|
||||
|
||||
## 5. Component Discovery
|
||||
|
||||
Components are discovered by scanning the file system:
|
||||
|
||||
```bash
|
||||
find src/*/plugin.json -maxdepth 2
|
||||
```
|
||||
|
||||
Each `src/{name}/` directory containing a `plugin.json` is a component. The dashboard data file is at `data/{name}.json` on the `gh-pages` branch.
|
||||
|
||||
To fetch benchmark data via the GitHub API (without `curl`):
|
||||
```
|
||||
GET https://raw.githubusercontent.com/{owner}/{repo}/gh-pages/data/{component}.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Known Noise Patterns
|
||||
|
||||
The `cache-memory` key `known-noise` stores a list of fingerprint prefixes or patterns that should be demoted to 🔵 Info severity. Example patterns:
|
||||
|
||||
- `pipeline:copilot-code-review` — org-level workflow with known chronic failures
|
||||
- `infra:verdict-warn-only` — intentional configuration, always Info
|
||||
|
||||
When a finding's fingerprint matches any known-noise pattern (prefix match), demote its severity to 🔵 Info. The finding is still reported in the output (in the EXISTING section if recurring) — it is NOT hidden.
|
||||
|
||||
New patterns can be added via the `/health-check suppress <pattern>` slash command, which appends to the `known-noise` list in `cache-memory`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Investigation Dispatch Rules
|
||||
|
||||
Only 🆕 NEW findings that meet these criteria qualify for investigation dispatch:
|
||||
|
||||
| Condition | Action |
|
||||
|-----------|--------|
|
||||
| 🆕 + 🔴 Critical | **Always dispatch** |
|
||||
| 🆕 + 🟡 Warning + `pipeline` or `quality` category | **Dispatch** |
|
||||
| 🆕 + 🟡 Warning + `pr` or `infra` category | **Skip** |
|
||||
| 🆕 + 🔵 Info | **Never dispatch** |
|
||||
| 📌 EXISTING or ✅ RESOLVED | **Never dispatch** |
|
||||
|
||||
**Budget cap:** Maximum 10 dispatches per run.
|
||||
**Priority order when cap is hit:**
|
||||
1. 🔴 Critical findings first
|
||||
2. Pipeline findings before quality
|
||||
3. Other categories last
|
||||
|
||||
---
|
||||
|
||||
## 8. Output Templates
|
||||
|
||||
### 8.1 Issue Title
|
||||
|
||||
```
|
||||
🏥 Repository Health Dashboard
|
||||
```
|
||||
|
||||
### 8.2 Issue Label
|
||||
|
||||
```
|
||||
devops-health
|
||||
```
|
||||
- Color: `#0E8A16`
|
||||
- Description: `Daily automated health check report`
|
||||
|
||||
### 8.3 First Run Notice
|
||||
|
||||
If no previous fingerprints exist in `cache-memory`:
|
||||
|
||||
```markdown
|
||||
> ⚠️ This is the first health check run. All findings appear as new.
|
||||
> Starting from the next run, only changes will be highlighted.
|
||||
```
|
||||
|
||||
### 8.4 Trends Arrow Legend
|
||||
|
||||
| Condition | Arrow | Meaning |
|
||||
|-----------|-------|---------|
|
||||
| Δ positive and good (e.g., success rate up) | ✅ | Improving |
|
||||
| Δ positive and bad (e.g., compute hours up) | ↗️ | Increasing (watch) |
|
||||
| Δ negative and good (e.g., open PRs down) | ✅ | Improving |
|
||||
| Δ negative and bad (e.g., success rate down) | ⚠️ | Degrading |
|
||||
| Δ ≈ 0 | ➡️ | Stable |
|
||||
|
||||
### 8.5 Investigation Island Template
|
||||
|
||||
```markdown
|
||||
<!-- investigation:{fingerprint} -->
|
||||
⏳ Investigation dispatched — results arriving shortly...
|
||||
<!-- /investigation:{fingerprint} -->
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Operational Guardrails
|
||||
|
||||
### 9.1 API Rate Limits
|
||||
- Use targeted, date-filtered queries to minimize API calls
|
||||
- The `github` MCP toolset handles pagination automatically
|
||||
- Space dispatches 5 seconds apart
|
||||
|
||||
### 9.2 Issue Body Size
|
||||
- GitHub issues have a ~65,535 character limit
|
||||
- If body exceeds 60k: truncate EXISTING section (keep top 20 by severity)
|
||||
- Footer: `> … N additional existing findings omitted`
|
||||
- The daily comment always includes complete summary counts
|
||||
|
||||
### 9.3 Cache Memory Keys
|
||||
|
||||
| Key | Contents | Updated |
|
||||
|-----|----------|---------|
|
||||
| `health-check-fingerprints` | Map of fingerprint → finding (with occurrences, first_seen) | Every run |
|
||||
| `health-check-history` | Array of daily summaries (date, counts by diff type and severity) | Appended each run |
|
||||
| `known-noise` | Array of fingerprint patterns to demote to Info | Via `/health-check suppress` |
|
||||
|
||||
### 9.4 Graceful Degradation
|
||||
|
||||
If any data source is unavailable:
|
||||
- Skip that check category entirely
|
||||
- Note the skip in the output: `> ⚠️ Skipped {category} checks: {reason}`
|
||||
- Do NOT fail the entire workflow
|
||||
- Continue with available data
|
||||
|
||||
### 9.5 Cache Memory Loss
|
||||
|
||||
If `cache-memory` returns no previous state:
|
||||
- Treat all findings as 🆕 NEW
|
||||
- Display the first-run notice (§8.3)
|
||||
- The diff will resume automatically on the next run
|
||||
@@ -0,0 +1,309 @@
|
||||
<!-- AUTO-GENERATED — DO NOT EDIT -->
|
||||
<!-- Source: devops-health-investigate.md knowledge compilation -->
|
||||
|
||||
# DevOps Investigation — Compiled Knowledge
|
||||
|
||||
This document contains category-specific investigation playbooks, root-cause patterns, and remediation templates for the DevOps Health Investigation worker agent.
|
||||
|
||||
---
|
||||
|
||||
## 1. Pipeline Investigation Playbook
|
||||
|
||||
When `finding_type == "pipeline"`:
|
||||
|
||||
### Step-by-Step Protocol
|
||||
|
||||
1. **Fetch the failed run** using `resource_url`:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/runs/{run_id}
|
||||
```
|
||||
Extract: workflow name, conclusion, created_at, updated_at, triggering_actor, head_sha.
|
||||
|
||||
2. **Identify the failed job and step**:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs
|
||||
```
|
||||
Find the job with `conclusion: failure`. Within that job, find the step with `conclusion: failure`.
|
||||
|
||||
3. **Read the failed step's logs** (last 200 lines):
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/jobs/{job_id}/logs
|
||||
```
|
||||
Extract error messages, exception types, exit codes. Look for patterns:
|
||||
- `.NET SDK version mismatch` → check `global.json`
|
||||
- `process exited with code 1` → build/test failure
|
||||
- `Error: ENOMEM` or `killed` → resource exhaustion
|
||||
- `rate limit` or `403` → API throttling
|
||||
- `timeout` → long-running operation exceeded limit
|
||||
|
||||
4. **Fetch the last 5 successful runs** of the same workflow:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/workflows/{workflow_id}/runs?branch=main&status=success&per_page=5
|
||||
```
|
||||
|
||||
5. **Compare: what changed between last success and this failure?**
|
||||
- Get the `head_sha` of the last successful run
|
||||
- Get the `head_sha` of the failed run
|
||||
- Compare commits between them:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/compare/{success_sha}...{failure_sha}
|
||||
```
|
||||
- Look for changes to: workflow YAML files, build scripts, `global.json`, dependency files, the code being tested.
|
||||
|
||||
6. **Check if the failure is in repo code or a GitHub Action version update**:
|
||||
- Compare action versions between the failing and last successful workflow YAML
|
||||
- Check if a new action version was released recently
|
||||
|
||||
7. **Determine root cause** with confidence level:
|
||||
- **High**: Error message explicitly identifies the cause (e.g., "SDK version 9.0.200 not found")
|
||||
- **Medium**: Timing strongly correlates with a specific commit
|
||||
- **Low**: No clear evidence — multiple possibilities
|
||||
|
||||
8. **Generate 1–3 specific remediation steps**:
|
||||
- Include exact file paths, version numbers, or commands
|
||||
- Order by recommended priority
|
||||
|
||||
9. **Check for existing tracking**:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/issues?state=open&labels=bug
|
||||
```
|
||||
Search for issues mentioning the same workflow or error.
|
||||
|
||||
### Common Pipeline Root Causes
|
||||
|
||||
| Pattern | Typical Cause | Remediation |
|
||||
|---------|---------------|-------------|
|
||||
| `SDK version not found` | `global.json` pins a version not installed on the runner | Update `global.json` or add `setup-dotnet` step |
|
||||
| `process exited with code 1` in test step | Test failure (assertion or runtime error) | Check test output for specific failure |
|
||||
| `Error: HttpError: rate limit exceeded` | GitHub API rate limiting | Add retry logic or reduce API calls |
|
||||
| `The operation was canceled` | Timeout (default 360 min for Actions) | Optimize the step or increase timeout |
|
||||
| `No space left on device` | Runner disk full (14 GB limit) | Add cleanup steps or reduce artifact size |
|
||||
| Action `X` failed with `Node.js 16 actions are deprecated` | Action needs version update | Update action to latest version |
|
||||
|
||||
---
|
||||
|
||||
## 2. Quality Investigation Playbook
|
||||
|
||||
When `finding_type == "quality"`:
|
||||
|
||||
### Step-by-Step Protocol
|
||||
|
||||
1. **Fetch benchmark data** for the affected component (last 14+ entries):
|
||||
```
|
||||
GET https://raw.githubusercontent.com/{owner}/{repo}/gh-pages/data/{component}.json
|
||||
```
|
||||
|
||||
2. **Analyze the trend**:
|
||||
- Extract the time series for the affected scenario's "Skilled Quality" bench
|
||||
- Is this a **sudden drop** (step function) or **gradual degradation** (slope)?
|
||||
- Sudden drops (>2 points between consecutive entries) typically indicate a specific triggering change
|
||||
- Gradual degradation may indicate model drift or accumulating prompt issues
|
||||
|
||||
3. **For anomaly flags** (`notActivated`, `timedOut`, `testOverfitted`, etc.):
|
||||
a. Identify which flag(s) are set on the bench entry
|
||||
b. For `notActivated`:
|
||||
- The skill failed to activate — the model didn't use any skill-provided context
|
||||
- Check the skill definition file (`src/{component}/skills/{skill}/SKILL.md`) for syntax or trigger issues
|
||||
- Check if the skill's `description` field changed recently
|
||||
- Check if the test prompt still aligns with the skill's trigger keywords
|
||||
c. For `timedOut`:
|
||||
- The evaluation exceeded the time limit
|
||||
- Check recent complexity changes in the skill or test
|
||||
- Check if the timeout threshold was changed
|
||||
d. For other/unknown flags:
|
||||
- Describe the flag name and its value
|
||||
- Check `eng/dashboard/generate-benchmark-data.ps1` for context on when this flag is set
|
||||
- Report the flag as-is with whatever context is available
|
||||
|
||||
4. **For regression** (quality score dropped):
|
||||
- Identify the exact entry where quality dropped (compare consecutive entries)
|
||||
- Get the commit SHA from the regression entry's `commit` field
|
||||
- Check what changed in that commit:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/commits/{sha}
|
||||
```
|
||||
- Look for changes to: skill definition, test definition, shared knowledge files, prompt templates
|
||||
|
||||
5. **For high variance**:
|
||||
- Compute the range (max - min) across recent entries
|
||||
- Check if variance is skill-specific or affects multiple skills (model instability)
|
||||
- Check if test prompts are ambiguous (allowing valid but different approaches)
|
||||
|
||||
6. **For no-uplift** (skilled ≤ vanilla):
|
||||
- Compare the skill's prompt additions to the vanilla baseline
|
||||
- Check if the skill knowledge is relevant to the test scenario
|
||||
- Check if the skill is triggering correctly (not `notActivated`)
|
||||
|
||||
7. **Check recent skill/test changes**:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/commits?path=src/{component}/skills/{skill}&per_page=5
|
||||
GET /repos/{owner}/{repo}/commits?path=src/{component}/tests/{skill}&per_page=5
|
||||
```
|
||||
|
||||
8. **Determine root cause** with confidence level.
|
||||
|
||||
9. **Generate remediation steps** specific to skill quality:
|
||||
- For activation issues: suggest trigger keyword adjustments
|
||||
- For regression: identify the specific change and suggest reverting or fixing
|
||||
- For variance: suggest test prompt clarification or eval parameter tuning
|
||||
|
||||
### Common Quality Root Causes
|
||||
|
||||
| Pattern | Typical Cause | Remediation |
|
||||
|---------|---------------|-------------|
|
||||
| `notActivated` on all scenarios of a skill | Skill description doesn't match test prompts | Review skill trigger keywords vs test prompts |
|
||||
| `notActivated` on one scenario only | Test prompt is too different from skill scope | Adjust test prompt or broaden skill description |
|
||||
| Sudden quality drop after commit X | Skill definition or knowledge was changed | Review diff of commit X; consider partial revert |
|
||||
| Gradual quality decline over 7+ days | Model behavior drift or prompt degradation | Re-evaluate skill knowledge for staleness |
|
||||
| Skilled ≤ Vanilla consistently | Skill knowledge may confuse rather than help | Review skill content for misleading information |
|
||||
| `timedOut` on complex scenarios | Scenario requires too many tool calls or reasoning steps | Simplify scenario or increase timeout |
|
||||
| High variance (stddev > 2.0) | Ambiguous test prompt allows multiple valid approaches | Tighten test prompt expectations |
|
||||
|
||||
---
|
||||
|
||||
## 3. PR Investigation Playbook
|
||||
|
||||
When `finding_type == "pr"`:
|
||||
|
||||
### Step-by-Step Protocol
|
||||
|
||||
1. **Fetch PR details**:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/pulls/{pr_number}
|
||||
```
|
||||
Extract: title, author, created_at, updated_at, draft status, labels, body summary.
|
||||
|
||||
2. **Fetch PR timeline** (reviews, comments, status changes):
|
||||
```
|
||||
GET /repos/{owner}/{repo}/pulls/{pr_number}/reviews
|
||||
GET /repos/{owner}/{repo}/issues/{pr_number}/comments
|
||||
```
|
||||
|
||||
3. **For no-review PRs**:
|
||||
- Check if CODEOWNERS would auto-assign reviewers
|
||||
- Check `git blame` or recent contributors to the changed files for potential reviewers
|
||||
- Look at the PR size (files changed, lines) — large PRs may discourage review
|
||||
|
||||
4. **For stale PRs**:
|
||||
- Check last activity date (any comment or push)
|
||||
- Check if the author is still active (recent commits/PRs)
|
||||
- Check if there are related issues that are still relevant
|
||||
|
||||
5. **For failing checks**:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/commits/{head_sha}/check-runs
|
||||
```
|
||||
- Identify which checks fail
|
||||
- Cross-reference with known pipeline issues from the same health check
|
||||
- Determine if check failures are PR-specific or repo-wide
|
||||
|
||||
6. **Provide actionable summary**:
|
||||
- What's blocking the PR
|
||||
- Who should review
|
||||
- Whether the PR is still relevant
|
||||
|
||||
---
|
||||
|
||||
## 4. Infrastructure Investigation Playbook
|
||||
|
||||
When `finding_type == "infra"`:
|
||||
|
||||
### Step-by-Step Protocol
|
||||
|
||||
1. **Audit the configuration**:
|
||||
- For missing files (CODEOWNERS, dependabot): confirm absence and explain impact
|
||||
- For relaxed settings: read the config file and explain what the setting does
|
||||
|
||||
2. **Check if intentional**:
|
||||
- Search for issues or PRs that discuss the configuration choice
|
||||
- Check commit history of the config file for context
|
||||
|
||||
3. **Compare with best practices**:
|
||||
- Reference GitHub's recommended security settings
|
||||
- Note any compliance or security implications
|
||||
|
||||
4. **For Pages deployment failures**:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/pages/builds
|
||||
```
|
||||
- Read the latest build log
|
||||
- Identify the failure cause (build error, quota, DNS, etc.)
|
||||
|
||||
---
|
||||
|
||||
## 5. Resource Investigation Playbook
|
||||
|
||||
When `finding_type == "resource"`:
|
||||
|
||||
### Step-by-Step Protocol
|
||||
|
||||
1. **Gather usage data**:
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/runs?per_page=100
|
||||
```
|
||||
- Compute daily/weekly compute hours by summing run durations
|
||||
- Break down by workflow
|
||||
|
||||
2. **Identify cost drivers**:
|
||||
- Which workflows consume the most time?
|
||||
- Has a new workflow been added recently?
|
||||
- Did an existing workflow's duration increase?
|
||||
|
||||
3. **For eval duration warnings**:
|
||||
- Check if the number of skills/scenarios being evaluated increased
|
||||
- Check if individual scenario duration increased
|
||||
- Look for parallelism changes in the workflow configuration
|
||||
|
||||
4. **Provide optimization suggestions**:
|
||||
- Can any workflows be consolidated?
|
||||
- Are there unnecessary re-runs (e.g., missing `concurrency` groups)?
|
||||
- Can caching reduce execution time?
|
||||
|
||||
---
|
||||
|
||||
## 6. Report Format
|
||||
|
||||
All investigation results follow this template:
|
||||
|
||||
```markdown
|
||||
🔍 **Investigation Complete** — [Worker Run #{run_number}]({run_url})
|
||||
|
||||
**Root cause:** {Clear, evidence-based description of what went wrong and why.
|
||||
Include specific error messages, commit SHAs, or file paths as evidence.}
|
||||
|
||||
**Confidence:** {High|Medium|Low} — {One sentence justifying the confidence level}
|
||||
|
||||
**Blast radius:** {What else is affected by this issue. Be specific about which
|
||||
components, workflows, or metrics are impacted.}
|
||||
|
||||
**Suggested fix:**
|
||||
1. {Most recommended action — include specific file, line, or command}
|
||||
2. {Alternative action if applicable}
|
||||
3. {Additional step if needed}
|
||||
|
||||
**Related:** {List related commits (with SHA + author), PRs (with #number), or
|
||||
issues (with #number). Say "None found" if nothing is related.}
|
||||
```
|
||||
|
||||
### Confidence Level Guidelines
|
||||
|
||||
| Level | Criteria | Example |
|
||||
|-------|----------|---------|
|
||||
| **High** | Direct evidence links cause to effect | Error log says "SDK 9.0.200 not found"; commit changed SDK version |
|
||||
| **Medium** | Strong circumstantial correlation | Quality dropped on the same day a skill file was modified |
|
||||
| **Low** | Possible but speculative | Multiple recent changes could explain the issue; no clear winner |
|
||||
|
||||
---
|
||||
|
||||
## 7. Common Cross-Category Patterns
|
||||
|
||||
These patterns span multiple check categories and may help identify systemic issues:
|
||||
|
||||
| Pattern | Indicates |
|
||||
|---------|-----------|
|
||||
| Pipeline failure + stale benchmark data | Pipeline is blocking data publication |
|
||||
| Multiple quality regressions on same date | Common causal commit |
|
||||
| PR failing checks + same check failing on main | Repo-wide issue, not PR-specific |
|
||||
| `notActivated` + recent skill definition change | Skill trigger broke |
|
||||
| Eval duration spike + new skill/scenario added | Expected growth, not a bug |
|
||||
| Cost increase + new scheduled workflow | Expected growth, not waste |
|
||||
+1413
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,461 @@
|
||||
---
|
||||
name: "DevOps Daily Health Check"
|
||||
description: >
|
||||
Orchestrator workflow that collects repo health signals daily (pipelines,
|
||||
skill quality, PRs, infrastructure), computes a fingerprint-based diff
|
||||
against the previous run, updates a pinned health dashboard issue, and
|
||||
dispatches investigation workers for new critical/warning findings.
|
||||
|
||||
on:
|
||||
schedule: daily
|
||||
slash_command: health-check
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
issues: read
|
||||
pull-requests: read
|
||||
|
||||
imports:
|
||||
- ../aw/shared/devops-health.lock.md
|
||||
|
||||
tools:
|
||||
github:
|
||||
toolsets: [repos, issues, pull_requests, actions]
|
||||
cache-memory:
|
||||
bash: ["cat", "grep", "head", "tail", "find", "ls", "wc", "jq", "date", "sort", "uniq", "diff"]
|
||||
edit:
|
||||
|
||||
safe-outputs:
|
||||
create-issue:
|
||||
max: 1
|
||||
update-issue:
|
||||
max: 1
|
||||
add-comment:
|
||||
max: 1
|
||||
dispatch-workflow:
|
||||
workflows:
|
||||
- devops-health-investigate
|
||||
max: 10
|
||||
|
||||
network:
|
||||
allowed:
|
||||
- defaults
|
||||
---
|
||||
|
||||
# DevOps Daily Health Check — Orchestrator
|
||||
|
||||
You are a DevOps health monitoring agent. Your job is to collect repo health signals, compute a diff against the previous run, and produce a comprehensive yet actionable health dashboard.
|
||||
|
||||
## High-Level Workflow
|
||||
|
||||
1. **Data Collection** (deterministic — use API calls and bash tools)
|
||||
2. **Fingerprint & Diff** (compare against previous run via `cache-memory`)
|
||||
3. **Analysis** (LLM-powered: correlate findings, identify root causes, write summary)
|
||||
4. **Output** (update pinned issue + post daily comment)
|
||||
5. **Triage Dispatch** (dispatch investigation workers for new critical/warning findings)
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Data Collection
|
||||
|
||||
### 1.1 Discover Components
|
||||
|
||||
Scan the repository to find all skill components:
|
||||
|
||||
```
|
||||
find src/*/plugin.json -maxdepth 2
|
||||
```
|
||||
|
||||
Each `src/{name}/` directory containing a `plugin.json` is a component. The corresponding dashboard data file is `data/{name}.json` on `gh-pages`.
|
||||
|
||||
### 1.2 Pipeline Health (P1–P4)
|
||||
|
||||
**P1 — Failed workflow runs on `main` in last 24h:**
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/runs?branch=main&status=failure&per_page=30
|
||||
```
|
||||
Filter to runs created within the last 24 hours. For each failed run:
|
||||
- Extract `workflow_name`, `conclusion`, `job_name`, `failed_step`
|
||||
- Fingerprint: `pipeline:{workflow_name}:{job_name}:{failed_step}:{conclusion}`
|
||||
- Severity: 🔴 Critical if `evaluation` workflow fails; 🟡 Warning for others
|
||||
- **Noise suppression:** Check if the finding matches any pattern in the `known-noise` list from `cache-memory`. If it matches, demote severity to 🔵 Info.
|
||||
|
||||
**P2 — Cancelled/timed-out runs in last 24h:**
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/runs?branch=main&status=cancelled&per_page=10
|
||||
```
|
||||
- Fingerprint: `pipeline:{workflow_name}:{job_name}:timeout`
|
||||
- Severity: 🟡 Warning
|
||||
|
||||
**P3 — Evaluation duration trend:**
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/workflows/evaluation.yml/runs?branch=main&per_page=30
|
||||
```
|
||||
Compute average run duration over the last 14 days.
|
||||
- 🟡 Warning if avg > 50 min (83% of 60-min timeout)
|
||||
- 🔴 Critical if avg > 55 min
|
||||
- Fingerprint: `resource:eval-duration:{bucket}` (bucket = "warning" or "critical")
|
||||
|
||||
**P4 — Workflow failure rate (7-day rolling):**
|
||||
```
|
||||
GET /repos/{owner}/{repo}/actions/runs?branch=main&per_page=100
|
||||
```
|
||||
Group by workflow name, compute success/failure ratio over the last 7 days.
|
||||
- 🔵 Info (metric only — reported in trends table, not fingerprinted)
|
||||
|
||||
### 1.3 Skill Quality (Q1–Q7)
|
||||
|
||||
Fetch benchmark data for each discovered component:
|
||||
```
|
||||
GET https://raw.githubusercontent.com/{owner}/{repo}/gh-pages/data/{component}.json
|
||||
```
|
||||
|
||||
**Q1 — Skill inventory overview table:**
|
||||
Compile a comprehensive table of all skills combining local discovery with benchmark data. For each skill, classify its health status:
|
||||
- **🟢 OK** — Skill has tests, scenarios pass, skilled > vanilla, no anomaly flags
|
||||
- **🟡 Warning** — Skill is functional but has issues: timeouts, overfitting, or high variance (stddev > 1.5)
|
||||
- **🟡 Low Value** — Some scenarios show skilled ≤ vanilla (but others show uplift)
|
||||
- **🔴 No Value** — All scenarios show skilled ≤ vanilla (skill adds nothing)
|
||||
- **🔴 Critical** — Skill not activated by the agent (`notActivated` flag)
|
||||
- **⚪ Untested** — No test directory, no eval.yaml, or eval.yaml has 0 scenarios
|
||||
- **⚪ No Data** — Skill exists locally but has no benchmark data
|
||||
|
||||
This table is informational (🔵 Info) and not fingerprinted. It is rendered in the issue body as a dedicated "Skill Inventory" section.
|
||||
|
||||
For each skill, compute:
|
||||
- **Avg Skilled score**: average of all scenario "Skilled Quality" bench values in the latest entry
|
||||
- **Avg Vanilla score**: average of all scenario "Vanilla Quality" bench values in the latest entry
|
||||
- **Delta**: Skilled − Vanilla
|
||||
- **Scenario count**: number of scenarios with benchmark data
|
||||
- **Issue summary**: comma-separated list of issues (timeout, overfitting, no-uplift, high-variance, etc.)
|
||||
|
||||
**Q2 — Bench entries with anomaly flags:**
|
||||
Scan the latest entry in **both** `entries.Quality` and `entries.Efficiency` arrays. For each bench entry, check for any property beyond the standard `name`/`unit`/`value` fields. Any extra boolean property is an anomaly flag (e.g., `notActivated`, `timedOut`, `testOverfitted`, or future flags).
|
||||
- Extract the skill name and scenario from the bench `name` field (format: `"{skill}/{scenario} - {metric}"`)
|
||||
- 🔴 Critical if `notActivated` (skill broken)
|
||||
- 🟡 Warning for all other flags
|
||||
- Fingerprint: `quality:{skill}:{scenario}:{flag-name}`
|
||||
- **Deduplicate:** If the same skill/scenario/flag appears in both Quality and Efficiency arrays, report it only once.
|
||||
|
||||
**Q3 — Quality regression (>1.0 point drop vs 7-day rolling avg):**
|
||||
For each scenario's "Skilled Quality" bench, compare the latest value to the rolling average of all entries from the last 7 calendar days (filter by `date` field).
|
||||
- 🔴 Critical if drop > 2.0 points
|
||||
- 🟡 Warning if drop > 1.0 points
|
||||
- Fingerprint: `quality:{skill}:{scenario}:regressed`
|
||||
|
||||
**Q4 — Skilled ≤ Vanilla (skill adds no value):**
|
||||
For the latest entry, compare `"{skill}/{scenario} - Skilled Quality"` vs `"{skill}/{scenario} - Vanilla Quality"` bench values.
|
||||
- 🟡 Warning if Skilled ≤ Vanilla
|
||||
- Fingerprint: `quality:{skill}:{scenario}:no-uplift`
|
||||
|
||||
**Q5 — High variance across runs:**
|
||||
Compute the standard deviation of `"Skilled Quality"` scores across all entries from the last 7 calendar days.
|
||||
- 🟡 Warning if stddev > 1.5
|
||||
- Fingerprint: `quality:{skill}:{scenario}:high-variance`
|
||||
|
||||
**Q6 — Skills without eval tests:**
|
||||
```
|
||||
find src/*/skills/ -mindepth 1 -maxdepth 1 -type d
|
||||
```
|
||||
For each skill directory, check if a corresponding test directory exists under `src/{component}/tests/{skill-name}/`.
|
||||
If the test directory exists, verify that `eval.yaml` exists and contains at least one scenario.
|
||||
- 🟡 Warning if no test directory, no eval.yaml, or eval.yaml has no scenarios
|
||||
- Fingerprint: `coverage:{skill}:no-tests`
|
||||
|
||||
**Q7 — Benchmark data staleness:**
|
||||
Check if the latest entry's `date` timestamp is > 24h old (compare to current time).
|
||||
- 🟡 Warning (pipeline may not be publishing)
|
||||
- Fingerprint: `quality:benchmark-stale:{component}`
|
||||
|
||||
### 1.4 PR & Review Health (R1–R5)
|
||||
|
||||
```
|
||||
GET /repos/{owner}/{repo}/pulls?state=open&sort=created&direction=asc&per_page=50
|
||||
```
|
||||
|
||||
**R1 — PRs open > 7 days without review:**
|
||||
Filter by `created_at` older than 7 days, then check review count (0 reviews).
|
||||
- 🟡 Warning
|
||||
- Fingerprint: `pr:{pr_number}:no-review`
|
||||
|
||||
**R2 — PRs open > 14 days (any state of review):**
|
||||
- 🟡 Warning (possibly abandoned)
|
||||
- Fingerprint: `pr:{pr_number}:stale`
|
||||
|
||||
**R3 — PRs with all checks failing:**
|
||||
For each open PR, check its check runs. If all checks are failing:
|
||||
- 🟡 Warning
|
||||
- Fingerprint: `pr:{pr_number}:failing-checks`
|
||||
|
||||
**R4 — Draft PRs with no activity > 7 days:**
|
||||
Filter for `draft=true` and `updated_at` older than 7 days.
|
||||
- 🔵 Info
|
||||
- Fingerprint: `pr:{pr_number}:stale-draft`
|
||||
|
||||
**R5 — PR merge velocity trend:**
|
||||
```
|
||||
GET /repos/{owner}/{repo}/pulls?state=closed&sort=updated&direction=desc&per_page=50
|
||||
```
|
||||
Count merged PRs per day over the last 7 days.
|
||||
- 🔵 Info (metric only — reported in trends table, not fingerprinted)
|
||||
|
||||
### 1.5 Infrastructure Checks (I1–I6)
|
||||
|
||||
**I1 — Missing CODEOWNERS:**
|
||||
```
|
||||
GET /repos/{owner}/{repo}/contents/CODEOWNERS
|
||||
```
|
||||
If 404, also check `.github/CODEOWNERS` and `docs/CODEOWNERS`.
|
||||
- 🟡 Warning if none found
|
||||
- Fingerprint: `infra:no-codeowners`
|
||||
|
||||
**I2 — Missing Dependabot config:**
|
||||
```
|
||||
GET /repos/{owner}/{repo}/contents/.github/dependabot.yml
|
||||
```
|
||||
- 🟡 Warning if 404
|
||||
- Fingerprint: `infra:no-dependabot`
|
||||
|
||||
**I3 — Relaxed skill validation:**
|
||||
Check if `.github/workflows/validate-skills.yml` contains `fail-on-warning: false`.
|
||||
- 🟡 Warning
|
||||
- Fingerprint: `infra:relaxed-skill-validation`
|
||||
|
||||
**I4 — Verdict-warn-only mode:**
|
||||
Check if `.github/workflows/evaluation.yml` contains `--verdict-warn-only`.
|
||||
- 🔵 Info
|
||||
- Fingerprint: `infra:verdict-warn-only`
|
||||
|
||||
**I5 — Dashboard deployment health:**
|
||||
```
|
||||
GET /repos/{owner}/{repo}/pages
|
||||
```
|
||||
Check last deployment status.
|
||||
- 🔴 Critical if deployment failed
|
||||
- Fingerprint: `infra:pages-deployment-failed`
|
||||
|
||||
**I6 — Third-party action version drift:**
|
||||
Scan workflow YAML files for non-`actions/*` references. Flag those pinned to tags instead of SHAs.
|
||||
- 🔵 Info
|
||||
- Fingerprint: `infra:unpinned-action:{action_name}`
|
||||
|
||||
### 1.6 Resource Usage (U1–U3)
|
||||
|
||||
**U1 — Daily compute hours:**
|
||||
Sum all workflow run durations from the last 24h.
|
||||
- 🔵 Info (metric only — for trends table)
|
||||
|
||||
**U2 — Eval runs count:**
|
||||
Count `evaluation` workflow runs in last 24h.
|
||||
- 🔵 Info (metric only)
|
||||
|
||||
**U3 — Cost trending up:**
|
||||
Use `cache-memory` to compare this week's compute hours to last week.
|
||||
- 🟡 Warning if >20% increase
|
||||
- Fingerprint: `resource:cost-increase`
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Fingerprint & Diff
|
||||
|
||||
After collecting all findings, perform the diff:
|
||||
|
||||
1. **Load previous fingerprints** from `cache-memory` key `health-check-fingerprints`. If not available, treat as empty (first run).
|
||||
|
||||
2. **Compute current fingerprints** for all findings collected in Step 1.
|
||||
|
||||
3. **Classify each finding:**
|
||||
- **🆕 NEW**: fingerprint is in current set but NOT in previous set
|
||||
- **📌 EXISTING**: fingerprint is in both current and previous sets
|
||||
- **✅ RESOLVED**: fingerprint is in previous set but NOT in current set
|
||||
|
||||
4. **Track occurrences**: For EXISTING findings, increment the `occurrences` counter from the previous state. Record `first_seen` date from when the finding first appeared.
|
||||
|
||||
5. **Save state** to `cache-memory`:
|
||||
- `health-check-fingerprints`: current fingerprint set (with occurrence counts and first_seen dates)
|
||||
- `health-check-history`: append today's summary `{ date, new_count, existing_count, resolved_count, by_severity: { critical, warning, info } }`
|
||||
|
||||
6. **Sort findings** within each diff category:
|
||||
- Primary sort: severity (🔴 → 🟡 → 🔵)
|
||||
- Secondary sort: category (pipeline → quality → pr → infra → resource)
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Analysis
|
||||
|
||||
Using the classified findings, generate:
|
||||
|
||||
1. **Executive summary**: One sentence describing what changed (e.g., "2 new issues detected, 1 resolved — eval pipeline is now healthy but a skill quality regression appeared")
|
||||
|
||||
2. **Correlation insights**: Identify connections between findings. For example:
|
||||
- A pipeline failure AND stale benchmark data → pipeline likely blocking data publication
|
||||
- Multiple quality regressions after the same date → look for a common commit
|
||||
|
||||
3. **Recommendations**: Prioritized list of suggested actions.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Output
|
||||
|
||||
### 4.1 Find or Create the Pinned Issue
|
||||
|
||||
Search for open issues with label `devops-health`:
|
||||
- If exactly one exists → update it
|
||||
- If none exist → create one with title `🏥 Repository Health Dashboard` and label `devops-health`
|
||||
- If multiple exist → update the most recently created one, close the others
|
||||
|
||||
Before creating/updating, ensure the `devops-health` label exists. If not, create it with color `#0E8A16` and description `Daily automated health check report`.
|
||||
|
||||
### 4.2 Issue Body Format
|
||||
|
||||
Replace the entire issue body with the following structure:
|
||||
|
||||
```markdown
|
||||
# 🏥 Daily Health Check — {date}
|
||||
|
||||
**Status:** 🔴 {critical_count} critical · 🟡 {warning_count} warnings · 🔵 {info_count} info
|
||||
**Since yesterday:** 🆕 {new_count} new · ✅ {resolved_count} resolved · 📌 {existing_count} unchanged
|
||||
|
||||
---
|
||||
|
||||
## 🧩 Skill Inventory
|
||||
|
||||
> Comprehensive health status of all skills derived from Q1–Q7 checks.
|
||||
|
||||
| Status | Component | Skill | Skilled | Vanilla | Δ | Scenarios | Issues |
|
||||
|--------|-----------|-------|--------:|--------:|--:|----------:|--------|
|
||||
{For each skill, sorted by component then skill name:}
|
||||
| {status_emoji} {status_label} | {component} | {skill_name} | {avg_skilled} | {avg_vanilla} | {delta} | {scenario_count} | {issue_summary} |
|
||||
|
||||
**Legend:** 🟢 OK · 🟡 Warning / Low Value · 🔴 No Value / Critical · ⚪ Untested / No Data
|
||||
|
||||
---
|
||||
|
||||
## 🆕 New Findings ({new_count})
|
||||
|
||||
> These appeared since the last health check ({previous_date}).
|
||||
|
||||
{For each new finding, render a full section with title, details, link, and suggested action}
|
||||
{Include investigation placeholder islands for findings that qualify for dispatch — see Step 5}
|
||||
|
||||
---
|
||||
|
||||
## ✅ Resolved Since Yesterday ({resolved_count})
|
||||
|
||||
> These were in yesterday's report but are no longer detected.
|
||||
|
||||
{For each resolved finding, render with strikethrough title and resolution info}
|
||||
|
||||
---
|
||||
|
||||
## 📌 Existing Findings ({existing_count})
|
||||
|
||||
> These have been present since before today. Sorted by age.
|
||||
|
||||
{Each existing finding in a collapsed <details> tag with first_seen and occurrence count}
|
||||
|
||||
---
|
||||
|
||||
## 📊 Trends (7-day)
|
||||
|
||||
| Metric | Today | 7d Avg | Δ | Trend |
|
||||
|--------|-------|--------|---|-------|
|
||||
| Eval duration (min) | {today} | {avg} | {delta} | {arrow} |
|
||||
| Eval success rate | {today} | {avg} | {delta} | {arrow} |
|
||||
| PRs merged/day | {today} | {avg} | {delta} | {arrow} |
|
||||
| Open PRs | {today} | {avg} | {delta} | {arrow} |
|
||||
| Compute hours/day | {today} | {avg} | {delta} | {arrow} |
|
||||
| Active skills | {count} | {avg} | {delta} | {arrow} |
|
||||
| Skills with issues | {count} | {avg} | {delta} | {arrow} |
|
||||
|
||||
---
|
||||
|
||||
<sub>🤖 Generated by DevOps Health Check agentic workflow · [Run #{run_number}](link) · {timestamp} UTC</sub>
|
||||
```
|
||||
|
||||
**Size guard:** If the issue body exceeds 60k characters:
|
||||
- Show all 🆕 NEW findings in full (up to 10)
|
||||
- Show all ✅ RESOLVED in full (up to 5)
|
||||
- Limit 📌 EXISTING to top 20 by severity in collapsed `<details>` tags
|
||||
- Append footer: `> … N additional existing findings omitted — see run artifacts for full report.`
|
||||
|
||||
### 4.3 Daily Comment
|
||||
|
||||
Append a short summary comment for the audit trail:
|
||||
|
||||
```markdown
|
||||
## 📋 Health Check — {date}
|
||||
|
||||
🆕 {new_count} new · ✅ {resolved_count} resolved · 📌 {existing_count} unchanged
|
||||
|
||||
**New:**
|
||||
{bullet list of new findings with emojis and links}
|
||||
|
||||
**Resolved:**
|
||||
{bullet list of resolved findings with strikethrough}
|
||||
|
||||
[Full report →]({issue_url})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Triage Dispatch
|
||||
|
||||
For each 🆕 NEW finding that qualifies for investigation, dispatch a worker:
|
||||
|
||||
### 5.1 Dispatch Rules
|
||||
|
||||
| Condition | Action |
|
||||
|-----------|--------|
|
||||
| 🆕 NEW + 🔴 Critical | **Always dispatch** |
|
||||
| 🆕 NEW + 🟡 Warning + category `pipeline` or `quality` | **Dispatch** |
|
||||
| 🆕 NEW + 🟡 Warning + category `pr` or `infra` | **Skip** (self-explanatory) |
|
||||
| 🆕 NEW + 🔵 Info | **Never dispatch** |
|
||||
| 📌 EXISTING (any) | **Never dispatch** |
|
||||
| ✅ RESOLVED (any) | **Never dispatch** |
|
||||
|
||||
**Budget:** Maximum 10 dispatches per run. If more than 10 qualify, prioritize by:
|
||||
1. Severity descending (🔴 first)
|
||||
2. Pipeline findings first
|
||||
3. Quality findings second
|
||||
|
||||
### 5.2 For Each Dispatched Finding
|
||||
|
||||
1. **Insert a placeholder island** in the issue body (within the 🆕 section, right after the finding details):
|
||||
|
||||
```markdown
|
||||
<!-- investigation:{finding_fingerprint} -->
|
||||
⏳ Investigation dispatched — results arriving shortly...
|
||||
<!-- /investigation:{finding_fingerprint} -->
|
||||
```
|
||||
|
||||
2. **Dispatch the worker:**
|
||||
|
||||
```
|
||||
dispatch-workflow:
|
||||
workflow: devops-health-investigate
|
||||
inputs:
|
||||
finding_id: "{fingerprint}"
|
||||
finding_type: "{category}"
|
||||
finding_title: "{title}"
|
||||
finding_severity: "{severity}"
|
||||
resource_url: "{link}"
|
||||
health_issue_number: "{issue_number}"
|
||||
correlation_id: "hc-{date}-{sequence}"
|
||||
```
|
||||
|
||||
3. **Wait 5 seconds** between dispatches (platform rate limit).
|
||||
|
||||
---
|
||||
|
||||
## Guidelines
|
||||
|
||||
- **Be data-driven**: Include specific numbers, durations, percentages, and links.
|
||||
- **Be precise with fingerprints**: Use the exact fingerprint formulas from the knowledge file. Consistency is critical — the same finding MUST produce the same fingerprint across runs.
|
||||
- **First run handling**: If `cache-memory` has no previous state, note: "⚠️ This is the first health check run. All findings appear as new. Diff will resume from next run."
|
||||
- **Graceful degradation**: If an API call fails, skip that check category and note the skip in the output. Don't fail the entire workflow.
|
||||
- **Noise awareness**: Demote known-noise findings (matching patterns in `cache-memory` `known-noise` list) to 🔵 Info severity, but still show them in the output for audit.
|
||||
- **Issue body limit**: Keep under 60k characters. Truncate EXISTING section if needed.
|
||||
- **Links everywhere**: Every finding should include at least one actionable link (to the run, PR, config file, etc.).
|
||||
+1174
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,150 @@
|
||||
---
|
||||
name: "DevOps Health — Deep Investigation"
|
||||
description: >
|
||||
Worker agent that performs deep root-cause analysis on a single
|
||||
health check finding. Dispatched by the health check orchestrator.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
finding_id:
|
||||
description: "Fingerprint ID of the finding to investigate"
|
||||
required: true
|
||||
finding_type:
|
||||
description: "Category: pipeline | quality | pr | infra | resource"
|
||||
required: true
|
||||
finding_title:
|
||||
description: "Human-readable title of the finding"
|
||||
required: true
|
||||
finding_severity:
|
||||
description: "Severity: critical | warning | info"
|
||||
required: true
|
||||
resource_url:
|
||||
description: "URL to the primary resource (run, PR, etc.)"
|
||||
required: true
|
||||
health_issue_number:
|
||||
description: "Issue number of the pinned health dashboard"
|
||||
required: true
|
||||
correlation_id:
|
||||
description: "Unique ID linking this investigation to the health check run"
|
||||
required: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
issues: read
|
||||
pull-requests: read
|
||||
|
||||
imports:
|
||||
- ../aw/shared/devops-investigate.lock.md
|
||||
|
||||
tools:
|
||||
github:
|
||||
toolsets: [repos, issues, pull_requests, actions]
|
||||
bash: ["cat", "grep", "head", "tail", "find", "ls", "wc", "jq", "date", "sort", "diff"]
|
||||
|
||||
safe-outputs:
|
||||
update-issue:
|
||||
max: 1
|
||||
add-comment:
|
||||
max: 1
|
||||
|
||||
network:
|
||||
allowed:
|
||||
- defaults
|
||||
---
|
||||
|
||||
# DevOps Health — Deep Investigation Worker
|
||||
|
||||
You are a specialized investigation agent. You have been dispatched by the DevOps Health Check orchestrator to perform a deep root-cause analysis on **one specific finding**.
|
||||
|
||||
## Your Mission
|
||||
|
||||
Investigate the finding identified by the inputs provided to this workflow run. Determine the root cause, assess the blast radius, and generate actionable remediation steps. Report your findings back to the pinned health issue.
|
||||
|
||||
## Inputs Available
|
||||
|
||||
- `finding_id`: `${{ inputs.finding_id }}` — The fingerprint ID of the finding
|
||||
- `finding_type`: `${{ inputs.finding_type }}` — Category (pipeline, quality, pr, infra, resource)
|
||||
- `finding_title`: `${{ inputs.finding_title }}` — Human-readable title
|
||||
- `finding_severity`: `${{ inputs.finding_severity }}` — Severity level
|
||||
- `resource_url`: `${{ inputs.resource_url }}` — URL to the primary resource
|
||||
- `health_issue_number`: `${{ inputs.health_issue_number }}` — Issue to update
|
||||
- `correlation_id`: `${{ inputs.correlation_id }}` — Links this investigation to the health check run
|
||||
|
||||
---
|
||||
|
||||
## Investigation Protocol
|
||||
|
||||
### Step 1: Route to Category-Specific Playbook
|
||||
|
||||
Based on `finding_type`, follow the appropriate investigation playbook from the compiled knowledge file:
|
||||
|
||||
- **pipeline** → Pipeline Investigation Playbook
|
||||
- **quality** → Quality Investigation Playbook
|
||||
- **pr** → PR Investigation Playbook
|
||||
- **infra** → Infrastructure Investigation Playbook
|
||||
- **resource** → Resource Investigation Playbook
|
||||
|
||||
### Step 2: Gather Evidence
|
||||
|
||||
Follow the playbook steps meticulously. For each piece of evidence:
|
||||
- Record the **source** (API endpoint, file path, log excerpt)
|
||||
- Note the **timestamp** of the evidence
|
||||
- Assess **relevance** to the finding
|
||||
|
||||
### Step 3: Determine Root Cause
|
||||
|
||||
Based on the gathered evidence:
|
||||
1. Identify the **most likely root cause**
|
||||
2. Assign a **confidence level**: High / Medium / Low
|
||||
- **High**: Direct evidence (error message explicitly states the cause, code change directly correlates)
|
||||
- **Medium**: Strong circumstantial evidence (timing correlates, pattern matches known issues)
|
||||
- **Low**: Inferential (possible but no direct evidence found)
|
||||
3. Identify the **blast radius** — what else is affected?
|
||||
4. Check for **related issues** — is this already tracked?
|
||||
|
||||
### Step 4: Generate Remediation Steps
|
||||
|
||||
Provide 1–3 specific, actionable remediation steps. Each step should:
|
||||
- Be concrete (include file paths, commands, or config changes)
|
||||
- Be ordered by recommended priority
|
||||
- Include any caveats or risks
|
||||
|
||||
### Step 5: Report Back
|
||||
|
||||
Update the pinned health issue by replacing the investigation island:
|
||||
|
||||
```
|
||||
update-issue:
|
||||
issue: {health_issue_number}
|
||||
operation: replace-island
|
||||
island: "investigation:{finding_id}"
|
||||
body: |
|
||||
🔍 **Investigation Complete** — [Worker Run #{this_run_number}]({this_run_url})
|
||||
|
||||
**Root cause:** {one-paragraph description with evidence}
|
||||
|
||||
**Confidence:** {High|Medium|Low} — {justification}
|
||||
|
||||
**Blast radius:** {what else is affected}
|
||||
|
||||
**Suggested fix:**
|
||||
1. {step 1}
|
||||
2. {step 2}
|
||||
3. {step 3} (if applicable)
|
||||
|
||||
**Related:** {commits, PRs, issues, or "None found"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Guidelines
|
||||
|
||||
- **Be factual**: Every claim must be backed by evidence from API responses, logs, or code.
|
||||
- **Don't hallucinate**: If you cannot determine the root cause, say so honestly. A "Low confidence" finding with honest uncertainty is better than a fabricated "High confidence" answer.
|
||||
- **Be concise**: The investigation report appears inline in the health dashboard. Keep it focused — 1-2 paragraphs for root cause, 1 paragraph for blast radius, numbered list for fixes.
|
||||
- **Include source evidence**: Quote specific error messages, log lines, or commit SHAs. Use code blocks for log excerpts.
|
||||
- **Check recent commits**: For pipeline and quality findings, always check commits between the last successful state and the current failure.
|
||||
- **Cross-reference**: Look for related open issues or PRs that might already be tracking this problem.
|
||||
- **Time-box yourself**: If evidence is insufficient after reasonable investigation, report what you found with appropriate confidence level rather than spiraling.
|
||||
@@ -0,0 +1,78 @@
|
||||
# DevOps Agentic Workflows
|
||||
|
||||
Cross-cutting [GitHub Agentic Workflow](https://github.com/github/gh-aw) workflows for repository-wide DevOps automation.
|
||||
|
||||
The workflow source files live in `.github/workflows/` and are compiled with `gh aw compile` to generate `.lock.yml` files (standard GitHub Actions YAML with security hardening). These workflows monitor the _entire repo_ (all components, all pipelines, all PRs), unlike the component-specific workflows in `src/dotnet-msbuild/agentic-workflows/`.
|
||||
|
||||
## Available Workflows
|
||||
|
||||
| Workflow | Description | Trigger |
|
||||
|----------|-------------|---------|
|
||||
| [devops-health-check](../../.github/workflows/devops-health-check.md) | Daily orchestrator that collects repo health signals (pipelines, skill quality, PRs, infrastructure), computes a fingerprint-based diff against the previous run, and updates a pinned health dashboard issue | `schedule: daily` (fuzzy daily), `/health-check` slash command, `workflow_dispatch` |
|
||||
| [devops-health-investigate](../../.github/workflows/devops-health-investigate.md) | Worker agent dispatched by the health check orchestrator to perform deep root-cause analysis on individual findings | `workflow_dispatch` (dispatched by orchestrator via `dispatch-workflow`) |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
devops-health-check (Orchestrator)
|
||||
├─ Collects health signals from 5 categories:
|
||||
│ Pipeline · Quality · PRs · Infrastructure · Resources
|
||||
├─ Fingerprints each finding for stable diff tracking
|
||||
├─ Classifies: 🆕 NEW · 📌 EXISTING · ✅ RESOLVED
|
||||
├─ Updates pinned health dashboard issue
|
||||
└─ Dispatches investigation workers (up to 10)
|
||||
│
|
||||
▼
|
||||
devops-health-investigate (Worker × N)
|
||||
├─ Investigates ONE finding with fresh context
|
||||
├─ Follows category-specific playbook
|
||||
├─ Determines root cause + remediation
|
||||
└─ Updates health issue via replace-island
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install the `gh aw` CLI extension: `gh extension install github/gh-aw`
|
||||
2. Compile: `gh aw compile` (from the repo root — this compiles all `.md` files in `.github/workflows/`)
|
||||
3. Commit both the `.md` and generated `.lock.yml` files
|
||||
4. The health check runs daily, or on-demand via `/health-check`
|
||||
|
||||
## Local Development
|
||||
|
||||
```powershell
|
||||
# Compile workflows (generates .lock.yml from .md frontmatter)
|
||||
gh aw compile
|
||||
|
||||
# Compile with validation
|
||||
gh aw compile --strict
|
||||
|
||||
# Dry-run (validates without triggering on GitHub Actions)
|
||||
gh aw run devops-health-check --dry-run
|
||||
|
||||
# Run on GitHub Actions (from a pushed branch)
|
||||
gh aw run devops-health-check --push --ref <branch>
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
.github/
|
||||
├── workflows/
|
||||
│ ├── devops-health-check.md # Orchestrator workflow
|
||||
│ ├── devops-health-check.lock.yml # Compiled workflow (generated by gh aw compile)
|
||||
│ ├── devops-health-investigate.md # Worker workflow
|
||||
│ └── devops-health-investigate.lock.yml # Compiled workflow (generated by gh aw compile)
|
||||
└── aw/
|
||||
└── shared/
|
||||
├── devops-health.lock.md # Health check catalog & fingerprinting rules
|
||||
└── devops-investigate.lock.md # Investigation playbooks & remediation templates
|
||||
|
||||
eng/agentic-workflows/
|
||||
├── README.md # This file (documentation)
|
||||
└── pr-141-review-fixes.md # PR review analysis
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [src/dotnet-msbuild/agentic-workflows/](../../src/dotnet-msbuild/agentic-workflows/) — MSBuild-specific agentic workflows
|
||||
- [eng/dashboard/](../dashboard/) — Benchmark dashboard (data source for quality checks)
|
||||
Reference in New Issue
Block a user