Improve DevOps health incident automation

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

.github/aw/actions-lock.json

@@ -20,15 +20,10 @@
       "version": "v7.0.1",
       "sha": "043fb46d1a93c77aae656e7c1c64a875d1fc6a0a"
     },
-    "github/gh-aw-actions/setup-cli@v0.88.2": {
-      "repo": "github/gh-aw-actions/setup-cli",
-      "version": "v0.88.2",
-      "sha": "9271a1804551c0dc4fb0085a97979950aa2f8489"
-    },
-    "github/gh-aw-actions/setup@v0.88.2": {
+    "github/gh-aw-actions/setup@v0.88.7": {
       "repo": "github/gh-aw-actions/setup",
-      "version": "v0.88.2",
-      "sha": "9271a1804551c0dc4fb0085a97979950aa2f8489"
+      "version": "v0.88.7",
+      "sha": "5e508589e03a7757a7e05b26e834292f5445bfb6"
     }
   },
   "containers": {

.github/workflows/devops-health-check.md

@@ -276,6 +276,11 @@ After collecting all findings, perform the diff:
    - Primary sort: severity (🔴 → 🟡 → 🔵)
    - Secondary sort: category (pipeline → infra → resource)
 
+The `known-noise` key is optional configuration. If it is absent, use an empty
+list and continue normally. Do NOT call `missing-data` or report a cache miss for
+an absent `known-noise` key. Only report missing cache data when a required key
+was restored successfully but cannot be read or parsed.
+
 ---
 
 ## Step 3: Analysis
@@ -488,7 +493,10 @@ Before finishing, verify:
 - [ ] At least one `dispatch-workflow` call was made (if any 🔴 critical or qualifying 🟡 warning findings exist)
 - [ ] All 🔴 critical NEW findings have been dispatched (up to budget cap)
 - [ ] The "🔍 Investigation Results" section in the issue body includes newly dispatched findings as "🔄 Dispatched" and preserves existing rows from the previous body
-- [ ] The noop summary message mentions how many investigations were dispatched
+- [ ] If no other safe output was emitted, the `noop` summary mentions that zero
+      investigations were dispatched
+- [ ] If `update-issue`, `add-comment`, or `dispatch-workflow` was emitted, do
+      not call `noop`
 
 ---
 
@@ -496,6 +504,7 @@ Before finishing, verify:
 
 - **Time budget**: You have a 60-minute timeout. Prioritize reaching Steps 4 and 5 (issue update + dispatch). Do NOT write intermediate scripts or analysis files. Work through each check, collect findings in memory, and proceed directly to output. Aim to complete data collection (Step 1) within 30 minutes.
 - **`cache-memory` persists automatically — do NOT manage it with `git`**: The `cache-memory` tool loads and saves state on its own. Never run `git` commands (e.g. `git config`, `git -C /tmp/gh-aw/cache-memory log/add/commit`) against the cache directory to inspect or persist state — use the `cache-memory` load/save operations described in Step 2. Manual git plumbing is unnecessary and only burns the effective-token budget.
+- **Optional cache keys are not missing data**: `known-noise` is optional. Its absence means "no noise patterns configured." Continue with an empty list and do not call `missing-data`. Reserve `missing-data` for required inputs that are unavailable and prevent a required result.
 - **Token budget — don't retry denied commands**: The bash tool only permits the commands in the `bash:` allowlist. If a command is denied, do NOT re-issue the same or a slightly reworded command in a loop — repeated denials re-process the full context and exhaust the effective-token budget, failing the run. Use an allowed alternative (`jq`/`grep`/`sed`) or skip that sub-step and note it, then move on.
 - **Efficiency**: Process API responses in memory. Do NOT create Python/bash scripts to analyze data — parse JSON directly using `jq` or inline analysis. Do NOT write intermediate files unless explicitly required by the output format. The bash allowlist does NOT include `python`, `python3`, `node`, or other general-purpose language runtimes — any attempt to invoke them WILL be blocked by security policy. Use `jq` for all JSON processing.
 - **CRITICAL — Safe output body must be inline**: When calling `update-issue`, the `body` field must contain the **complete, literal issue body text**. NEVER write the body to a file and use a shell reference like `$(cat file.txt)` — safe outputs are literal JSON strings, not shell-evaluated. Pass the body directly as the string value.

.github/workflows/devops-health-groom.md

@@ -24,8 +24,8 @@ permissions:
   issues: read
 
 tools:
-  bash: []
-  cli-proxy: false
+  bash: ["github", "safeoutputs"]
+  cli-proxy: true
   github:
     toolsets: [repos, issues, actions]
     min-integrity: none
@@ -345,10 +345,10 @@ it in the Step 6 `noop` message **only** when that `noop` summary is emitted
 
 ## Step 6: Summary
 
-Call safe-output tools directly. Never invoke `safeoutputs` through a shell,
-pipeline, or generated command. A successful shell command does not record a
-safe-output declaration. The `safeoutputs` CLI is unavailable in this
-workflow; use the direct tool even if generic CLI guidance says otherwise.
+Prefer direct safe-output tools. If the runtime presents the same tools through
+the authenticated MCP CLI proxy, `safeoutputs <tool>` is an allowed fallback
+and records the same safe-output declaration. Never use `gh` for GitHub reads
+or writes in this workflow.
 
 After completing all steps, if no `update-issue` or `hide-comment` calls were made, call `noop` with a summary message:
 
@@ -366,7 +366,7 @@ If changes were made, the summary is implicit in the safe-output calls. Do NOT c
 ## Guidelines
 
 - **CRITICAL — Use `operation: "replace-island"`**: When calling `update-issue`, you **MUST** set `operation: "replace-island"`. This replaces only the `## 🔍 Investigation Results` section in the issue body, leaving all other sections untouched. The `body` field must contain only the Investigation Results section content (from the `## 🔍 Investigation Results` heading up to but not including the next `##`-level heading). Do NOT pass the full issue body — `replace-island` handles scoping automatically. If multiple `## 🔍 Investigation Results` sections exist in the body, `replace-island` targets the first one — the groomer must merge all rows from every occurrence into that single section before calling `replace-island`. Later duplicate sections are not automatically removed; the next health-check run (which replaces the full body) will clean them up.
-- **CRITICAL — Call safe-output tools directly**: Use the `update_issue`, `hide_comment`, or `noop` tool. Do NOT call `safeoutputs` from a shell or pipe JSON to it. Shell execution is not a safe-output declaration.
+- **CRITICAL — Produce a safe output**: Use `update_issue`, `hide_comment`, or `noop` directly. If direct invocation is unavailable, use the authenticated `safeoutputs` MCP CLI proxy as a fallback. Do not finish with only a text response.
 - **CRITICAL — Safe output body must be inline**: When calling `update-issue`, the `body` field must contain the **literal section text**. NEVER write the body to a file and use a shell reference like `$(cat file.txt)` — safe outputs are literal JSON strings, not shell-evaluated. The body must be passed directly as the string value.
 - **Minimal edits only**: You are a groomer, not a rewriter. Only change: (a) investigation table rows (status + link), (b) resolved-finding annotations. Copy all other sections **byte-for-byte** from the original body. Do not reformat, re-wrap, or reorganize sections you are not changing.
 - **Be precise with comment parsing**: The comment format is well-defined (see the investigation worker template). Match the exact patterns — don't be fuzzy.
@@ -379,4 +379,4 @@ If changes were made, the summary is implicit in the safe-output calls. Do NOT c
 - **No intermediate files**: Do all work in memory. Do NOT write intermediate scripts, JSON files, or body text files. Hold parsed data and the issue body as in-memory variables.
 - **Use MCP `issue_read` for fetching comments**: Use the GitHub MCP `issue_read` tool with `method: get_comments` for fetching issue comments. If the response includes a `[Filtered]` notice, continue working with the comments that were returned — filtered items are from non-bot authors and are irrelevant to grooming. Do NOT call `report_incomplete` or `missing_tool` because of filtered items.
 - **Missing `node_id` never fails the run**: `hide-comment` needs a comment's GraphQL `node_id`, but `issue_read(get_comments)` sometimes omits it. When a comment has no `node_id`, skip hiding that one comment and continue — do NOT call `missing_tool`/`report_incomplete` or report missing data. Result linking (Steps 3–4) does not use `node_id`, and the weekly cleanup workflow removes old comments by age regardless.
-- **`gh` CLI is NOT authenticated in the sandbox**: Never use `gh api` or other `gh` commands for GitHub API calls — the sandbox strips credentials by design. Use MCP tools for all GitHub reads.
+- **Use authenticated MCP tools**: Prefer direct GitHub MCP and safe-output tools. The `github` and `safeoutputs` MCP CLI proxy commands are available as a fallback. The ordinary `gh` CLI is not authenticated in the sandbox and must not be used.

.github/workflows/devops-health-investigate.md

@@ -3,7 +3,9 @@ name: "DevOps Health — Deep Investigation"
 description: >
   Worker agent that performs deep root-cause analysis on a single
   health check finding (pipeline, infrastructure, or resource).
-  Dispatched by the health check orchestrator.
+  Dispatched by the health check orchestrator. For repository-controlled
+  infrastructure faults, it validates and multi-model reviews a minimal fix,
+  then opens a draft pull request.
 
 on:
   permissions: {}
@@ -30,9 +32,15 @@ on:
       correlation_id:
         description: "Unique ID linking this investigation to the health check run"
         required: true
+      dry_run:
+        description: "Investigate and validate without posting comments or creating a PR"
+        required: false
+        type: boolean
+        default: false
 
 concurrency:
   group: gh-aw-${{ github.workflow }}-${{ inputs.finding_id }}
+  job-discriminator: ${{ github.run_id }}
 
 model: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'gpt-5.6-sol' }}
 
@@ -45,11 +53,24 @@ permissions:
 tools:
   github:
     toolsets: [repos, issues, pull_requests, actions]
-  bash: ["cat", "grep", "head", "tail", "find", "ls", "wc", "jq", "date", "sort", "diff"]
+  bash: ["cat", "grep", "head", "tail", "find", "ls", "wc", "jq", "date", "sort", "diff", "git", "python", "python3", "node", "npm", "npx", "dotnet", "pwsh"]
+  edit:
 
 safe-outputs:
+  staged: ${{ inputs.dry_run }}
   add-comment:
     max: 1
+  create-pull-request:
+    max: 1
+    draft: true
+    protected-files: fallback-to-issue
+    fallback-as-issue: true
+    max-patch-files: 20
+    max-patch-size: 1024
+    allowed-files:
+      - "eng/**"
+      - "plugins/*/plugin.json"
+      - "Directory.Build.*"
   noop:
     report-as-issue: false
 
@@ -97,6 +118,7 @@ Investigate the finding identified by the inputs provided to this workflow run.
 - `resource_url`: `${{ inputs.resource_url }}` — URL to the primary resource
 - `health_issue_number`: `${{ inputs.health_issue_number }}` — Issue to update
 - `correlation_id`: `${{ inputs.correlation_id }}` — Links this investigation to the health check run
+- `dry_run`: `${{ inputs.dry_run }}` — When true, do not post a comment or create a PR
 
 ---
 
@@ -116,6 +138,10 @@ Follow the playbook steps meticulously. For each piece of evidence:
 - Record the **source** (API endpoint, file path, log excerpt)
 - Note the **timestamp** of the evidence
 - Assess **relevance** to the finding
+- Read the relevant repository files and their recent Git history.
+- Find the last successful run of the same workflow and compare its commit with
+  the failed run.
+- Search open and closed issues and pull requests for the same failure signature.
 
 ### Step 3: Determine Root Cause
 
@@ -128,14 +154,117 @@ Based on the gathered evidence:
 3. Identify the **blast radius** — what else is affected?
 4. Check for **related issues** — is this already tracked?
 
-### Step 4: Generate Remediation Steps
+### Step 4: Decide Whether an Automatic Fix Is Safe
 
-Provide 1–3 specific, actionable remediation steps. Each step should:
-- Be concrete (include file paths, commands, or config changes)
-- Be ordered by recommended priority
-- Include any caveats or risks
+Classify the finding before editing files.
 
-### Step 5: Report Back
+An automatic fix is eligible only when all conditions are true:
+
+1. The root cause is in repository-controlled files.
+2. Confidence is High, with direct log, diff, or configuration evidence.
+3. The change is minimal, reversible, and within the `create-pull-request`
+   `allowed-files` scope.
+4. The change does not modify secrets, credentials, repository settings,
+   permissions, deployment behavior, billing, or external service state.
+5. The change does not remove dependencies, upgrade a major dependency version,
+   or weaken validation, security, required checks, or error reporting.
+6. A targeted validation can reproduce the failure or prove the configuration
+   defect, and the same validation passes after the change.
+7. No existing open pull request already contains an equivalent fix.
+
+If any condition is false or uncertain, do not edit files. Report the evidence,
+the suggested fix, and the owner who must take the next action.
+
+Files under `.github/` and protected root manifests are outside the automatic
+edit scope. This repository does not provide the GitHub App credential required
+for automated workflow-file pushes. For a validated fix that touches one of
+these files, do not edit files. Report the complete proposed patch, validation
+evidence, MMR results, and permission limit. Do not claim that a pull request
+was created.
+
+### Step 5: Generate and Implement the Fix
+
+First, provide 1–3 specific remediation steps. Each step must:
+- Be concrete and include file paths, commands, or config changes.
+- Be ordered by recommended priority.
+- Include caveats and risks.
+
+When the automatic-fix gate passes:
+
+1. Make the smallest repository change that fixes the root cause.
+2. Add or update a regression test when the repository has a suitable test
+   surface.
+3. Run the smallest targeted validation that reproduces the original failure.
+4. Run directly related format, compile, lint, and test checks.
+5. If an agentic workflow source changes, run
+   `gh aw compile <workflow-id> --strict`, include its generated lock file, and
+   inspect the lock-file diff. Do not edit generated lock files by hand.
+6. If any required validation is unavailable, fails, or does not cover the
+   original failure, stop. Revert the attempted edits and report a suggested
+   fix only.
+
+### Step 6: Mandatory Multi-Model Review
+
+Before creating a pull request, prepare one review brief with:
+
+- finding, root cause, and evidence;
+- relevant history and last-success comparison;
+- complete diff;
+- tests and exact results;
+- risks, assumptions, and blast radius.
+
+Send the same brief to all three review agents:
+
+1. `infra-review-claude`
+2. `infra-review-gpt`
+3. `infra-review-gemini`
+
+Invoke each named inline agent and keep its separate response as review
+evidence. Do not write a review on an agent's behalf.
+
+Each reviewer must check correctness, security, performance, maintainability,
+customer regression risk, whether the change matches the finding, whether
+history shows hidden behavior, secret exposure, and whether shipped artifacts
+change unexpectedly.
+
+Consolidate all findings. Do not average away disagreements. Quote material
+dissent exactly. Fix every confirmed blocking or high-confidence finding, rerun
+the affected checks, and repeat the three reviews on the final diff if the fix
+changed materially.
+
+Create a PR only when:
+
+- all three model families returned a review;
+- there are no unresolved blocking findings;
+- the original failure is covered by passing validation;
+- the final diff stays within the automatic-fix gate;
+- the safe-output handler can create the branch for every changed file.
+
+### Step 7: Create a Draft Pull Request
+
+If `dry_run` is true, skip this step. Do not emit a safe output here; Step 8
+emits the one dry-run result.
+
+Otherwise, call `create_pull_request` with:
+
+- a concise branch name under `automation/infra-fix-`;
+- a title that states the fix, not the investigation process;
+- `draft: true`;
+- a body that follows the repository pull request description style:
+  - `Fixes #<issue>` when a tracking issue exists, otherwise `Relates to
+    #<health_issue_number>`;
+  - `## Summary` with what changed and why;
+  - `## Root cause` with direct evidence and history;
+  - `## Validation` with exact commands and results;
+  - `## Multi-model review` with the three models, consolidated findings, fixes,
+    and any material dissent;
+  - `## Risk` with remaining limits and rollback guidance.
+
+Never enable auto-merge. Never mark the PR ready for review.
+If protected-file policy produces a fallback issue instead, report it as a
+validated fix proposal, not as a draft PR.
+
+### Step 8: Report Back
 
 Post your investigation results as a comment on the pinned health issue.
 
@@ -165,6 +294,12 @@ add-comment:
     2. {step 2}
     3. {step 3} (if applicable)
 
+    ### Automatic Fix
+    {Draft PR link and validation summary, or why the automatic-fix gate did not pass}
+
+    ### Multi-Model Review
+    {Claude, GPT, and Gemini verdicts; consolidated findings; material dissent}
+
     ### Evidence
     {key log excerpts, API responses, or code references}
 
@@ -175,6 +310,12 @@ add-comment:
     <sub>🔍 [Investigation Run #{this_run_number}]({this_run_url}) · Dispatched by health check · {correlation_id}</sub>
 ```
 
+If `dry_run` is true, do not call `add-comment` or `create_pull_request`. Call
+`noop` exactly once with a compact summary of the root cause, automatic-fix
+decision, proposed patch, validation plan, and MMR result. Safe outputs are
+also staged for dry runs, so an accidental mutating output can only produce a
+preview and cannot change GitHub state.
+
 ---
 
 ## Guidelines
@@ -185,4 +326,38 @@ add-comment:
 - **Include source evidence**: Quote specific error messages, log lines, or commit SHAs. Use code blocks for log excerpts.
 - **Check recent commits**: For pipeline and quality findings, always check commits between the last successful state and the current failure.
 - **Cross-reference**: Look for related open issues or PRs that might already be tracking this problem.
+- **No speculative PRs**: A plausible fix is not enough. Require direct root-cause evidence, passing validation for the original failure, and three-family MMR.
+- **One fix per PR**: Do not combine unrelated findings. If one root cause explains several failures, list every covered failure in the PR body.
+- **Existing fix wins**: If an open PR already fixes the root cause, do not create a duplicate. Link that PR in the report.
 - **Time-box yourself**: If evidence is insufficient after reasonable investigation, report what you found with appropriate confidence level rather than spiraling.
+
+## agent: `infra-review-claude`
+---
+description: Reviews an infrastructure fix for correctness, safety, regression risk, and historical consistency
+model: claude-sonnet-5
+---
+Review only the supplied evidence, diff, history, and test results. Identify
+blocking defects and high-confidence risks. Verify that the patch fixes the
+reported root cause without weakening controls or changing unrelated behavior.
+Quote evidence for every finding. Return `APPROVE` only when no blocking issue
+remains.
+
+## agent: `infra-review-gpt`
+---
+description: Reviews an infrastructure fix for correctness, security, validation quality, and scope
+model: gpt-5.6-terra
+---
+Review only the supplied evidence, diff, history, and test results. Check the
+failure-to-fix chain, test adequacy, security boundaries, error handling, and
+scope. Identify hidden behavior changes and artifact changes. Quote evidence for
+every finding. Return `APPROVE` only when no blocking issue remains.
+
+## agent: `infra-review-gemini`
+---
+description: Reviews an infrastructure fix for alternative explanations, edge cases, and operational reliability
+model: gemini-3.7-flash
+---
+Review only the supplied evidence, diff, history, and test results. Challenge the
+root-cause hypothesis, search for missed edge cases in the provided material,
+and assess operational reliability and rollback. Quote evidence for every
+finding. Return `APPROVE` only when no blocking issue remains.

eng/evaluation/test_token_failover.py

@@ -126,6 +126,50 @@ class TokenFailoverTests(unittest.TestCase):
                 )
                 self.assertEqual(frontmatter["environment"], "copilot-pat-pool")
 
+    def test_devops_health_automation_can_safely_propose_fixes(self) -> None:
+        workflows = REPO_ROOT / ".github" / "workflows"
+        health_check = (workflows / "devops-health-check.md").read_text(
+            encoding="utf-8"
+        )
+        groom_source = workflows / "devops-health-groom.md"
+        groom = groom_source.read_text(encoding="utf-8")
+        groom_frontmatter = yaml.safe_load(groom.split("---", 2)[1])
+        investigate_source = workflows / "devops-health-investigate.md"
+        investigate = investigate_source.read_text(encoding="utf-8")
+        investigate_frontmatter = yaml.safe_load(investigate.split("---", 2)[1])
+
+        self.assertIn("Optional cache keys are not missing data", health_check)
+        self.assertIn("do not call `missing-data`", health_check)
+        self.assertIn("If `update-issue`, `add-comment`, or `dispatch-workflow`", health_check)
+        self.assertTrue(groom_frontmatter["tools"]["cli-proxy"])
+        self.assertIn("Do not finish with only a text response", groom)
+
+        trigger = investigate_frontmatter.get("on", investigate_frontmatter.get(True))
+        dispatch_inputs = trigger["workflow_dispatch"]["inputs"]
+        self.assertEqual(dispatch_inputs["dry_run"]["type"], "boolean")
+        self.assertFalse(dispatch_inputs["dry_run"]["default"])
+
+        create_pr = investigate_frontmatter["safe-outputs"]["create-pull-request"]
+        self.assertEqual(
+            investigate_frontmatter["safe-outputs"]["staged"],
+            "${{ inputs.dry_run }}",
+        )
+        self.assertTrue(create_pr["draft"])
+        self.assertNotIn("allow-workflows", create_pr)
+        self.assertEqual(create_pr["protected-files"], "fallback-to-issue")
+        self.assertNotIn(".github/workflows/**", create_pr["allowed-files"])
+        self.assertFalse(
+            any(path.startswith(".github/") for path in create_pr["allowed-files"])
+        )
+        self.assertLessEqual(create_pr["max-patch-files"], 20)
+        self.assertNotIn("gh", investigate_frontmatter["tools"]["bash"])
+
+        for model in ("claude-sonnet-5", "gpt-5.6-terra", "gemini-3.7-flash"):
+            self.assertIn(f"model: {model}", investigate)
+        self.assertIn("all three model families returned a review", investigate)
+        self.assertIn("If `dry_run` is true, skip this step", investigate)
+        self.assertIn("`noop` exactly once", investigate)
+
     def run_selector(
         self,
         tokens: dict[int, str],