feat: launch Herdr investigator for gateway alerts

Persist and deduplicate gateway incidents, open one Aqua Herdr workspace, launch the DGX GLM investigator, and retain native notification fallback.

.brain/resources/gateway-herdr-alert-investigator.svx

@@ -0,0 +1,54 @@
+---
+title: "Gateway alerts open a Herdr investigator"
+type: "resource"
+created_at: "2026-09-10"
+privacy: "private"
+---
+
+# Gateway alerts open a Herdr investigator
+
+A gateway heartbeat outage should create useful work, not repeat the same macOS notification every five minutes.
+
+`infra/gateway-tripwire.sh` remains the sensor. On each `missing` or `stale` check, it invokes `infra/gateway-alert-investigator.sh`. The investigator owns deduplication, retries, and the Herdr lifecycle.
+
+## State machine
+
+```text
+idle
+  -> ensuringHerdr
+  -> openingWorkspace
+  -> startingAgent
+  -> promptingAgent
+  -> investigating
+  -> recovering
+  -> recovered
+
+Any setup failure -> blocked -> retry
+A failed recovery receipt -> recoveryBlocked -> recovering
+A lost workspace -> ensuringHerdr
+A lost agent -> startingAgent in the same workspace
+A new alert after recovery -> a new incident
+```
+
+The state record lives at `~/.local/state/joelclaw/gateway-alert-investigator.json`. One active incident owns one workspace and one agent. Repeated tripwire checks reuse it.
+
+## Runtime contract
+
+- Use the default Aqua Herdr session. Clear `HERDR_SESSION` and `HERDR_SOCKET` before Herdr commands.
+- If the default server is unavailable, kickstart `gui/<uid>/com.joelclaw.herdr-server`.
+- Create a no-focus workspace in the canonical joelclaw checkout. Use the incident ID in its first label so a crash retry can discover it instead of creating a duplicate.
+- Verify that the configured DGX endpoint advertises `glm-5.3-flash`, then start Pi with `dgx-glm/glm-5.3-flash:high` and normal tools.
+- If the model endpoint is unavailable, keep the workspace, enter `blocked`, and retry on a later tripwire check.
+- The investigator may diagnose and make safe, reversible repairs under the normal fleet and project instructions.
+- Never start a second gateway or outward communications transport.
+- Never send an outward canary without explicit approval.
+- On recovery, rename the workspace to `review ready` and ask the same agent for a final health and root-cause receipt. Keep recovery pending until both actions succeed. Healthy sensor checks retry this receipt without falsifying the tripwire health state.
+- If Herdr or the investigator cannot start, retain one edge-triggered native notification as the failure path.
+
+## Verification
+
+```sh
+bun test packages/gateway/src/gateway-alert-investigator.test.ts
+infra/gateway-tripwire.test.sh
+shellcheck infra/gateway-alert-investigator.sh infra/gateway-tripwire.sh infra/gateway-tripwire.test.sh
+```

infra/gateway-alert-investigator.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+# Start or update the default-Herdr DGX GLM investigator for a gateway alert.
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+BUN_BIN="${GATEWAY_ALERT_INVESTIGATOR_BUN_BIN:-$HOME/.bun/bin/bun}"
+
+exec "$BUN_BIN" run "$ROOT/packages/gateway/src/gateway-alert-investigator.ts" "$@"

infra/gateway-tripwire.sh

@@ -1,5 +1,6 @@
 #!/bin/bash
-# Gateway tripwire — checks every five minutes and notifies on unhealthy state changes.
+# Gateway tripwire — checks every five minutes and dispatches unhealthy states
+# to one deduplicated DGX GLM investigator in the default Herdr session.
 # The gateway transport writes its PID and heartbeat after dependency preflight succeeds.
 set -u
 
@@ -8,6 +9,7 @@ THRESHOLD="${GATEWAY_TRIPWIRE_THRESHOLD_SECONDS:-1800}"
 PID_FILE="${GATEWAY_PID_FILE:-/tmp/joelclaw/gateway.pid}"
 STATE_FILE="${GATEWAY_TRIPWIRE_STATE_FILE:-/tmp/joelclaw/gateway-tripwire.state}"
 OSASCRIPT_BIN="${GATEWAY_TRIPWIRE_OSASCRIPT_BIN:-/usr/bin/osascript}"
+INVESTIGATOR_BIN="${GATEWAY_ALERT_INVESTIGATOR_BIN:-$HOME/Code/joelhooks/joelclaw/infra/gateway-alert-investigator.sh}"
 
 etime_to_seconds() {
   local etime="$1"
@@ -42,17 +44,40 @@ write_state() {
   printf '%s\n' "$1" > "$STATE_FILE"
 }
 
-notify_on_transition() {
+fallback_notification_on_transition() {
   local next_state="$1"
   local message="$2"
   local previous_state
   previous_state="$(current_state)"
   if [ "$previous_state" != "$next_state" ]; then
-    "$OSASCRIPT_BIN" -e "display notification \"$message\" with title \"🚨 joelclaw\""
+    if ! "$OSASCRIPT_BIN" -e "display notification \"$message\" with title \"🚨 joelclaw\""; then
+      return 1
+    fi
   fi
   write_state "$next_state"
 }
 
+dispatch_alert() {
+  local alert_state="$1"
+  local age_seconds="${2:-}"
+  if [ -x "$INVESTIGATOR_BIN" ]; then
+    if [ -n "$age_seconds" ]; then
+      "$INVESTIGATOR_BIN" alert "$alert_state" "$age_seconds"
+    else
+      "$INVESTIGATOR_BIN" alert "$alert_state"
+    fi
+    return
+  fi
+  return 127
+}
+
+dispatch_recovery() {
+  if [ ! -x "$INVESTIGATOR_BIN" ]; then
+    return 1
+  fi
+  "$INVESTIGATOR_BIN" recover
+}
+
 if [ ! -f "$HEARTBEAT_FILE" ]; then
   # Reboot/startup grace: the transport can be healthy before dependency
   # preflight completes and publishes its first heartbeat.
@@ -70,7 +95,11 @@ if [ ! -f "$HEARTBEAT_FILE" ]; then
     fi
   fi
 
-  notify_on_transition missing "Gateway heartbeat file missing!"
+  if dispatch_alert missing; then
+    write_state missing
+  else
+    fallback_notification_on_transition missing "Gateway heartbeat missing; Herdr investigator failed to start"
+  fi
   exit 0
 fi
 
@@ -79,8 +108,13 @@ NOW="$(date +%s)"
 AGE=$((NOW - LAST))
 
 if [ "$AGE" -gt "$THRESHOLD" ]; then
-  notify_on_transition stale "Gateway heartbeat stale (${AGE}s ago)"
+  if dispatch_alert stale "$AGE"; then
+    write_state stale
+  else
+    fallback_notification_on_transition stale "Gateway heartbeat stale; Herdr investigator failed to start"
+  fi
   exit 0
 fi
 
+dispatch_recovery || true
 write_state healthy

infra/gateway-tripwire.test.sh

@@ -10,22 +10,44 @@ FAKE_OSASCRIPT="$TMP/osascript"
 cat > "$FAKE_OSASCRIPT" <<'SCRIPT'
 #!/bin/bash
 printf '%s\n' "$*" >> "$FAKE_NOTIFICATION_LOG"
+if [ -f "$FAKE_OSASCRIPT_FAIL" ]; then
+  exit 1
+fi
 SCRIPT
 chmod +x "$FAKE_OSASCRIPT"
 
+FAKE_INVESTIGATOR="$TMP/gateway-alert-investigator"
+cat > "$FAKE_INVESTIGATOR" <<'SCRIPT'
+#!/bin/bash
+printf '%s\n' "$*" >> "$FAKE_INVESTIGATOR_LOG"
+if [ "$1" = "recover" ] && [ -f "$FAKE_INVESTIGATOR_FAIL_RECOVER_ONCE" ]; then
+  rm "$FAKE_INVESTIGATOR_FAIL_RECOVER_ONCE"
+  exit 1
+fi
+if [ -f "$FAKE_INVESTIGATOR_FAIL" ]; then
+  exit 1
+fi
+SCRIPT
+chmod +x "$FAKE_INVESTIGATOR"
+
 run_tripwire() {
   GATEWAY_HEARTBEAT_FILE="$TMP/heartbeat.ts" \
   GATEWAY_PID_FILE="$TMP/gateway.pid" \
   GATEWAY_TRIPWIRE_STATE_FILE="$TMP/tripwire.state" \
   GATEWAY_TRIPWIRE_THRESHOLD_SECONDS="${GATEWAY_TRIPWIRE_THRESHOLD_SECONDS:-1}" \
   GATEWAY_TRIPWIRE_OSASCRIPT_BIN="$FAKE_OSASCRIPT" \
+  GATEWAY_ALERT_INVESTIGATOR_BIN="$FAKE_INVESTIGATOR" \
   FAKE_NOTIFICATION_LOG="$TMP/notifications.log" \
+  FAKE_OSASCRIPT_FAIL="$TMP/osascript.fail" \
+  FAKE_INVESTIGATOR_LOG="$TMP/investigator.log" \
+  FAKE_INVESTIGATOR_FAIL="$TMP/investigator.fail" \
+  FAKE_INVESTIGATOR_FAIL_RECOVER_ONCE="$TMP/investigator-recover-once.fail" \
     "$TRIPWIRE"
 }
 
-notification_count() {
-  if [ -f "$TMP/notifications.log" ]; then
-    wc -l < "$TMP/notifications.log" | tr -d '[:space:]'
+line_count() {
+  if [ -f "$1" ]; then
+    wc -l < "$1" | tr -d '[:space:]'
   else
     printf '0\n'
   fi
@@ -38,34 +60,76 @@ assert_eq() {
   fi
 }
 
-# One prolonged missing-file outage emits one notification, not one every run.
+# Every unhealthy check reaches the investigator so its own state machine can
+# retry failures. The investigator owns incident deduplication.
 run_tripwire
 run_tripwire
-assert_eq 1 "$(notification_count)"
+assert_eq 2 "$(line_count "$TMP/investigator.log")"
+assert_eq 0 "$(line_count "$TMP/notifications.log")"
 assert_eq missing "$(cat "$TMP/tripwire.state")"
+assert_eq "alert missing" "$(head -1 "$TMP/investigator.log")"
 
-# Recovery rearms the next outage.
+# Recovery tells the active investigator to verify and leave a receipt.
 printf 'export const lastHeartbeatTs = 1;\n' > "$TMP/heartbeat.ts"
 run_tripwire
+assert_eq 3 "$(line_count "$TMP/investigator.log")"
+assert_eq "recover" "$(tail -1 "$TMP/investigator.log")"
 assert_eq healthy "$(cat "$TMP/tripwire.state")"
+
+# A later outage is a new incident after recovery.
 rm "$TMP/heartbeat.ts"
 run_tripwire
-assert_eq 2 "$(notification_count)"
+assert_eq 4 "$(line_count "$TMP/investigator.log")"
+assert_eq "alert missing" "$(tail -1 "$TMP/investigator.log")"
 
-# A stale heartbeat is its own alert state and is also deduplicated.
+# Sensor health stays truthful while the investigator keeps a failed recovery
+# pending and retries it on the next healthy check.
 printf 'export const lastHeartbeatTs = 1;\n' > "$TMP/heartbeat.ts"
+touch "$TMP/investigator-recover-once.fail"
+run_tripwire
+assert_eq healthy "$(cat "$TMP/tripwire.state")"
+run_tripwire
+assert_eq healthy "$(cat "$TMP/tripwire.state")"
+assert_eq 6 "$(line_count "$TMP/investigator.log")"
+
+# Stale checks include the observed heartbeat age.
 touch -t 200001010000 "$TMP/heartbeat.ts"
 run_tripwire
 run_tripwire
-assert_eq 3 "$(notification_count)"
+assert_eq 8 "$(line_count "$TMP/investigator.log")"
+if ! tail -2 "$TMP/investigator.log" | grep -Eq '^alert stale [0-9]+$'; then
+  echo "stale alert did not include heartbeat age" >&2
+  exit 1
+fi
 assert_eq stale "$(cat "$TMP/tripwire.state")"
 
-# A live, young gateway gets startup grace without a notification.
+# A live, young gateway gets startup grace without dispatching an alert.
 rm "$TMP/heartbeat.ts"
 printf '%s\n' "$$" > "$TMP/gateway.pid"
 printf 'healthy\n' > "$TMP/tripwire.state"
 GATEWAY_TRIPWIRE_THRESHOLD_SECONDS=3600 run_tripwire
-assert_eq 3 "$(notification_count)"
+assert_eq 8 "$(line_count "$TMP/investigator.log")"
 assert_eq starting "$(cat "$TMP/tripwire.state")"
 
-printf 'PASS: gateway tripwire notifications are edge-triggered\n'
+# If the Herdr investigator cannot start, retain one edge-triggered native
+# notification as the failure path rather than silently dropping the outage.
+rm -f "$TMP/gateway.pid"
+printf 'healthy\n' > "$TMP/tripwire.state"
+touch "$TMP/investigator.fail"
+run_tripwire
+run_tripwire
+assert_eq 1 "$(line_count "$TMP/notifications.log")"
+assert_eq missing "$(cat "$TMP/tripwire.state")"
+
+# A failed native notification is retried because its outage state was not committed.
+printf 'healthy\n' > "$TMP/tripwire.state"
+touch "$TMP/osascript.fail"
+run_tripwire
+assert_eq healthy "$(cat "$TMP/tripwire.state")"
+assert_eq 2 "$(line_count "$TMP/notifications.log")"
+rm "$TMP/osascript.fail"
+run_tripwire
+assert_eq missing "$(cat "$TMP/tripwire.state")"
+assert_eq 3 "$(line_count "$TMP/notifications.log")"
+
+printf 'PASS: gateway tripwire dispatches one deduplicated Herdr investigator\n'

packages/gateway/src/gateway-alert-investigator.test.ts

@@ -0,0 +1,435 @@
+import { afterEach, describe, expect, test } from "bun:test";
+import { readFile, rm } from "node:fs/promises";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import {
+  GatewayAlertInvestigator,
+  type CommandResult,
+  type CommandRunner,
+  type GatewayAlertIncident,
+} from "./gateway-alert-investigator";
+
+const roots: string[] = [];
+
+afterEach(async () => {
+  await Promise.all(roots.splice(0).map((root) => rm(root, { recursive: true, force: true })));
+});
+
+type RecordedCommand = {
+  readonly command: string;
+  readonly args: readonly string[];
+};
+
+function commandResult(exitCode = 0, stdout = "{}\n", stderr = ""): CommandResult {
+  return { exitCode, stdout, stderr };
+}
+
+async function harness() {
+  const root = join(tmpdir(), `gateway-alert-investigator-${crypto.randomUUID()}`);
+  roots.push(root);
+  const commands: RecordedCommand[] = [];
+  const workspaces = new Set<string>();
+  const workspaceLabels = new Map<string, string>();
+  const paneByWorkspace = new Map<string, string>();
+  const agents = new Set<string>();
+  let malformedWorkspaceCreateOnce = false;
+  let failRecoveryPromptOnce = false;
+  let herdrReady = true;
+  let launchctlReady = true;
+  let modelReady = true;
+  let modelProbeDelayMs = 0;
+  let workspaceSequence = 0;
+  let now = new Date("2026-09-10T18:00:00.000Z");
+
+  const run: CommandRunner = async (input) => {
+    commands.push({ command: input.command, args: input.args });
+    if (input.command === "/fake/joelclaw") return commandResult();
+    if (input.command === "/fake/launchctl") {
+      if (launchctlReady) {
+        herdrReady = true;
+        return commandResult();
+      }
+      return commandResult(1, "", "launchd unavailable");
+    }
+    if (input.command !== "/fake/herdr") return commandResult(127, "", "unknown command");
+
+    if (input.args[0] === "status") {
+      return herdrReady ? commandResult() : commandResult(1, "", "not ready");
+    }
+    if (input.args[0] === "workspace" && input.args[1] === "get") {
+      return workspaces.has(String(input.args[2]))
+        ? commandResult()
+        : commandResult(1, "", "not found");
+    }
+    if (input.args[0] === "workspace" && input.args[1] === "list") {
+      return commandResult(
+        0,
+        JSON.stringify({
+          result: {
+            workspaces: [...workspaces].map((workspaceId) => ({
+              workspace_id: workspaceId,
+              label: workspaceLabels.get(workspaceId),
+            })),
+          },
+        }),
+      );
+    }
+    if (input.args[0] === "pane" && input.args[1] === "list") {
+      const workspaceId = String(input.args[input.args.indexOf("--workspace") + 1]);
+      const paneId = paneByWorkspace.get(workspaceId);
+      return paneId
+        ? commandResult(0, JSON.stringify({ result: { panes: [{ pane_id: paneId }] } }))
+        : commandResult(1, "", "not found");
+    }
+    if (input.args[0] === "workspace" && input.args[1] === "create") {
+      workspaceSequence += 1;
+      const workspaceId = `w${workspaceSequence}`;
+      const paneId = `${workspaceId}:p1`;
+      const labelIndex = input.args.indexOf("--label");
+      workspaces.add(workspaceId);
+      workspaceLabels.set(workspaceId, String(input.args[labelIndex + 1]));
+      paneByWorkspace.set(workspaceId, paneId);
+      if (malformedWorkspaceCreateOnce) {
+        malformedWorkspaceCreateOnce = false;
+        return commandResult(0, "{}\n");
+      }
+      return commandResult(
+        0,
+        JSON.stringify({
+          result: {
+            workspace: { workspace_id: workspaceId },
+            root_pane: { pane_id: paneId },
+          },
+        }),
+      );
+    }
+    if (input.args[0] === "workspace" && input.args[1] === "rename") {
+      return commandResult();
+    }
+    if (input.args[0] === "agent" && input.args[1] === "get") {
+      return agents.has(String(input.args[2]))
+        ? commandResult()
+        : commandResult(1, "", "not found");
+    }
+    if (input.args[0] === "agent" && input.args[1] === "start") {
+      agents.add(String(input.args[2]));
+      return commandResult();
+    }
+    if (input.args[0] === "agent" && input.args[1] === "prompt") {
+      if (failRecoveryPromptOnce && input.args.join(" ").includes("heartbeat recovered")) {
+        failRecoveryPromptOnce = false;
+        return commandResult(1, "", "prompt failed");
+      }
+      return commandResult();
+    }
+    return commandResult(1, "", `unexpected herdr args: ${input.args.join(" ")}`);
+  };
+
+  const statePath = join(root, "incident.json");
+  const investigator = new GatewayAlertInvestigator({
+    statePath,
+    repoRoot: "/repo",
+    herdrBin: "/fake/herdr",
+    launchctlBin: "/fake/launchctl",
+    joelclawBin: "/fake/joelclaw",
+    retryMs: 300_000,
+    commandRunner: run,
+    modelProbe: async () => {
+      now = new Date(now.getTime() + modelProbeDelayMs);
+      if (!modelReady) throw new Error("dgx-glm model probe timed out");
+    },
+    insideHerdr: false,
+    now: () => now,
+    sleep: async () => undefined,
+  });
+
+  return {
+    investigator,
+    commands,
+    workspaces,
+    agents,
+    setNow(value: Date) {
+      now = value;
+    },
+    setHerdrReady(value: boolean) {
+      herdrReady = value;
+    },
+    setLaunchctlReady(value: boolean) {
+      launchctlReady = value;
+    },
+    setModelReady(value: boolean) {
+      modelReady = value;
+    },
+    setModelProbeDelay(milliseconds: number) {
+      modelProbeDelayMs = milliseconds;
+    },
+    makeNextWorkspaceReceiptMalformed() {
+      malformedWorkspaceCreateOnce = true;
+    },
+    failNextRecoveryPrompt() {
+      failRecoveryPromptOnce = true;
+    },
+    async state(): Promise<GatewayAlertIncident> {
+      return JSON.parse(await readFile(statePath, "utf8")) as GatewayAlertIncident;
+    },
+  };
+}
+
+describe("GatewayAlertInvestigator", () => {
+  test("opens one Herdr workspace and starts the requested DGX GLM Pi agent", async () => {
+    const tested = await harness();
+    const started = await tested.investigator.alert("missing");
+
+    expect(started).toMatchObject({
+      ok: true,
+      action: "started",
+      incident: {
+        phase: "investigating",
+        kind: "missing",
+        workspaceId: "w1",
+        paneId: "w1:p1",
+      },
+    });
+    const start = tested.commands.find(
+      (command) => command.args[0] === "agent" && command.args[1] === "start",
+    );
+    expect(start?.args).toContain("dgx-glm/glm-5.3-flash:high");
+    expect(start?.args).toContain("w1:p1");
+    const prompt = tested.commands.find(
+      (command) => command.args[0] === "agent" && command.args[1] === "prompt",
+    );
+    expect(prompt?.args.join(" ")).toContain("Never start a second gateway");
+  });
+
+  test("deduplicates repeated checks for the same active incident", async () => {
+    const tested = await harness();
+    await tested.investigator.alert("missing");
+    const firstCommandCount = tested.commands.length;
+
+    const existing = await tested.investigator.alert("stale", 1900);
+
+    expect(existing).toMatchObject({
+      ok: true,
+      action: "existing",
+      incident: { phase: "investigating", kind: "stale", workspaceId: "w1" },
+    });
+    expect(
+      tested.commands.filter(
+        (command) => command.args[0] === "workspace" && command.args[1] === "create",
+      ),
+    ).toHaveLength(1);
+    expect(
+      tested.commands.filter(
+        (command) => command.args[0] === "agent" && command.args[1] === "start",
+      ),
+    ).toHaveLength(1);
+    expect(tested.commands.length).toBe(firstCommandCount + 3);
+    expect(
+      tested.commands.find(
+        (command) =>
+          command.args[0] === "workspace" &&
+          command.args[1] === "rename" &&
+          command.args.includes("[jc] gateway heartbeat stale · GLM investigating"),
+      )?.args,
+    ).toContain("[jc] gateway heartbeat stale · GLM investigating");
+  });
+
+  test("marks the workspace review-ready and asks the same agent for a recovery receipt", async () => {
+    const tested = await harness();
+    await tested.investigator.alert("missing");
+    tested.setNow(new Date("2026-09-10T18:05:00.000Z"));
+
+    const recovered = await tested.investigator.recover();
+    const repeated = await tested.investigator.recover();
+
+    expect(recovered).toMatchObject({
+      ok: true,
+      action: "recovered",
+      incident: { phase: "recovered", workspaceId: "w1" },
+    });
+    expect(repeated.action).toBe("idle");
+    expect(
+      tested.commands.find(
+        (command) =>
+          command.args[0] === "workspace" &&
+          command.args[1] === "rename" &&
+          command.args.includes("[jc] gateway recovered · review ready"),
+      )?.args,
+    ).toContain("[jc] gateway recovered · review ready");
+    const recoveryPrompts = tested.commands.filter(
+      (command) =>
+        command.args[0] === "agent" &&
+        command.args[1] === "prompt" &&
+        command.args.join(" ").includes("heartbeat recovered"),
+    );
+    expect(recoveryPrompts).toHaveLength(1);
+  });
+
+  test("retries recovery side effects before recording the incident recovered", async () => {
+    const tested = await harness();
+    await tested.investigator.alert("missing");
+    tested.failNextRecoveryPrompt();
+
+    const blocked = await tested.investigator.recover();
+    const recovered = await tested.investigator.recover();
+
+    expect(blocked).toMatchObject({
+      ok: false,
+      action: "blocked",
+      incident: { phase: "recoveryBlocked", lastError: expect.stringContaining("prompt failed") },
+    });
+    expect(recovered).toMatchObject({
+      ok: true,
+      action: "recovered",
+      incident: { phase: "recovered" },
+    });
+    expect(
+      tested.commands.filter(
+        (command) =>
+          command.args[0] === "agent" &&
+          command.args[1] === "prompt" &&
+          command.args.join(" ").includes("heartbeat recovered"),
+      ),
+    ).toHaveLength(2);
+  });
+
+  test("kickstarts the default Aqua Herdr job before opening an incident", async () => {
+    const tested = await harness();
+    tested.setHerdrReady(false);
+
+    const started = await tested.investigator.alert("missing");
+
+    expect(started.action).toBe("started");
+    const uid = process.getuid?.();
+    if (uid === undefined) throw new Error("test requires process.getuid");
+    expect(tested.commands.find((command) => command.command === "/fake/launchctl")?.args).toEqual([
+      "kickstart",
+      "-k",
+      `gui/${uid}/com.joelclaw.herdr-server`,
+    ]);
+  });
+
+  test("keeps the workspace and retries when the requested DGX model is unavailable", async () => {
+    const tested = await harness();
+    tested.setModelReady(false);
+    tested.setModelProbeDelay(30_000);
+
+    const blocked = await tested.investigator.alert("missing");
+
+    expect(blocked).toMatchObject({
+      ok: false,
+      action: "blocked",
+      incident: {
+        phase: "blocked",
+        workspaceId: "w1",
+        paneId: "w1:p1",
+        lastError: "dgx-glm model probe timed out",
+      },
+    });
+    expect(
+      tested.commands.filter(
+        (command) => command.args[0] === "agent" && command.args[1] === "start",
+      ),
+    ).toHaveLength(0);
+
+    tested.setNow(new Date("2026-09-10T18:05:00.000Z"));
+    tested.setModelReady(true);
+    tested.setModelProbeDelay(0);
+    const retried = await tested.investigator.alert("missing");
+    expect(retried).toMatchObject({
+      ok: true,
+      action: "started",
+      incident: { phase: "investigating", workspaceId: "w1", attempt: 2 },
+    });
+    expect(
+      tested.commands.filter(
+        (command) => command.args[0] === "workspace" && command.args[1] === "create",
+      ),
+    ).toHaveLength(1);
+  });
+
+  test("backs off after a failed default Herdr start", async () => {
+    const tested = await harness();
+    tested.setHerdrReady(false);
+    tested.setLaunchctlReady(false);
+
+    const blocked = await tested.investigator.alert("missing");
+    const commandCount = tested.commands.length;
+    const stillBlocked = await tested.investigator.alert("missing");
+
+    expect(blocked).toMatchObject({
+      ok: false,
+      action: "blocked",
+      incident: { phase: "blocked", attempt: 1 },
+    });
+    expect(blocked.incident?.lastError).toContain("Could not start default Herdr");
+    expect(stillBlocked.action).toBe("blocked");
+    expect(tested.commands).toHaveLength(commandCount);
+  });
+
+  test("restarts a dead investigator in the existing incident workspace", async () => {
+    const tested = await harness();
+    const started = await tested.investigator.alert("missing");
+    tested.agents.clear();
+
+    const restarted = await tested.investigator.alert("missing");
+
+    expect(restarted).toMatchObject({
+      ok: true,
+      action: "started",
+      incident: { phase: "investigating", workspaceId: started.incident?.workspaceId },
+    });
+    expect(
+      tested.commands.filter(
+        (command) => command.args[0] === "workspace" && command.args[1] === "create",
+      ),
+    ).toHaveLength(1);
+    expect(
+      tested.commands.filter(
+        (command) => command.args[0] === "agent" && command.args[1] === "start",
+      ),
+    ).toHaveLength(2);
+  });
+
+  test("discovers a created workspace after an ambiguous create receipt", async () => {
+    const tested = await harness();
+    tested.makeNextWorkspaceReceiptMalformed();
+
+    const blocked = await tested.investigator.alert("missing");
+    expect(blocked).toMatchObject({ ok: false, action: "blocked" });
+
+    tested.setNow(new Date("2026-09-10T18:05:00.000Z"));
+    const recovered = await tested.investigator.alert("missing");
+
+    expect(recovered).toMatchObject({
+      ok: true,
+      action: "started",
+      incident: { phase: "investigating", workspaceId: "w1", paneId: "w1:p1" },
+    });
+    expect(
+      tested.commands.filter(
+        (command) => command.args[0] === "workspace" && command.args[1] === "create",
+      ),
+    ).toHaveLength(1);
+  });
+
+  test("opens a replacement workspace when the active incident workspace disappeared", async () => {
+    const tested = await harness();
+    const started = await tested.investigator.alert("missing");
+    tested.workspaces.delete(String(started.incident?.workspaceId));
+    tested.agents.clear();
+
+    const replaced = await tested.investigator.alert("missing");
+
+    expect(replaced).toMatchObject({
+      ok: true,
+      action: "started",
+      incident: { phase: "investigating", workspaceId: "w2", paneId: "w2:p1" },
+    });
+    expect(
+      tested.commands.filter(
+        (command) => command.args[0] === "workspace" && command.args[1] === "create",
+      ),
+    ).toHaveLength(2);
+  });
+});

packages/gateway/src/gateway-alert-investigator.ts

@@ -0,0 +1,945 @@
+import { mkdir, readFile, rename, rm, stat, utimes, writeFile } from "node:fs/promises";
+import { randomUUID } from "node:crypto";
+import { homedir } from "node:os";
+import { dirname, resolve } from "node:path";
+import { createActor, createMachine } from "xstate";
+
+export type GatewayAlertKind = "missing" | "stale";
+export type GatewayAlertPhase =
+  | "idle"
+  | "ensuringHerdr"
+  | "openingWorkspace"
+  | "startingAgent"
+  | "promptingAgent"
+  | "investigating"
+  | "blocked"
+  | "recovering"
+  | "recoveryBlocked"
+  | "recovered";
+
+export type GatewayAlertIncident = {
+  readonly version: 1;
+  readonly phase: GatewayAlertPhase;
+  readonly incidentId: string;
+  readonly kind: GatewayAlertKind;
+  readonly ageSeconds?: number;
+  readonly createdAt: string;
+  readonly updatedAt: string;
+  readonly attempt: number;
+  readonly attemptStartedAt: string;
+  readonly retryAfter?: string;
+  readonly workspaceId?: string;
+  readonly paneId?: string;
+  readonly agentName?: string;
+  readonly lastError?: string;
+  readonly recoveredAt?: string;
+};
+
+type GatewayAlertEvent =
+  | { readonly type: "ALERT" }
+  | { readonly type: "HERDR_READY" }
+  | { readonly type: "WORKSPACE_READY" }
+  | { readonly type: "AGENT_READY" }
+  | { readonly type: "PROMPT_ACCEPTED" }
+  | { readonly type: "FAILED" }
+  | { readonly type: "RETRY" }
+  | { readonly type: "WORKSPACE_LOST" }
+  | { readonly type: "AGENT_LOST" }
+  | { readonly type: "BEGIN_RECOVERY" }
+  | { readonly type: "RECOVERY_DELIVERED" }
+  | { readonly type: "RECOVERY_FAILED" }
+  | { readonly type: "RETRY_RECOVERY" };
+
+export const gatewayAlertInvestigatorMachine = createMachine({
+  id: "gatewayAlertInvestigator",
+  initial: "idle",
+  states: {
+    idle: { on: { ALERT: "ensuringHerdr" } },
+    ensuringHerdr: {
+      on: { HERDR_READY: "openingWorkspace", FAILED: "blocked", BEGIN_RECOVERY: "recovering" },
+    },
+    openingWorkspace: {
+      on: { WORKSPACE_READY: "startingAgent", FAILED: "blocked", BEGIN_RECOVERY: "recovering" },
+    },
+    startingAgent: {
+      on: { AGENT_READY: "promptingAgent", FAILED: "blocked", BEGIN_RECOVERY: "recovering" },
+    },
+    promptingAgent: {
+      on: { PROMPT_ACCEPTED: "investigating", FAILED: "blocked", BEGIN_RECOVERY: "recovering" },
+    },
+    investigating: {
+      on: {
+        ALERT: "investigating",
+        WORKSPACE_LOST: "ensuringHerdr",
+        AGENT_LOST: "startingAgent",
+        BEGIN_RECOVERY: "recovering",
+      },
+    },
+    blocked: {
+      on: { ALERT: "blocked", RETRY: "ensuringHerdr", BEGIN_RECOVERY: "recovering" },
+    },
+    recovering: {
+      on: {
+        RECOVERY_DELIVERED: "recovered",
+        RECOVERY_FAILED: "recoveryBlocked",
+        ALERT: "ensuringHerdr",
+      },
+    },
+    recoveryBlocked: {
+      on: { RETRY_RECOVERY: "recovering", ALERT: "ensuringHerdr" },
+    },
+    recovered: { on: { ALERT: "ensuringHerdr" } },
+  },
+});
+
+export type CommandResult = {
+  readonly exitCode: number;
+  readonly stdout: string;
+  readonly stderr: string;
+};
+
+export type CommandRunner = (input: {
+  readonly command: string;
+  readonly args: readonly string[];
+  readonly cwd?: string;
+  readonly env?: Readonly<Record<string, string | undefined>>;
+  readonly timeoutMs?: number;
+}) => Promise<CommandResult>;
+
+export type GatewayAlertInvestigatorOptions = {
+  readonly statePath?: string;
+  readonly repoRoot?: string;
+  readonly herdrBin?: string;
+  readonly launchctlBin?: string;
+  readonly joelclawBin?: string;
+  readonly model?: string;
+  readonly modelsPath?: string;
+  readonly retryMs?: number;
+  readonly commandRunner?: CommandRunner;
+  readonly modelProbe?: (model: string) => Promise<void>;
+  readonly insideHerdr?: boolean;
+  readonly now?: () => Date;
+  readonly sleep?: (milliseconds: number) => Promise<void>;
+};
+
+export type GatewayAlertResult = {
+  readonly ok: boolean;
+  readonly action: "started" | "existing" | "blocked" | "recovered" | "idle";
+  readonly incident?: GatewayAlertIncident;
+};
+
+const DEFAULT_STATE_PATH = resolve(
+  homedir(),
+  ".local/state/joelclaw/gateway-alert-investigator.json",
+);
+const DEFAULT_REPO_ROOT = resolve(homedir(), "Code/joelhooks/joelclaw");
+const DEFAULT_MODEL = "dgx-glm/glm-5.3-flash:high";
+const DEFAULT_MODELS_PATH = resolve(homedir(), ".pi/agent/models.json");
+const DEFAULT_RETRY_MS = 5 * 60_000;
+const LOCK_STALE_MS = 2 * 60_000;
+const LOCK_HEARTBEAT_MS = 10_000;
+
+function transitionIncident(
+  incident: GatewayAlertIncident,
+  event: GatewayAlertEvent,
+  patch: Partial<GatewayAlertIncident> = {},
+): GatewayAlertIncident {
+  const actor = createActor(gatewayAlertInvestigatorMachine, {
+    snapshot: gatewayAlertInvestigatorMachine.resolveState({
+      value: incident.phase,
+      context: {},
+    }),
+  }).start();
+  actor.send(event);
+  const phase = String(actor.getSnapshot().value) as GatewayAlertPhase;
+  actor.stop();
+  return { ...incident, ...patch, phase };
+}
+
+function cleanDefaultHerdrEnvironment(): Record<string, string | undefined> {
+  const env = { ...process.env };
+  delete env.HERDR_SESSION;
+  delete env.HERDR_SOCKET;
+  return env;
+}
+
+async function defaultCommandRunner(input: {
+  readonly command: string;
+  readonly args: readonly string[];
+  readonly cwd?: string;
+  readonly env?: Readonly<Record<string, string | undefined>>;
+  readonly timeoutMs?: number;
+}): Promise<CommandResult> {
+  const processHandle = Bun.spawn([input.command, ...input.args], {
+    cwd: input.cwd,
+    env: input.env,
+    stdout: "pipe",
+    stderr: "pipe",
+  });
+  let timeout: ReturnType<typeof setTimeout> | undefined;
+  const timeoutPromise = new Promise<never>((_, reject) => {
+    timeout = setTimeout(() => {
+      processHandle.kill();
+      reject(new Error(`Command timed out: ${input.command}`));
+    }, input.timeoutMs ?? 30_000);
+  });
+  try {
+    const exitCode = await Promise.race([processHandle.exited, timeoutPromise]);
+    const [stdout, stderr] = await Promise.all([
+      new Response(processHandle.stdout).text(),
+      new Response(processHandle.stderr).text(),
+    ]);
+    return { exitCode, stdout, stderr };
+  } finally {
+    if (timeout) clearTimeout(timeout);
+  }
+}
+
+async function probeConfiguredModel(model: string, modelsPath: string): Promise<void> {
+  const slash = model.indexOf("/");
+  if (slash < 1) throw new Error(`Model has no provider prefix: ${model}`);
+  const providerId = model.slice(0, slash);
+  const modelId = model.slice(slash + 1).replace(/:(off|minimal|low|medium|high|xhigh|max)$/, "");
+  const config = JSON.parse(await readFile(modelsPath, "utf8")) as {
+    readonly providers?: Readonly<
+      Record<
+        string,
+        {
+          readonly baseUrl?: string;
+          readonly apiKey?: string;
+        }
+      >
+    >;
+  };
+  const provider = config.providers?.[providerId];
+  if (!provider?.baseUrl) throw new Error(`No base URL configured for ${providerId}`);
+
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), 3_000);
+  try {
+    const response = await fetch(`${provider.baseUrl.replace(/\/$/, "")}/models`, {
+      headers: provider.apiKey ? { Authorization: `Bearer ${provider.apiKey}` } : undefined,
+      signal: controller.signal,
+    });
+    if (!response.ok) {
+      throw new Error(`${providerId} model probe returned HTTP ${response.status}`);
+    }
+    const body = (await response.json()) as {
+      readonly data?: ReadonlyArray<{ readonly id?: string }>;
+    };
+    if (!body.data?.some((candidate) => candidate.id === modelId)) {
+      throw new Error(`${providerId} did not advertise ${modelId}`);
+    }
+  } catch (error) {
+    if (error instanceof Error && error.name === "AbortError") {
+      throw new Error(`${providerId} model probe timed out`);
+    }
+    throw error;
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
+function parseHerdrResult(stdout: string): Record<string, unknown> {
+  const parsed = JSON.parse(stdout) as { readonly result?: Record<string, unknown> };
+  if (!parsed.result || typeof parsed.result !== "object") {
+    throw new Error("Herdr returned no result object");
+  }
+  return parsed.result;
+}
+
+async function readIncident(path: string): Promise<GatewayAlertIncident | undefined> {
+  try {
+    const parsed = JSON.parse(await readFile(path, "utf8")) as GatewayAlertIncident;
+    if (parsed.version !== 1 || typeof parsed.phase !== "string") {
+      throw new Error(`Unsupported gateway alert state at ${path}`);
+    }
+    return parsed;
+  } catch (error) {
+    if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined;
+    throw error;
+  }
+}
+
+async function writeIncident(path: string, incident: GatewayAlertIncident): Promise<void> {
+  await mkdir(dirname(path), { recursive: true });
+  const temporaryPath = `${path}.${process.pid}.tmp`;
+  await writeFile(temporaryPath, `${JSON.stringify(incident, null, 2)}\n`, {
+    encoding: "utf8",
+    mode: 0o600,
+  });
+  await rename(temporaryPath, path);
+}
+
+async function wait(milliseconds: number): Promise<void> {
+  await new Promise((resolvePromise) => setTimeout(resolvePromise, milliseconds));
+}
+
+async function lockAgeMs(lockPath: string): Promise<number> {
+  try {
+    const owner = JSON.parse(await readFile(resolve(lockPath, "owner.json"), "utf8")) as {
+      readonly heartbeatFile?: string;
+    };
+    if (owner.heartbeatFile?.match(/^heartbeat-[0-9a-f-]+$/)) {
+      return Date.now() - (await stat(resolve(lockPath, owner.heartbeatFile))).mtimeMs;
+    }
+  } catch (error) {
+    if ((error as NodeJS.ErrnoException).code === "ENOENT") {
+      return Date.now() - (await stat(lockPath)).mtimeMs;
+    }
+  }
+  return Date.now() - (await stat(lockPath)).mtimeMs;
+}
+
+async function withFileLock<T>(path: string, run: () => Promise<T>): Promise<T> {
+  const lockPath = `${path}.lock`;
+  await mkdir(dirname(lockPath), { recursive: true });
+  const startedAt = Date.now();
+  let heartbeatPath: string | undefined;
+  while (true) {
+    try {
+      await mkdir(lockPath, { mode: 0o700 });
+      const heartbeatFile = `heartbeat-${randomUUID()}`;
+      heartbeatPath = resolve(lockPath, heartbeatFile);
+      try {
+        await writeFile(heartbeatPath, "", { encoding: "utf8", mode: 0o600 });
+        await writeFile(
+          resolve(lockPath, "owner.json"),
+          `${JSON.stringify({
+            pid: process.pid,
+            createdAt: new Date().toISOString(),
+            heartbeatFile,
+          })}\n`,
+          { encoding: "utf8", mode: 0o600 },
+        );
+      } catch (error) {
+        await rm(lockPath, { recursive: true, force: true });
+        throw error;
+      }
+      break;
+    } catch (error) {
+      if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error;
+      try {
+        if ((await lockAgeMs(lockPath)) > LOCK_STALE_MS) {
+          await rm(lockPath, { recursive: true, force: true });
+          continue;
+        }
+      } catch (statError) {
+        if ((statError as NodeJS.ErrnoException).code === "ENOENT") continue;
+        throw statError;
+      }
+      if (Date.now() - startedAt > 10_000) {
+        throw new Error(`Timed out acquiring gateway alert lock ${lockPath}`);
+      }
+      await wait(25);
+    }
+  }
+
+  const heartbeat = setInterval(() => {
+    if (!heartbeatPath) return;
+    const now = new Date();
+    void utimes(heartbeatPath, now, now).catch(() => undefined);
+  }, LOCK_HEARTBEAT_MS);
+  heartbeat.unref();
+  try {
+    return await run();
+  } finally {
+    clearInterval(heartbeat);
+    await rm(lockPath, { recursive: true, force: true });
+  }
+}
+
+function incidentName(incidentId: string): string {
+  return `gateway_alert_${incidentId.replaceAll("-", "").slice(0, 12)}`;
+}
+
+function workspaceLabel(kind: GatewayAlertKind): string {
+  const problem = kind === "missing" ? "heartbeat missing" : "heartbeat stale";
+  return `[jc] gateway ${problem} · GLM investigating`;
+}
+
+function incidentWorkspaceLabel(incident: GatewayAlertIncident): string {
+  return `${workspaceLabel(incident.kind)} · ${incident.incidentId.slice(0, 8)}`;
+}
+
+function investigationPrompt(incident: GatewayAlertIncident): string {
+  const age = incident.ageSeconds === undefined ? "unknown" : `${incident.ageSeconds} seconds`;
+  return `The local gateway tripwire detected a ${incident.kind} heartbeat at ${incident.createdAt}. Heartbeat age: ${age}.
+
+Own this incident. Diagnose it and make safe, reversible repairs. Use the normal project and fleet instructions already loaded in this Pi session.
+
+Start with the supported gateway surface:
+- joelclaw gateway doctor --json
+- joelclaw gateway status
+
+Hard boundaries:
+- Never start a second gateway, Telegram poller, Slack socket, Discord listener, or comms transport.
+- Use joelclaw gateway commands for gateway lifecycle changes. Do not manipulate its launchd job directly.
+- Do not send a live canary or any outward message without Joel's explicit approval.
+- Do not print secrets or credentials.
+- Do not use broad process-name kills.
+- Preserve dirty work and shared history.
+
+Find the root cause. Repair it when the action is safe and reversible. Verify the heartbeat, readiness file, gateway doctor, and the dependency that failed. Keep this Herdr workspace label current. Finish with root cause, changes, verification, and remaining blockers.`;
+}
+
+function recoveryPrompt(recoveredAt: string): string {
+  return `The gateway heartbeat recovered at ${recoveredAt}. Verify current health with the supported gateway commands. Summarize the root cause, any changes you made, and the final checks. Leave the workspace ready for review.`;
+}
+
+export class GatewayAlertInvestigator {
+  private readonly statePath: string;
+  private readonly repoRoot: string;
+  private readonly herdrBin: string;
+  private readonly launchctlBin: string;
+  private readonly joelclawBin: string;
+  private readonly model: string;
+  private readonly retryMs: number;
+  private readonly runCommand: CommandRunner;
+  private readonly probeModel: (model: string) => Promise<void>;
+  private readonly insideHerdr: boolean;
+  private readonly now: () => Date;
+  private readonly sleep: (milliseconds: number) => Promise<void>;
+  private readonly herdrEnv: Readonly<Record<string, string | undefined>>;
+
+  constructor(options: GatewayAlertInvestigatorOptions = {}) {
+    this.statePath = options.statePath ?? DEFAULT_STATE_PATH;
+    this.repoRoot = options.repoRoot ?? DEFAULT_REPO_ROOT;
+    this.herdrBin = options.herdrBin ?? resolve(homedir(), ".local/bin/herdr");
+    this.launchctlBin = options.launchctlBin ?? "/bin/launchctl";
+    this.joelclawBin = options.joelclawBin ?? resolve(homedir(), ".local/bin/joelclaw");
+    this.model = options.model ?? DEFAULT_MODEL;
+    this.retryMs = options.retryMs ?? DEFAULT_RETRY_MS;
+    this.runCommand = options.commandRunner ?? defaultCommandRunner;
+    const modelsPath = options.modelsPath ?? DEFAULT_MODELS_PATH;
+    this.probeModel = options.modelProbe ?? ((model) => probeConfiguredModel(model, modelsPath));
+    this.insideHerdr = options.insideHerdr ?? Boolean(process.env.HERDR_PANE_ID);
+    this.now = options.now ?? (() => new Date());
+    this.sleep = options.sleep ?? wait;
+    this.herdrEnv = cleanDefaultHerdrEnvironment();
+  }
+
+  async alert(kind: GatewayAlertKind, ageSeconds?: number): Promise<GatewayAlertResult> {
+    return withFileLock(this.statePath, async () => {
+      const existing = await readIncident(this.statePath);
+      const now = this.now();
+      let incident: GatewayAlertIncident;
+
+      if (!existing || existing.phase === "idle" || existing.phase === "recovered") {
+        const incidentId = crypto.randomUUID();
+        const initial: GatewayAlertIncident = {
+          version: 1,
+          phase: existing?.phase ?? "idle",
+          incidentId,
+          kind,
+          ...(ageSeconds === undefined ? {} : { ageSeconds }),
+          createdAt: now.toISOString(),
+          updatedAt: now.toISOString(),
+          attempt: 1,
+          attemptStartedAt: now.toISOString(),
+          agentName: incidentName(incidentId),
+        };
+        incident = transitionIncident(
+          initial,
+          { type: "ALERT" },
+          {
+            workspaceId: undefined,
+            paneId: undefined,
+            lastError: undefined,
+            retryAfter: undefined,
+            recoveredAt: undefined,
+          },
+        );
+        await writeIncident(this.statePath, incident);
+      } else if (existing.phase === "investigating") {
+        incident = transitionIncident(
+          existing,
+          { type: "ALERT" },
+          {
+            kind,
+            ...(ageSeconds === undefined ? {} : { ageSeconds }),
+            updatedAt: now.toISOString(),
+          },
+        );
+        if (incident.workspaceId && (await this.workspaceExists(incident.workspaceId))) {
+          if (existing.kind !== kind) {
+            await this.runHerdr([
+              "workspace",
+              "rename",
+              incident.workspaceId,
+              workspaceLabel(kind),
+            ]).catch(() => undefined);
+          }
+          if (incident.agentName && (await this.agentExists(incident.agentName))) {
+            await writeIncident(this.statePath, incident);
+            return { ok: true, action: "existing", incident };
+          }
+          incident = transitionIncident(
+            incident,
+            { type: "AGENT_LOST" },
+            {
+              agentName: incidentName(crypto.randomUUID()),
+              updatedAt: now.toISOString(),
+            },
+          );
+          await writeIncident(this.statePath, incident);
+          return this.advance(incident);
+        }
+        incident = transitionIncident(
+          incident,
+          { type: "WORKSPACE_LOST" },
+          { updatedAt: now.toISOString() },
+        );
+        await writeIncident(this.statePath, incident);
+      } else if (existing.phase === "recovering" || existing.phase === "recoveryBlocked") {
+        incident = transitionIncident(
+          existing,
+          { type: "ALERT" },
+          {
+            kind,
+            ...(ageSeconds === undefined ? {} : { ageSeconds }),
+            attemptStartedAt: now.toISOString(),
+            updatedAt: now.toISOString(),
+            attempt: existing.attempt + 1,
+            lastError: undefined,
+          },
+        );
+        await writeIncident(this.statePath, incident);
+      } else if (existing.phase === "blocked") {
+        const retryAt = existing.retryAfter ? Date.parse(existing.retryAfter) : 0;
+        if (retryAt > now.getTime()) {
+          return { ok: false, action: "blocked", incident: existing };
+        }
+        incident = transitionIncident(
+          existing,
+          { type: "RETRY" },
+          {
+            kind,
+            ...(ageSeconds === undefined ? {} : { ageSeconds }),
+            updatedAt: now.toISOString(),
+            attemptStartedAt: now.toISOString(),
+            attempt: existing.attempt + 1,
+            retryAfter: undefined,
+            lastError: undefined,
+          },
+        );
+        await writeIncident(this.statePath, incident);
+      } else {
+        incident = {
+          ...existing,
+          kind,
+          ...(ageSeconds === undefined ? {} : { ageSeconds }),
+          updatedAt: now.toISOString(),
+        };
+      }
+
+      return this.advance(incident);
+    });
+  }
+
+  async recover(): Promise<GatewayAlertResult> {
+    return withFileLock(this.statePath, async () => {
+      const existing = await readIncident(this.statePath);
+      if (!existing || existing.phase === "idle" || existing.phase === "recovered") {
+        return { ok: true, action: "idle", incident: existing };
+      }
+
+      const recoveryStartedAt = this.now().toISOString();
+      let recovering =
+        existing.phase === "recovering"
+          ? existing
+          : transitionIncident(
+              existing,
+              { type: existing.phase === "recoveryBlocked" ? "RETRY_RECOVERY" : "BEGIN_RECOVERY" },
+              {
+                updatedAt: recoveryStartedAt,
+                retryAfter: undefined,
+                lastError: undefined,
+              },
+            );
+      await writeIncident(this.statePath, recovering);
+
+      try {
+        await this.ensureDefaultHerdr();
+        const workspace = await this.ensureWorkspace(recovering);
+        recovering = {
+          ...recovering,
+          workspaceId: workspace.workspaceId,
+          paneId: workspace.paneId,
+          ...(workspace.created && recovering.workspaceId
+            ? { agentName: incidentName(crypto.randomUUID()) }
+            : {}),
+          updatedAt: this.now().toISOString(),
+        };
+        await writeIncident(this.statePath, recovering);
+        await this.probeModel(this.model);
+        await this.ensureAgent(recovering);
+
+        const renamed = await this.runHerdr([
+          "workspace",
+          "rename",
+          workspace.workspaceId,
+          "[jc] gateway recovered · review ready",
+        ]);
+        if (renamed.exitCode !== 0) {
+          throw new Error(
+            `Could not mark recovered workspace review-ready: ${renamed.stderr.trim()}`,
+          );
+        }
+        if (!recovering.agentName) throw new Error("Recovered incident has no Herdr agent name");
+        const prompted = await this.runHerdr([
+          "agent",
+          "prompt",
+          recovering.agentName,
+          recoveryPrompt(recoveryStartedAt),
+        ]);
+        if (prompted.exitCode !== 0) {
+          throw new Error(`Could not request gateway recovery receipt: ${prompted.stderr.trim()}`);
+        }
+
+        const recovered = transitionIncident(
+          recovering,
+          { type: "RECOVERY_DELIVERED" },
+          {
+            updatedAt: this.now().toISOString(),
+            recoveredAt: recoveryStartedAt,
+            lastError: undefined,
+          },
+        );
+        await writeIncident(this.statePath, recovered);
+        await this.emitTelemetry("gateway_alert_investigator.recovered", recovered, true);
+        return { ok: true, action: "recovered", incident: recovered };
+      } catch (error) {
+        const message = error instanceof Error ? error.message : String(error);
+        const blocked = transitionIncident(
+          recovering,
+          { type: "RECOVERY_FAILED" },
+          {
+            updatedAt: this.now().toISOString(),
+            lastError: message,
+          },
+        );
+        await writeIncident(this.statePath, blocked);
+        await this.emitTelemetry(
+          "gateway_alert_investigator.recovery_failed",
+          blocked,
+          false,
+          message,
+        );
+        return { ok: false, action: "blocked", incident: blocked };
+      }
+    });
+  }
+
+  private async advance(start: GatewayAlertIncident): Promise<GatewayAlertResult> {
+    let incident = start;
+    try {
+      if (incident.phase === "ensuringHerdr") {
+        await this.ensureDefaultHerdr();
+        incident = transitionIncident(
+          incident,
+          { type: "HERDR_READY" },
+          {
+            updatedAt: this.now().toISOString(),
+          },
+        );
+        await writeIncident(this.statePath, incident);
+      }
+
+      if (incident.phase === "openingWorkspace") {
+        const workspace = await this.ensureWorkspace(incident);
+        incident = transitionIncident(
+          incident,
+          { type: "WORKSPACE_READY" },
+          {
+            workspaceId: workspace.workspaceId,
+            paneId: workspace.paneId,
+            ...(workspace.created && incident.workspaceId
+              ? { agentName: incidentName(crypto.randomUUID()) }
+              : {}),
+            updatedAt: this.now().toISOString(),
+          },
+        );
+        await writeIncident(this.statePath, incident);
+      }
+
+      if (incident.phase === "startingAgent") {
+        await this.probeModel(this.model);
+        await this.ensureAgent(incident);
+        incident = transitionIncident(
+          incident,
+          { type: "AGENT_READY" },
+          {
+            updatedAt: this.now().toISOString(),
+          },
+        );
+        await writeIncident(this.statePath, incident);
+      }
+
+      if (incident.phase === "promptingAgent") {
+        if (!incident.agentName) throw new Error("Incident has no Herdr agent name");
+        const prompted = await this.runHerdr(
+          ["agent", "prompt", incident.agentName, investigationPrompt(incident)],
+          15_000,
+        );
+        if (prompted.exitCode !== 0) {
+          throw new Error(`Herdr rejected investigator prompt: ${prompted.stderr.trim()}`);
+        }
+        incident = transitionIncident(
+          incident,
+          { type: "PROMPT_ACCEPTED" },
+          {
+            updatedAt: this.now().toISOString(),
+          },
+        );
+        await writeIncident(this.statePath, incident);
+        if (incident.workspaceId) {
+          await this.runHerdr([
+            "workspace",
+            "rename",
+            incident.workspaceId,
+            workspaceLabel(incident.kind),
+          ]).catch(() => undefined);
+        }
+        await this.emitTelemetry("gateway_alert_investigator.started", incident, true);
+        return { ok: true, action: "started", incident };
+      }
+
+      if (incident.phase === "investigating") {
+        return { ok: true, action: "existing", incident };
+      }
+
+      throw new Error(`Cannot advance gateway alert incident from ${incident.phase}`);
+    } catch (error) {
+      const message = error instanceof Error ? error.message : String(error);
+      const failed = transitionIncident(
+        incident,
+        { type: "FAILED" },
+        {
+          updatedAt: this.now().toISOString(),
+          retryAfter: new Date(Date.parse(incident.attemptStartedAt) + this.retryMs).toISOString(),
+          lastError: message,
+        },
+      );
+      await writeIncident(this.statePath, failed);
+      if (failed.workspaceId) {
+        await this.runHerdr([
+          "workspace",
+          "rename",
+          failed.workspaceId,
+          "[jc] gateway alert · investigator blocked",
+        ]).catch(() => undefined);
+      }
+      await this.emitTelemetry("gateway_alert_investigator.failed", failed, false, message);
+      return { ok: false, action: "blocked", incident: failed };
+    }
+  }
+
+  private async ensureDefaultHerdr(): Promise<void> {
+    const alreadyReady = await this.runHerdr(["status"], 5_000)
+      .then((result) => result.exitCode === 0)
+      .catch(() => false);
+    if (alreadyReady) return;
+
+    if (this.insideHerdr) {
+      throw new Error("Refusing to restart the default Herdr from one of its own panes");
+    }
+    const uid = typeof process.getuid === "function" ? process.getuid() : undefined;
+    if (uid === undefined)
+      throw new Error("Cannot resolve uid for the default Herdr launch domain");
+    const kicked = await this.runCommand({
+      command: this.launchctlBin,
+      args: ["kickstart", "-k", `gui/${uid}/com.joelclaw.herdr-server`],
+      timeoutMs: 10_000,
+    });
+    if (kicked.exitCode !== 0) {
+      throw new Error(`Could not start default Herdr: ${kicked.stderr.trim()}`);
+    }
+
+    for (let attempt = 0; attempt < 20; attempt += 1) {
+      await this.sleep(500);
+      const ready = await this.runHerdr(["status"], 5_000)
+        .then((result) => result.exitCode === 0)
+        .catch(() => false);
+      if (ready) return;
+    }
+    throw new Error("Default Herdr did not become ready after launchd kickstart");
+  }
+
+  private async workspaceExists(workspaceId: string): Promise<boolean> {
+    return this.runHerdr(["workspace", "get", workspaceId], 5_000)
+      .then((result) => result.exitCode === 0)
+      .catch(() => false);
+  }
+
+  private async ensureWorkspace(incident: GatewayAlertIncident): Promise<{
+    readonly workspaceId: string;
+    readonly paneId: string;
+    readonly created: boolean;
+  }> {
+    if (
+      incident.workspaceId &&
+      incident.paneId &&
+      (await this.workspaceExists(incident.workspaceId))
+    ) {
+      return {
+        workspaceId: incident.workspaceId,
+        paneId: incident.paneId,
+        created: false,
+      };
+    }
+
+    const discovered = await this.discoverWorkspace(incidentWorkspaceLabel(incident));
+    if (discovered) return { ...discovered, created: false };
+
+    const created = await this.runHerdr([
+      "workspace",
+      "create",
+      "--cwd",
+      this.repoRoot,
+      "--label",
+      incidentWorkspaceLabel(incident),
+      "--no-focus",
+    ]);
+    if (created.exitCode !== 0) {
+      throw new Error(`Could not create investigator workspace: ${created.stderr.trim()}`);
+    }
+    const result = parseHerdrResult(created.stdout);
+    const workspace = result.workspace as { readonly workspace_id?: string } | undefined;
+    const rootPane = result.root_pane as { readonly pane_id?: string } | undefined;
+    if (!workspace?.workspace_id || !rootPane?.pane_id) {
+      throw new Error("Herdr workspace receipt omitted workspace or root pane id");
+    }
+    return {
+      workspaceId: workspace.workspace_id,
+      paneId: rootPane.pane_id,
+      created: true,
+    };
+  }
+
+  private async discoverWorkspace(
+    label: string,
+  ): Promise<{ readonly workspaceId: string; readonly paneId: string } | undefined> {
+    const listed = await this.runHerdr(["workspace", "list"], 5_000);
+    if (listed.exitCode !== 0) return undefined;
+    const listResult = parseHerdrResult(listed.stdout);
+    const workspaces = listResult.workspaces as
+      | ReadonlyArray<{ readonly workspace_id?: string; readonly label?: string }>
+      | undefined;
+    const workspaceId = workspaces?.find((workspace) => workspace.label === label)?.workspace_id;
+    if (!workspaceId) return undefined;
+
+    const paneList = await this.runHerdr(["pane", "list", "--workspace", workspaceId], 5_000);
+    if (paneList.exitCode !== 0) return undefined;
+    const paneResult = parseHerdrResult(paneList.stdout);
+    const panes = paneResult.panes as ReadonlyArray<{ readonly pane_id?: string }> | undefined;
+    const paneId = panes?.[0]?.pane_id;
+    return paneId ? { workspaceId, paneId } : undefined;
+  }
+
+  private async agentExists(agentName: string): Promise<boolean> {
+    return this.runHerdr(["agent", "get", agentName], 5_000)
+      .then((result) => result.exitCode === 0)
+      .catch(() => false);
+  }
+
+  private async ensureAgent(incident: GatewayAlertIncident): Promise<void> {
+    if (!incident.agentName || !incident.paneId) {
+      throw new Error("Incident has no agent name or pane id");
+    }
+    if (await this.agentExists(incident.agentName)) return;
+    const started = await this.runHerdr(
+      [
+        "agent",
+        "start",
+        incident.agentName,
+        "--kind",
+        "pi",
+        "--pane",
+        incident.paneId,
+        "--timeout",
+        "60000",
+        "--",
+        "--model",
+        this.model,
+        "--name",
+        "🩺 Gateway Alert Investigator",
+      ],
+      70_000,
+    );
+    if (started.exitCode !== 0) {
+      throw new Error(`Could not start DGX GLM investigator: ${started.stderr.trim()}`);
+    }
+  }
+
+  private runHerdr(args: readonly string[], timeoutMs = 30_000): Promise<CommandResult> {
+    return this.runCommand({
+      command: this.herdrBin,
+      args,
+      cwd: this.repoRoot,
+      env: this.herdrEnv,
+      timeoutMs,
+    });
+  }
+
+  private async emitTelemetry(
+    action: string,
+    incident: GatewayAlertIncident,
+    success: boolean,
+    error?: string,
+  ): Promise<void> {
+    const metadata = JSON.stringify({
+      incidentId: incident.incidentId,
+      kind: incident.kind,
+      phase: incident.phase,
+      attempt: incident.attempt,
+      workspaceId: incident.workspaceId,
+      paneId: incident.paneId,
+      model: this.model,
+    });
+    const args = [
+      "otel",
+      "emit",
+      action,
+      "--source",
+      "gateway-tripwire",
+      "--component",
+      "gateway-alert-investigator",
+      "--level",
+      success ? "info" : "error",
+      "--success",
+      String(success),
+      "--metadata",
+      metadata,
+    ];
+    if (error) args.push("--error", error);
+    await this.runCommand({ command: this.joelclawBin, args, timeoutMs: 10_000 }).catch(
+      () => undefined,
+    );
+  }
+}
+
+function usage(): never {
+  console.error("usage: gateway-alert-investigator <alert missing|stale [age-seconds]|recover>");
+  process.exit(2);
+}
+
+async function main(args: readonly string[]): Promise<void> {
+  const investigator = new GatewayAlertInvestigator();
+  let result: GatewayAlertResult;
+  if (args[0] === "alert" && (args[1] === "missing" || args[1] === "stale")) {
+    const ageSeconds = args[2] === undefined ? undefined : Number.parseInt(args[2], 10);
+    if (ageSeconds !== undefined && (!Number.isSafeInteger(ageSeconds) || ageSeconds < 0)) usage();
+    result = await investigator.alert(args[1], ageSeconds);
+  } else if (args[0] === "recover") {
+    result = await investigator.recover();
+  } else {
+    usage();
+  }
+  console.log(JSON.stringify(result));
+  if (!result.ok) process.exitCode = 1;
+}
+
+if (import.meta.main) {
+  await main(process.argv.slice(2));
+}

skills/gateway/SKILL.md

@@ -108,6 +108,19 @@ Idle maintenance is autonomous for time-based pressure:
 - age-triggered rotation can also happen from the watchdog path; because Pi removed `AgentSession.newSession()`, gateway writes `/tmp/joelclaw/gateway.force-new-session.json` and exits cleanly so launchd restarts into a fresh `SessionManager`, then injects the compression summary as hidden context before the next inbound turn
 - those watchdog-triggered runs emit the same `daemon.maintenance.started|completed|failed` telemetry as turn-bound maintenance
 
+## Tripwire investigator
+
+The five-minute gateway heartbeat tripwire opens one visible investigator instead of repeating a native alert.
+
+- `infra/gateway-tripwire.sh` detects `missing` and `stale` heartbeat states.
+- `infra/gateway-alert-investigator.sh` starts or resumes one incident in the default Aqua Herdr session.
+- The incident verifies the configured DGX endpoint, then launches Pi with `dgx-glm/glm-5.3-flash:high` and normal tools.
+- Repeated unhealthy checks reuse the same workspace. A dead agent restarts in place. A failed model or Herdr launch retries after five minutes.
+- Recovery stays pending until the workspace is `review ready` and the same agent accepts the root-cause receipt prompt.
+- A native notification is only the fallback when Herdr or the investigator cannot start.
+
+The investigator follows this skill. It must use `joelclaw gateway` for lifecycle changes and must not start another communications transport. Durable design details live in `.brain/resources/gateway-herdr-alert-investigator.svx`.
+
 ## Interruptibility and supersession (ADR-0196 / ADR-0218 rank 4 slice)
 
 For direct human turns across Telegram, Discord, iMessage, and Slack invoke paths, the latest message now wins.
@@ -308,6 +321,8 @@ This keeps gateway automation hooks out of normal interactive pi sessions.
 | `packages/gateway/src/channels/telegram.ts` | Telegram bot channel |
 | `packages/gateway/src/command-queue.ts` | Serial FIFO queue → `session.prompt()` |
 | `packages/gateway/src/heartbeat.ts` | Periodic autonomous task runner |
+| `packages/gateway/src/gateway-alert-investigator.ts` | XState incident lifecycle and default-Herdr DGX GLM launcher |
+| `infra/gateway-tripwire.sh` | Five-minute heartbeat sensor and investigator dispatch |
 | `packages/system-bus/src/inngest/middleware/gateway.ts` | Middleware injecting `gateway` context |
 | `packages/cli/src/commands/gateway.ts` | CLI subcommands |
 | `~/.joelclaw/scripts/gateway-start.sh` | launchd start script |