Files
shitratgit[bot] 0719010a84 fix(nas): route docs and books through current storage placement
Move NAS placement behind shared configuration, update docs paths, and remove stale storage defaults.

Focused tests, CLI/docs-api typechecks, and changed-file Biome checks pass. CI failures are pre-existing workflow package-manager duplication and an unrelated Vercel ADR auth failure.

SR 🐀
2026-09-11 20:52:12 +00:00

57 KiB

joelclaw CLI

Canonical operator interface for joelclaw.

Contract

  • JSON envelope output (ok, command, result, next_actions)
  • Deterministic error codes via respondError
  • HATEOAS navigation in every command response
  • Heavy dependencies and privileged service credentials loaded lazily when possible; commands that do not need Inngest must not crash when INNGEST_EVENT_KEY is absent on thin Machines
  • Capability adapter registry with typed command contracts (packages/cli/src/capabilities/; otel/recall/deploy/log/secrets/notify/mail/subscribe/heal adapter implementations live in @joelclaw/sdk)

SDK surface (@joelclaw/sdk)

@joelclaw/sdk is the programmatic wrapper for software integrations that need CLI parity without rebuilding adapters.

Current contract:

  • transport modes:
    • subprocess — shell to joelclaw
    • inprocess — run SDK capability adapters directly (no shell) for supported capabilities (deploy, heal, log, mail, notify, otel, recall, secrets, subscribe)
    • hybrid (default) — inprocess first, subprocess fallback
  • parses canonical JSON envelopes (ok, command, result, next_actions)
  • provides typed convenience methods for common routes:
    • status()
    • deployWorker(options)
    • logWrite({ action, tool, detail, reason? })
    • notifySend({ message, channel?, priority?, context?, type?, source?, telegramOnly? }) — producer-facts surface; every field is evidence for the gateway agent, never routing instruction
    • secretsStatus/lease/revoke/audit/env
    • otelList/search/stats/emit
    • recall / recallRaw
    • vaultRead/search/ls/tree + vaultAdrList/collisions/audit/rank
  • exposes structured errors:
    • JoelclawProcessError (spawn/exit/parse failures)
    • JoelclawEnvelopeError (ok:false envelopes via runOrThrow)
    • JoelclawCapabilityError (inprocess capability failures)

Example:

import { createJoelclawClient } from "@joelclaw/sdk";

const client = createJoelclawClient({
  timeoutMs: 15_000,
  transport: "inprocess",
});
const otel = await client.otelSearch("gateway", { hours: 1, limit: 20 });

Health endpoint fallback (ADR-0182)

CLI health probes for Inngest and worker resolve endpoints in this order:

  1. localhost
  2. discovered Colima VM IP (JOELCLAW_COLIMA_VM_IP, fallback 192.168.64.2)
  3. k8s service DNS (*.joelclaw.svc.cluster.local)

Probe detail strings include the selected endpoint class (localhost|vm|svc_dns) and skipped-candidate counts.

Inngest status truth model (ADR-0187 + ADR-0159)

joelclaw inngest status [--heal] [--wait-ms 1500]

Semantics:

  • reports both raw and normalized checks:
    • checks_raw: direct probe outcomes
    • checks: normalized truth surface used for ok
  • worker truth is route-aware:
    • if direct worker endpoint probe fails but deployment/system-bus-worker is ready, worker is reported as healthy with summary.route = "k8s-only"
  • k8s truth uses core workload readiness (not every pod phase):
    • statefulset/inngest, statefulset/redis, statefulset/typesense, deployment/system-bus-worker
  • the lightweight joelclaw status pod summary now ignores terminal Succeeded / Completed pods from old ReplicaSets so reboot rollouts don't stay falsely red after replacement pods are healthy.
  • includes Talon watchdog health (http://127.0.0.1:9999/health) plus launchd state to guide escalation.

--heal performs targeted remediation before re-check:

  1. enforce worker single-source binding
  2. restart/register worker when worker checks fail
  3. kickstart Talon + run talon --check when server/k8s checks fail
  4. re-collect status and return before/after snapshots

Brain and observation recall

joelclaw recall <query> queries the disposable observations and brain_graph_nodes Typesense projections. It preserves the existing recall result envelope, including hits[].observation, so voice and other callers do not need a coordinated cutover. Brain .svx files remain canonical; Typesense is rebuildable.

The legacy memory_observations health/e2e probes are retired with that collection. Use joelclaw observations index to rebuild observation pages and the Brain graph index owner to rebuild brain_graph_nodes.

Inngest Connect WebSocket auth probe

joelclaw inngest connect-auth [--start-only] [--timeout-ms 8000] [--instance-id joelclaw-cli-probe] [--url http://localhost:8288]

Semantics:

  • uses INNGEST_SIGNING_KEY as Bearer auth against /v0/connect/start.
  • decodes the protobuf StartResponse (connection_id, gateway_endpoint, gateway_group, session/sync token presence).
  • full mode runs the websocket handshake on the returned gateway endpoint:
    1. connects with subprotocol v0.connect.inngest.com
    2. expects GATEWAY_HELLO
    3. sends WORKER_CONNECT with session_token + sync_token
    4. expects GATEWAY_CONNECTION_READY
  • output never includes raw tokens (metadata only: length/format/JWT timestamps).
  • --start-only validates HTTP auth path without websocket handshake.

Command roots

  • joelclaw status
    • --agent-dispatch-canary runs the deterministic non-LLM timeout canary and folds its terminal truth into the status envelope
  • joelclaw summary [--hours N] [--format json|text]
  • joelclaw runs
  • joelclaw run
  • joelclaw agent
  • joelclaw content
  • joelclaw gateway
  • joelclaw loop
  • joelclaw docs
  • joelclaw vault
  • joelclaw skills
  • joelclaw mail
  • joelclaw secrets
  • joelclaw log
  • joelclaw notify
  • joelclaw otel
  • joelclaw o11y
  • joelclaw recall
  • joelclaw memory
  • joelclaw subscribe
  • joelclaw webhook
  • joelclaw inngest
  • joelclaw restate
  • joelclaw pds
  • joelclaw knowledge
  • joelclaw capabilities
  • joelclaw queue
  • joelclaw workload
  • joelclaw jobs
  • joelclaw fleet

Fleet command tree

joelclaw fleet
├── status [--host <alias>] [--config <private-manifest>]
└── diff [--host <alias>] [--config <private-manifest>]

Both commands are read-only and return the standard JSON envelope. status gathers declared-host facts; diff classifies declared expectations as in_sync, expected_difference, drift, or unavailable. The manifest is local private configuration: never commit or print its contents. Fleet plan/apply commands do not exist yet; use a reviewed, host-scoped change path only after explicit approval.

PDS command tree

joelclaw pds
├── health
├── describe
├── collections
├── records <collection> [--limit <n>]
├── write <collection> --data <json>
├── delete <collection> <rkey>
└── session [--refresh]

Semantics:

  • targets the rebuilt host-published PDS at http://localhost:9627
  • caches sessions at ~/.joelclaw/pds-session.json
  • resolves the login identifier from pds_joel_did by calling describeRepo and using the repo handle before createSession, because the rebuilt PDS accepted the handle while rejecting raw DID login
  • returns token metadata in session output without dumping raw JWTs
  • write auto-adds $type and createdAt

Docs command tree (ADR-0234 docs pipeline v2)

joelclaw docs
├── add <path> [--title <title>] [--tags a,b,c] [--category <storage-category>]
├── search <query> [--limit <n>] [--category <category>] [--concept <concept-id>] [--chunk-type <section|snippet>] [--doc <doc-id>] [--semantic]
├── context <chunk-id> [--mode snippet-window|parent-section|section-neighborhood] [--before <n>] [--after <n>] [--neighbors <n>]
├── list [--category <category>] [--limit <n>]
├── show <doc-id>
├── markdown <doc-id>
├── summary <doc-id>
├── status
├── reconcile [--manifest <path>] [--sample <n>]
├── enrich <doc-id>
├── reindex [--doc <doc-id>]
├── reindex-v2 <path> [--title <title>] [--skip-existing]
└── batch-reindex [--from-collection] [--skip-existing]

Semantics:

  • active chunk search collection is switchable via DOCS_CHUNKS_COLLECTION and now defaults to docs_chunks_v2
  • markdown + summary read durable artifacts from DOCS_ARTIFACTS_DIR (default /Volumes/services/joelclaw/docs-artifacts)
  • docs status reports both docs_chunks (v1) and docs_chunks_v2 plus artifact directory availability
  • agentic expansion flow is docs searchdocs contextdocs markdown / docs summary

Skills command tree

joelclaw skills
├── ensure <skill>
│   [--source-root <repo>][--consumer all|agents|pi|claude]
└── audit [--deep] [--wait-ms <wait-ms>] [--poll-ms <poll-ms>]

joelclaw skills ensure semantics:

  • compatibility-shim maintenance surface for skills that already live in a repo skills/ directory
  • resolves skills/<name>/SKILL.md from --source-root, cwd/ancestor repos, or the joelclaw repo fallback
  • creates missing flat per-skill symlinks and repairs wrong symlinks in ~/.agents/skills/, ~/.pi/agent/skills/, and ~/.claude/skills/
  • expects those consumer roots to be real directories; whole-root symlinks are a layering error
  • fails loudly if a consumer target exists as a real file/dir instead of a symlink
  • returns the installed skill path so agents can read it immediately
  • for external third-party skill packages, the CLI points at the upstream installer: npx skills add -y -g <source>

Workload command tree

joelclaw workload
├── plan "<intent>"[--preset docs-truth|research-compare|refactor-handoff][--kind auto|repo.patch|repo.refactor|repo.docs|repo.review|research.spike|runtime.proof|cross-repo.integration][--shape auto|serial|parallel|chained][--autonomy inline|supervised|afk|blocked][--proof none|dry-run|canary|soak|full][--risk reversible-only,host-okay][--artifacts patch,verification,summary][--acceptance "criterion one|criterion two"][--repo /abs/path/or/owner/repo][--paths docs/workloads.md,docs/cli.md][--paths-from status|head|recent:<n>][--stages-from /abs/path/to/stages.json][--write-plan ~/.joelclaw/workloads/][--requested-by Joel]
├── dispatch <plan-artifact>
│   [--stage <stage-id>][--to <to>][--from <from>][--send-mail][--write-dispatch ~/.joelclaw/workloads/]
├── run <plan-artifact>
│   [--stage <stage-id>][--tool pi|codex|claude][--execution-mode auto|host|sandbox][--sandbox-backend local|k8s][--sandbox-mode minimal|full][--skip-dep-check][--repo-url <repo-url>][--dry-run]
└── sandboxes
    ├── list [--state active|completed|failed|cancelled] [--mode minimal|full] [--expired] [--limit <n>]
    ├── cleanup [--request-id <id> | --sandbox-id <id> | --expired | --all-terminal] [--dry-run] [--force]
    └── janitor [--dry-run]

joelclaw workload plan semantics:

  • planner surface for ADR-0217 Phase 4.3
  • returns a canonical request + plan envelope using docs/workloads.md
  • also returns guidance with:
    • recommendedExecution (execute-inline-now, tighten-scope-first, dispatch-after-health-check, etc.)
    • operatorSummary so the CLI says what to do next instead of shrugging
    • adrCoverage to show which ADRs likely govern the slice already; on fresh repo-local ADR clusters it remains best-effort guidance and may still need human reconciliation
    • recommendedSkills with install/read readiness, including joelclaw skills ensure <name> for local repo skills and npx skills add -y -g <source> for external skills
    • executionExamples for serial / parallel / chained coding workloads, including setup + execution few-shot patterns
    • executionLoop so the agent gets the honest plan → approve → execute/watch → summarize posture after the operator says yes
  • infers kind, shape, mode, and backend when the caller leaves them open
  • supports reusable planner presets for common docs/research/refactor shapes
  • preserves Acceptance: clauses embedded in the prompt when --acceptance is omitted
  • prefers implementation intent over docs follow-through, so mixed intents like refactor ... then update docs or extend ... then update README stay implementation-shaped
  • validates known risk and artifacts values and emits warnings for unknown ones instead of silently inventing vocabulary
  • mentioning sandboxes as the topic of a comparison does not force sandbox mode by itself; isolation has to be explicit or implied by AFK autonomy
  • deploy-allowed is inferred only from explicit release/deploy intent; nouns like published skills do not count as deploy requests
  • proof=canary|soak no longer forces supervised repo work onto durable / restate by itself
  • --paths-from can seed file scope from local git activity, and --write-plan writes the full envelope to a reusable JSON artifact
  • --stages-from <file> loads an explicit JSON stage DAG, validates dependencies/cycles, carries per-stage acceptance into plan verification, and adds DAG metadata to the plan result
  • when --shape auto is still in effect, an explicit stage DAG now decides whether the plan is serial, parallel, or chained
  • chained repo.patch/refactor work can decompose a Goal: section into explicit milestones and add a reflection/update stage when the prompt asks for it
  • defaults --repo to the current working directory and infers branch / baseSha when that target is a local git repo; if the cwd is not a git repo, it warns and points the caller at --repo
  • does not execute code or mutate repos

joelclaw workload dispatch semantics:

  • reads a saved plan artifact from joelclaw workload plan --write-plan ...
  • turns it into a stage-specific dispatch/handoff contract with canonical handoff data plus a clawmail-ready subject/body
  • also returns dispatch guidance so the CLI can say whether the right move is to execute the stage now, keep the slice inline, or pause for a health check/recipient clarification
  • dispatch guidance also carries executionLoop so the receiving agent knows the approval, progress-reporting, and closeout posture instead of inventing workflow theatre
  • defaults to the first stage, but --stage can target a later stage explicitly
  • preserves scoped file boundaries through selectedStage.reservedPaths / handoff.reservedPaths
  • carries forward ADR coverage + recommended skill setup/readiness for the receiving agent
  • --write-dispatch writes the dispatch contract as a reusable JSON artifact
  • --send-mail --to <to> --from <from> sends that contract through joelclaw mail
  • does not execute code or mutate repos

joelclaw workload run semantics:

  • reads a saved plan artifact and normalizes it into the canonical queue-backed runtime request
  • emits the queue family workload/requested, which the registry maps to system/agent.requested
  • current durable runtime path is Redis queue → Restate dagOrchestrator → dagWorker
  • dagWorker handlers currently cover shell, infer, and microvm
  • defaults to --tool pi, with codex|claude as explicit opt-ins
  • supports --sandbox-backend local|k8s plus --sandbox-mode minimal|full when sandbox execution is the point
  • explicit-stage plans now gate stage execution on dependency inbox truth; use --skip-dep-check only for deliberate manual recovery or replay
  • supports --dry-run for request inspection before queue admission
  • returns queue admission details once the request is enqueued
  • if queue admission fails before the runtime request is accepted, workload run now writes a terminal inbox snapshot for that requestId immediately instead of leaving operators with no truth artifact to inspect
  • status|explain|cancel remain planned, not shipped
  • joelclaw workload sandboxes is now the operator surface for ADR-0221 local sandbox state:
    • list reconciles the registry against per-sandbox sandbox.json metadata before reporting retention + filesystem truth, so operator output stops lying about terminal state after older partial writeback failures
    • cleanup is the bounded manual deletion path with --dry-run and --force, and it performs the same reconciliation before deciding whether a sandbox is still active
    • janitor is the dedicated expired-sandbox cleanup path instead of waiting for startup-time opportunistic pruning, and it also reconciles registry drift before computing candidates
    • scheduled janitoring now lives in the repo-managed launchd service com.joel.local-sandbox-janitor, which runs joelclaw workload sandboxes janitor at load and every 30 minutes

Restate command tree

joelclaw restate
├── status [--namespace <namespace>] [--admin-url <url>]
├── deployments [--admin-url <url>] [--cli-bin <bin>]
├── smoke [--script <path>]
├── enrich "<name>" [--github <user>] [--twitter <user>] [--depth quick|full] [--sync]
├── pi-mono-sync [--repo <owner/repo>] [--full-backfill] [--max-pages <n>] [--sync]
└── cron
    ├── status [--namespace <namespace>] [--service-name <service>] [--base-url <url>]
    ├── list [--namespace <namespace>] [--service-name <service>] [--base-url <url>]
    ├── enable-health [--schedule "0 7 * * * *"] [--run-now] [--restate-url <url>]
    └── delete <job>

joelclaw restate smoke semantics:

  • resolves the smoke script path in this order:
    1. exact absolute path
    2. relative to current working directory
    3. relative to JOELCLAW_ROOT
    4. relative to ~/Code/joelhooks/joelclaw
  • runs scripts/restate/test-workflow.sh by default.
  • default smoke validates deployGate end-to-end.
  • DAG smoke is available via script override:
    • joelclaw restate smoke --script scripts/restate/test-dag-workflow.sh

joelclaw restate pi-mono-sync semantics:

  • triggers the pi-mono-sync Restate DAG pipeline, which runs the host-side direct task runner scripts/restate/run-tier1-task.ts --task pi-mono-artifacts-sync.
  • syncs repo docs, issues, issue comments, pull requests, pull-request review comments, commits, and releases into the Typesense collection pi_mono_artifacts.
  • writes two materialized documents into the same collection:
    • maintainer profile (kind=maintainer_profile, currently for badlogic)
    • sync checkpoint (kind=sync_state) so later runs can stay incremental unless --full-backfill is set.
  • default repo is badlogic/pi-mono.
  • --sync waits for the DAG result; async mode returns a workflow ID and lets Restate finish in the background.
  • the new collection is queryable through joelclaw search --collection pi_mono_artifacts.

joelclaw restate cron semantics:

  • manages Dkron scheduler jobs for Restate pipelines.
  • default access path is a short-lived CLI-managed kubectl port-forward to svc/dkron-svc.
  • pass --base-url only when you already have a direct Dkron API endpoint.
  • enable-health seeds the health proof job: restate-health-check.
  • sync-tier1 upserts the full ADR-0216 tier-1 set:
    • restate-health-check
    • restate-skill-garden
    • restate-typesense-full-sync
    • restate-daily-digest
    • restate-subscription-check-feeds
  • list includes migratedFrom, successCount, errorCount, lastSuccess, and lastError so the soak is visible from the CLI without spelunking Dkron by hand.
  • the jobs use Dkron's shell executor plus wget; it appends epoch seconds to each workflow ID prefix so every scheduled run is a fresh Restate workflow.
  • Dkron cron expressions are six-field by default (sec min hour dom month dow), so hourly-at-minute-7 is 0 7 * * * *, not 7 * * * *.

Discover command

joelclaw discover <url> [-c <context>] [--site <site>] [--visibility <visibility>]

Semantics:

  • --site choices: joelclaw, wizardshit, shared
  • --visibility choices: public, private, archived, migration-only
  • sensible defaults when omitted: site=joelclaw, visibility=public
  • default path still emits discovery/noted directly to Inngest.
  • when QUEUE_PILOTS=discovery, joelclaw discover now posts raw event intent to the worker admission endpoint (POST /internal/queue/enqueue) instead of writing Redis directly:
    • the worker owns queue admission, static registry lookup, and bounded triage mode resolution
    • returns queue metadata (streamId, eventId, priority) instead of Inngest run ids
    • includes triageMode + triage metadata whenever the family is enabled for shadow or enforce
    • relies on the Restate queue drainer to forward the event onward
  • this keeps discovery pilot clients thin while the server remains the only queue policy surface.
  • joelclaw discover is the thin fire-and-forget shortcut. If you need the final link for the created piece in the same turn, use the canonical follow path instead:
joelclaw send discovery/noted --data '{"url":"<url>","context":"<optional>","site":"joelclaw","visibility":"public"}' --follow

The terminal result from discovery-capture now includes finalLink. send --follow treats transient Inngest run lookup misses (for example immediate sql: no rows in result set after event send) as polling gaps, not fatal send failures: it keeps the event-sent result, emits a warning if final state cannot be confirmed, and points operators to joelclaw run / joelclaw runs for inspection.

Subscribe check queue pilot

joelclaw subscribe check [--id <id>]

Semantics:

  • scoped checks (--id <id>) still emit subscription/check.requested directly to Inngest.
  • all-subscription checks keep the legacy Inngest path by default.
  • when QUEUE_PILOTS=subscriptions, joelclaw subscribe check without --id posts subscription/check-feeds.requested to the worker admission endpoint instead of writing Redis directly:
    • returns queue metadata (streamId, eventId, priority)
    • includes triageMode + triage metadata whenever the family is enabled for shadow or enforce
    • points next actions at joelclaw queue inspect / joelclaw queue depth
    • relies on the Restate queue drainer to forward the actual subscription/check-feeds.requested event name onward

Jobs command tree (ADR-0217 runtime monitor)

joelclaw jobs
└── status [--hours <n>] [--count <n>] [--namespace <namespace>] [--restate-admin-url <url>] [--dkron-service-name <service>] [--dkron-base-url <url>]

Semantics:

  • jobs status is the first operator glance for real workloads during the ADR-0217 transition. It aggregates the queue/Redis substrate, Restate runtime, Dkron scheduler, and still-live Inngest surfaces into one JSON snapshot.
  • overall.status is a bounded truth surface (healthy|degraded|down) derived from those four components, not a raw dump of every underlying health probe.
  • queue section:
    • reads the canonical Redis queue directly through @joelclaw/queue
    • reports depth, priority buckets, oldest age, and active deterministic pauses
  • Restate section:
    • mirrors joelclaw restate status in-place so the operator can see statefulset readiness + admin health without command hopping
  • Dkron section:
    • mirrors joelclaw restate cron status in-place and includes the count of Restate-tagged scheduler jobs when the API is reachable
  • Inngest section:
    • stays visible during migration, but its status is scoped to the transitional job path that still matters here: server/worker health plus recent run outcomes
    • broad informational checks (for example unrelated k8s pod drift) stay in the payload without poisoning the top-level job monitor status
  • next actions point directly at the management surfaces that actually move workload state: queue control status, queue observe, queue resume, restate status, restate cron status, and runs

Pi async jobs monitor (runtime_jobs_monitor)

The loaded pi extension at packages/pi-extensions/inngest-monitor/index.ts now does two jobs:

  1. tracks followed Inngest runs (inngest_send, inngest_runs)
  2. exposes runtime_jobs_monitor for the ADR-0217 runtime substrate

runtime_jobs_monitor semantics:

  • action=start|status|stop (default start)
  • on start, it polls joelclaw jobs status in the background, paints a persistent widget, emits OTEL on severity changes and meaningful workload-state changes, and sends hidden follow-up messages for async report-back
  • on status, it returns the latest runtime snapshot (overall status, queue depth, active pause count, Restate/Dkron/Inngest state)
  • on stop, it stops the poller and sends a final follow-up summary
  • widget posture is intentionally operator-first: current runtime state on top, active followed runs underneath

This is the canonical async monitoring path when you want a pi session to keep an eye on real workloads while you do other things.

Queue command tree (ADR-0217 Phase 1)

joelclaw queue
├── emit <event> [-d <json>] [-p P0|P1|P2|P3]
├── depth
├── stats [--hours <n>] [--limit <n>]
├── observe [--hours <n>] [--limit <n>] [--since <iso|ms>]
├── pause <family> [--ttl <duration>] [--reason <text>]
├── resume <family> [--reason <text>]
├── control
│   └── status [--hours <n>] [--limit <n>] [--since <iso|ms>]
├── list [--limit <n>]
└── inspect <stream-id>

Semantics:

  • all queue subcommands return a clean JSON envelope; read-oriented subcommands close their Redis client before exit and emit stays a thin worker client instead of writing Redis directly
  • queue state/control commands resolve Redis from the canonical joelclaw CLI config (~/.config/system-bus.envREDIS_URL) before considering ambient shell env so the installed operator surface stays pointed at the same localhost queue as the worker and Restate drainer
  • emit posts queue admission intent to the worker endpoint (POST /internal/queue/enqueue).
    • accepts event name (e.g., discovery/noted, content/updated)
    • accepts optional JSON payload via -d
    • optional priority override via -p is normalized client-side, but static registry routing and bounded triage stay server-side
    • the worker generates the canonical QueueEventEnvelope, adds trace metadata, resolves shadow/enforce mode for the event family, evaluates bounded triage, and persists the queue record
    • returns the Redis stream ID, priority, and any triageMode / triage metadata from admission
  • depth reports queue depth, priority distribution (P0/P1/P2/P3 counts), oldest/newest message timestamps
  • stats summarizes recent Restate queue-drainer behavior plus Phase 2 triage behavior from OTEL over a lookback window.
    • dispatch section reports sampled/found dispatch events, live queue depth, started/completed/failed counts, success rate, queue wait-time percentiles (p50/p95), dispatch-duration percentiles, promotion count, top event families, and recent failures
    • triage section reports attempts, completed/failed/fallback counts, fallback counts by reason, disagreement count, applied-change count, suggested-not-applied count, route mismatches, latency percentiles, per-family rollups, and recent mismatch/fallback samples
    • uses metadata.waitTimeMs from queue.dispatch.started as the Story 5 queue-to-dispatch latency signal
    • uses queue.triage.* OTEL metadata as the Story 3 source of truth for queue-admission disagreements and fallback behavior
    • --since <iso|ms> overrides the lower bound so operators can anchor soak evidence to a known clean point (for example the supervised queue.drainer.started after a rollout) instead of mixing fresh traffic with a dirty pre-fix window
    • keeps the operator in CLI-land; no raw Redis keys or manual OTEL spelunking required for the first sanity pass
  • jobs status is the first unified runtime view; drop to queue subcommands when the aggregated surface says the queue itself needs attention.
  • observe is now the Phase 3 Story 2-4 dry-run Sonnet operator surface.
    • builds a canonical live snapshot from current queue depth + queued messages + recent drainer OTEL + recent triage OTEL + gateway sleep/muted-channel state + active deterministic pauses
    • runs the bounded Sonnet observer in dry-run mode only and returns the current snapshot plus the current decision
    • long Sonnet summaries are trimmed instead of turning otherwise-useful observations into schema-only fallbacks
    • if all queued work is intentionally held behind fresh active manual pauses and no recent failures suggest downstream trouble, it short-circuits to a deterministic noop instead of wasting a 60s Sonnet call on an obvious hold state
    • history summarizes recent queue.observe.* OTEL for the same window so operators can compare the latest dry-run against raw history without spelunking Typesense by hand
    • control now reflects the shipped deterministic queue-control plane: active manual pauses, queue.control.applied|expired|rejected counts, and recent control events come from the same Redis + OTEL truth the drainer uses
    • --since <iso|ms> anchors the related OTEL history window the same way queue stats does
  • pause applies a deterministic manual family pause with bounded TTL and emits queue.control.applied telemetry
  • resume clears a deterministic family pause and emits either queue.control.applied or queue.control.rejected when the family was not paused
  • control status is the dedicated deterministic control-plane operator surface.
    • reports active pauses (family, reason, TTL, applied/expiry timestamps, actor)
    • summarizes queue.control.applied|expired|rejected OTEL for the same window
    • is the first CLI answer to “what queue controls are active right now?” before any automatic Sonnet mutation ships
  • list lists recent messages in priority order (highest priority first), does not ack/remove
  • inspect loads a message by Redis stream ID and returns full payload + metadata
    • if the message is already acked/expired, it now returns a structured QUEUE_MESSAGE_MISSING error envelope with queue-state next actions instead of crashing the CLI

Queue configuration:

  • Stream key: joelclaw:queue:events
  • Priority index: joelclaw:queue:priority
  • Consumer group: joelclaw:queue:cli
  • Phase 1 pilot events: discovery/noted, discovery/captured, content/updated, subscription/check-feeds.requested, github/workflow_run.completed

OTEL telemetry under queue.* namespace:

  • CLI queue commands forward queue package telemetry to OTEL with source=cli and component=queue
  • queue.enqueue — message enqueued
  • queue.lease — message leased for processing (includes wait time, priority, promotion metadata)
  • queue.ack — message acknowledged
  • queue.replay — unacked messages loaded for replay

Daily summary command

joelclaw summary [--hours 24] [--format json|text]

Semantics:

  • aggregates recent git activity (joelclaw + Vault), Inngest run rollups, k8s pod health, OTEL stats, slog deploy/config events, and ADR churn into one response envelope.
  • default output is JSON summary payload; --format text adds a compact text rendition under result.text for downstream chat/mobile surfaces.

Memory command group

joelclaw recall "<query>"
joelclaw recall --request-file -
joelclaw memory review --since 48h
joelclaw memory search "<legacy query>" # typed retirement response; no search occurs
joelclaw memory write "<legacy input>"  # typed retirement response; no write occurs

Semantics:

  • recall "<query>" is the canonical interactive composed-recall surface.
  • recall --request-file - accepts an exact composed-recall request on stdin.
  • memory review returns recent accepted-session, Git, Brain, OTEL, and flowing evidence in separate lanes.
  • memory search is a fail-closed compatibility pointer to recall; it returns MEMORY_SEARCH_RETIRED with exit 3.
  • memory write is a fail-closed compatibility pointer. Agent work enters through accepted Runs; durable curation belongs in Brain .svx pages.

Knowledge turn-write command

joelclaw knowledge note \
  --source gateway \
  --agent gateway-daemon \
  --session <session-id> \
  --turn <turn-number> \
  --summary "<what changed>"

Use --skip-reason routine-heartbeat|duplicate-signal|no-new-information when a turn is eligible but has no durable signal to capture.

Gateway channel control

joelclaw gateway channel list
joelclaw gateway channel status [channel]
joelclaw gateway channel disable <channel> [--restart] [--force]
joelclaw gateway channel enable <channel> [--restart]

Semantics:

  • edits ~/.joelclaw/scripts/gateway-start.sh for the known runtime channels: telegram, discord, imessage, slack.
  • writes a /tmp/joelclaw/gateway-start.sh.* backup before changing the script.
  • never prints secret values; output reports variable names plus source classes (secret, derived, literal, blank, missing).
  • --restart applies the change immediately via joelclaw gateway restart; otherwise next actions tell the agent to restart.
  • disabling telegram requires --force because it is a primary operator channel.

Gateway known issues / muted channels

joelclaw gateway known-issues
joelclaw gateway mute <channel> [--reason "<why muted>"]
joelclaw gateway unmute <channel>

Semantics:

  • stores muted channel IDs at Redis key gateway:health:muted-channels (JSON array).
  • stores optional mute reasons at Redis key gateway:health:mute-reasons (JSON object).
  • muted channels remain in probe telemetry but are excluded from gateway.channels.degraded alerts.

Gateway process-layer diagnostics (joelclaw gateway diagnose) now inspect exact launchd state for com.joel.gateway and report disabled services explicitly. joelclaw gateway restart now re-enables the launch agent before bootstrap/kickstart to avoid restart failures when launchd has the service disabled.

Use joelclaw gateway enable for direct launch-agent recovery (enable + bootstrap + kickstart) without manual launchctl usage.

Gateway status/diagnose now separate daemon availability from Redis bridge health:

  • joelclaw gateway status prefers daemon /health when available, so Redis loss no longer makes the gateway look fully dead.
  • status now returns mode (normal or redis_degraded), degradedCapabilities, sessionPressure (context %, compaction age, session age, next action), and guardrails (current turn tool budget state + pending deploy verifications).
  • joelclaw gateway diagnose treats redis_degraded as a degraded runtime, not a process failure, skips Redis-dependent E2E checks in that mode, and surfaces runtime guardrail findings when a checkpoint/deploy verification is active.

Gateway behavior control plane (ADR-0211)

joelclaw gateway behavior add --type keep|more|less|stop|start --text "..."
joelclaw gateway behavior list
joelclaw gateway behavior promote --id <candidate-id>
joelclaw gateway behavior remove --id <directive-id>
joelclaw gateway behavior apply
joelclaw gateway behavior stats

Semantics:

  • Single write authority is CLI. Extensions must call these commands; no direct Redis/Typesense writes.
  • Active runtime contract lives in Redis key joelclaw:gateway:behavior:contract.
  • Directive/candidate history lives in Typesense collection gateway_behavior_history.
  • add normalizes directives, enforces conflict + dedupe + cap rules, and updates Redis + Typesense.
  • promote moves a pending daily-review candidate into the active contract (manual gate; no auto-activation).
  • apply re-runs governance over the active contract (dedupe, conflict cleanup, cap enforcement) and expires stale candidates.
  • stats reports contract hash/version, candidate lifecycle counts, and governance settings.

Run listing semantics

joelclaw runs [--status RUNNING|FAILED|COMPLETED|QUEUED|CANCELLED]

Semantics:

  • applies backend status filtering and a local status guard so mixed-status payloads cannot leak through.
  • the runs list query uses a longer timeout budget than generic Inngest GraphQL calls because self-hosted runs(...) can take ~50s even for small filtered windows; timeout scales upward with requested count, and timeout aborts get one bounded retry with a larger budget before the CLI gives up.
  • for suspicious RUNNING rows (endedAt present while running, or long-running health checks), runs performs bounded detail reconciliation (max 5 lookups) to detect stale SDK-unreachable ghosts.
  • runs with stale indicators include staleSignal with likely, confidence, and machine-readable reasons.
  • response includes staleSignals summary (detected, likely, detailChecked) and suggests joelclaw inngest sweep-stale-runs when likely ghosts are present.
  • count in response reflects post-filter rows (what the operator actually sees).

Run inspection + cancellation

joelclaw run <run-id> [--cancel] [--wait-ms 3000]

Semantics:

  • default mode returns run detail, trigger event, trace, and step errors.
  • run detail and joelclaw event <event-id> now use a much larger timeout/retry budget than generic GraphQL calls because self-hosted detail lookups can take more than a minute under load even while the runtime itself is healthy.
  • --cancel issues Inngest GraphQL cancelRun for active runs, then polls status up to --wait-ms.
  • deterministic error envelopes:
    • RUN_CANCEL_FAILED when cancelRun mutation fails.
    • RUN_CANCEL_TIMEOUT when run remains RUNNING|QUEUED after the wait window.
    • RUN_STALE_SDK_UNREACHABLE when a run appears RUNNING but trace errors show Unable to reach SDK URL and cancellation has no live execution to target.
  • terminal runs are never re-cancelled; response includes cancellation.skipped = "already_terminal".

Capability adapter config precedence (ADR-0169 phase 0)

Resolution order is deterministic:

  1. CLI flags (e.g. --adapter)
  2. Environment variables
  3. Project config (.joelclaw/config.toml)
  4. User config (~/.joelclaw/config.toml)
  5. Built-in defaults

Current env keys:

  • JOELCLAW_CAPABILITY_<CAPABILITY>_ADAPTER
  • JOELCLAW_CAPABILITY_<CAPABILITY>_ENABLED

Capability-backed command roots (ADR-0169 through phase 4)

joelclaw secrets status
joelclaw secrets lease <name> --ttl 15m
joelclaw secrets revoke <lease-id>
joelclaw secrets revoke --all
joelclaw secrets audit --tail 50
joelclaw secrets env --dry-run [--ttl 1h] [--force]

joelclaw log write --action <action> --tool <tool> --detail <detail> [--reason <reason>] [--session <session>] [--system <system>]

joelclaw notify send "<message>" [--kind memory|alert|digest|ask|receipt] [--priority low|normal|high|urgent|critical] [--channel gateway|main|all] [--context '{"k":"v"}']
joelclaw notify wait <event-id> --source <source> [--timeout 15s]
joelclaw messages trace <flowId>

joelclaw heal {list|run}

joelclaw mail {status|register|send|inbox|read|reserve|renew|release|locks|search}

joelclaw otel {list|search|stats|emit}

joelclaw o11y {session|system}

joelclaw recall <query> [--limit N] [--min-score F] [--raw] [--include-hold] [--include-discard] [--budget auto|lean|balanced|deep] [--category <id|alias>]

joelclaw sessions search <query> [--source typesense|local|ssh|both] [--runtime pi|codex|claude-code|all] [--extract] [--machine dark-wizard] [--ssh-target joel@dark-wizard] [--limit N]
joelclaw session search <query> ...  # singular alias
joelclaw sessions extract <session-id-or-path> --query <topic> [--format json|markdown]
joelclaw sessions chunks <query> [--source typesense|local|both] [--limit N] [--context-before N] [--context-after N]
joelclaw sessions inspect <session-id-or-path> --around <regex> [--before N] [--after N]
joelclaw sessions signals [--kind friction|preference|decision|praise|mode-mismatch|any] [--source local|ssh|both] [--machine dark-wizard] [--since 14d] [--limit N] [--sample N] [--review-out path.jsonl] [--evaluate] [--format json|ndjson]
joelclaw sessions friction [--source local|ssh|both] [--machine dark-wizard] [--since 14d] [--limit N]

joelclaw satellite health [--notify] [--central-ssh joel@panda] [--priority high]
joelclaw satellite repair-request [--central-ssh joel@panda] [--priority high]

joelclaw subscribe {list|add|remove|check|summary}

Semantics:

  • log writes structured system entries (slog backend).
  • log write now accepts explicit --session / --system provenance flags and also falls back to SLOG_SESSION_ID / SLOG_SYSTEM_ID env vars before handing off to the slog backend.
  • logs reads/analyzes runtime logs.
  • notify is the canonical producer command for messages to Joel; gateway push remains transport/debug. Plain text is a complete payload. Producers do not need to add or change flags for the Agent Comms Gateway.
  • Message text, --kind, --priority, --channel, --telegram-only, and structured evidence are evidence for the gateway agent. They never instruct routing, urgency, timing, formatting, batching, suppression, or platform choice. Old flags remain accepted without deprecation nagging.
  • The current CLI accepts structured JSON evidence through --context. The producer-facts contract reserves --data <json> and repeated --ref, but joelclaw notify send does not implement those flags yet. Add run IDs, receipts, links, or available actions only when they help the agent make or verify its decision.
  • The gateway agent records one gateway.decision.recorded receipt for each consumed external event before advancing its stream cursor. Recorded deliver and aggregate/close-deliver decisions are executed mechanically by the slim transport. The current decision executor delivers only to Telegram.
  • messages trace <flowId> reads the stream and delivery lifecycle. Correlate replies, reactions, button taps, decisions, and platform receipts by flowId.
  • deploy, heal, log, notify, secrets, mail, otel, recall, and subscribe keep their existing UX/envelopes while executing through capability registry adapters (scripted-deploy, runbook-heal, slog-cli, gateway-redis, agent-secrets-cli, mcp-agent-mail, typesense-otel, typesense-recall, redis-subscriptions).
  • typesense-otel, typesense-recall, scripted-deploy, runbook-heal, slog-cli, agent-secrets-cli, gateway-redis, mcp-agent-mail, and redis-subscriptions adapter logic is canonical in @joelclaw/sdk (packages/sdk/src/capabilities/adapters/*); CLI adapter files are thin re-exports.
  • otel emit accepts stdin JSON payloads (or convenience args/positional action), normalizes defaults (id, timestamp, level=info, success=true), and forwards to the worker ingest endpoint (/observability/emit). Provenance defaults are now role-aware: explicit event/session fields win, then SLOG_SESSION_ID / SLOG_SYSTEM_ID, then session handle envs when present, then role/hostname fallbacks (gateway or JOELCLAW_ROLE, system hostname stripped to panda).
  • otel list and otel search accept exact --session / --system filters, mapped to sessionId / systemId in Typesense.
  • knowledge search now auto-heals the common post-rebuild failure mode where system_knowledge is missing: on a 404 Collection not found response it creates the collection, reindexes ADRs + skills once, retries the query, and reports the repair in-band.
  • o11y session / o11y system run a unified multi-search across otel_events and system_log, merge both timelines by timestamp, and tag each hit with its source collection.
  • sessions search is explicit raw-history drill-down. It reads native transcript adapters and the current local SQLite projection when available; --source local never depends on retired Typesense Run collections. Use --runtime pi|codex|claude-code|all and verify the returned native path/runtime before claiming capture success. Remote index pointers are compatibility evidence, not a memory lane.
  • sessions search --extract runs bounded deterministic extraction for top raw hits so recovery prompts can include decisions, commands, files, receipts, verification, blockers, next actions, and exact transcript line pointers without dumping the full JSONL. Each emitted hit carries its own extraction when available; .result.extractions remains as a backward-compatible convenience list. Large session envelopes are written through direct stdout, not Effect Console, so | jq remains safe above 64KB.
  • sessions extract reads one raw local transcript by path or session-id substring, redacts likely secrets, and returns JSON extraction fields. --format markdown still uses the standard JSON envelope and places rendered markdown at .result.markdown; raw markdown stdout would violate the CLI JSON contract.
  • sessions chunks exposes matching Typesense chunks and/or raw local neighboring transcript context. Use top-level .result.chunks/.result.hits; source-specific mirrors remain under .result.local.chunks and .result.typesense.chunks.
  • Source metadata uses rawReturned for pre-dedupe raw source count and emittedHits / emittedChunks for top-level emitted counts.
  • joelclaw session ... is a singular alias for joelclaw sessions ....
  • sessions signals mines role-aware high-signal user turns from raw local/remote Pi transcripts. It is ADR-0247-backed and starts deterministic: user turns are the primary signal source; assistant/tool turns are bounded evidence context. --kind friction detects correction/frustration/output-quality signals, including Joel's strong-emphasis profanity where it is attached to critique. sessions friction is an alias for sessions signals --kind friction. --sample N balances candidates across signal categories; --review-out path.jsonl writes a tiny golden-set review file with stable hitId, turnKind, prediction, improvement route, evidence, and empty review fields. --evaluate adds a small routing summary by improvement surface/kind/turnKind/confidence/review priority. V1 excludes task payloads, source material, and handoffs by default so friction analysis focuses on operator intent, review feedback, and approvals. --kind mode-mismatch finds cases where agents picked the wrong execution shape: inline vs background, visual vs response, durable Inngest vs ad-hoc script, or feedback-blocking vs async. --format ndjson streams meta, hit, and summary rows for deep scans without building one huge envelope.
  • satellite health is the thin-Machine self-check surface. It verifies local prerequisites like python3, jq, joelclaw, raw Pi session storage, and local session search JSON. With --notify, degraded health is relayed to Central via SSH (--central-ssh, default joel@panda) using joelclaw notify send --type satellite/repair.requested; this lets a satellite ask the gateway for repair without local Redis/secrets.
  • satellite repair-request always sends the Central gateway a repair request with the local probe payload.
  • New satellite Machines should use scripts/setup-satellite-rig.sh plus docs/runbooks/satellite-rig-setup.md. The bootstrap keeps satellites thin: repo + CLI + skills + raw session search + Central repair relay. It writes ~/.local/bin/joelclaw as a wrapper that sources ~/.config/system-bus.env, so Typesense and PDF Brain docs-api credentials work over non-interactive SSH without installing Panda's agent-secrets stack. Do not install Panda's k8s/Inngest/Redis/gateway stack on a satellite by default.
  • Software surfaces should route OTEL through this command contract (or shared CLI ingest helper), not ad-hoc raw HTTP calls.
  • mail search auto-falls back to /mail/api/unified-inbox filtering when MCP search_messages returns transient DB/tool errors, so steering signals remain usable.
  • mail reserve now sends explicit lease TTL (--ttl-seconds, default 900) and enforces a minimum of 60s.
  • mail renew extends active file reservations without releasing/reacquiring (--extend-seconds, default 900, optional --paths).
  • mail locks now prefers the local git-mailbox file_reservations/ artifact store when available because /mail/api/locks can under-report advisory file reservations while still reporting mailbox internals like archive/commit locks. Responses expose source and fallback_reason when artifact fallback was required.
  • subscribe check emits Inngest request events for scoped checks and for all-subscription checks when the queue pilot is off; response.ids are event/request IDs (inspect via joelclaw event <event-id>), not run IDs unless explicitly returned as runIds.
  • when QUEUE_PILOTS=subscriptions, unscoped subscribe check returns queue metadata instead of Inngest response ids because the request first lands in Redis and is forwarded by the Restate drainer.
  • recall rewrite telemetry now exposes rewrite.strategy (disabled|skipped|haiku|openai|fallback) and rewrite.reason so low-ROI rewrite skips/fallbacks are queryable; short/literal/direct-id queries skip LLM rewrite by design. The default rewrite model is openai-codex/gpt-5.6-sol; set JOELCLAW_RECALL_REWRITE_MODEL only to an approved model.

Webhook command tree (ADR-0185)

joelclaw webhook
├── subscribe <provider> <event>
│   [--repo <owner/repo>] [--workflow <name>] [--branch <name>] [--conclusion <status>][--session <session-id>] [--ttl <duration>] [--stream] [--timeout <seconds>] [--replay <count>]
├── unsubscribe <subscription-id>
├── list [--provider <provider>] [--event <event>] [--session <session-id>]
└── stream <subscription-id> [--timeout <seconds>] [--replay <count>]

Semantics:

  • Subscriptions are Redis-backed and session-scoped (joelclaw:webhook:*).
  • subscribe --stream starts an NDJSON stream immediately after creation.
  • stream emits ADR-0058 NDJSON (start, log, event, terminal result|error).
  • Default session target is gateway for central gateway role, otherwise pid-<ppid>.
  • TTL defaults to 24h and is enforced at match time.

Skills command tree (ADR-0179)

joelclaw skills
└── audit [--deep] [--wait-ms <wait-ms>] [--poll-ms <poll-ms>]

joelclaw skills audit purpose

  • triggers the skill-garden/check event on-demand
  • waits for the corresponding run and returns the findings report in-envelope
  • supports --deep for LLM staleness checks

Agent command tree (ADR-0180 phases 2-4)

joelclaw agent
├── list
├── show <name>
├── run <name> <task> [--cwd <cwd>] [--timeout <seconds>]
├── chain <steps> --task <task> [--cwd <cwd>] [--fail-fast]
└── watch <id> [--timeout <seconds>]

Semantics:

  • run emits agent/task.run for single roster agent execution and returns taskId plus eventIds from the Inngest send response.
  • run next_actions are truthful: use joelclaw event <event-id> when an event ID exists (or joelclaw events ... fallback), and never assume taskId is a run ID.
  • chain emits agent/chain.run with comma-separated sequential steps and + parallel groups (e.g. scout,planner+reviewer,coder).
  • watch streams NDJSON progress for a task (at-...) or chain (ac-...) by subscribing to joelclaw:notify:gateway, replaying joelclaw:events:gateway, and falling back to Inngest polling.
  • watch default timeout is 300 seconds for tasks and 900 seconds for chains; terminal events always include next_actions on completion, timeout, or interrupt.
  • Runtime-proof recipe (ADR-0180):
    1. joelclaw agent list (expect builtin coder/designer/ops/story-executor)
    2. joelclaw agent run coder "reply with OK" --timeout 20
    3. joelclaw event <event-id> (expect Agent Task Run status COMPLETED with output payload)
  • If Unknown agent roster entry: coder appears, treat it as worker-runtime drift: deploy latest system-bus-worker, restart the host worker, then rerun the three-step proof.

Vault command tree

joelclaw vault
├── read <ref>
├── search <query> [--semantic] [--limit <limit>]
├── ls [section]
├── tree
└── adr
    ├── list [--status <status>] [--limit <limit>]
    ├── collisions
    ├── audit
    ├── locate <query> [--limit <limit>]
    ├── refs <query>
    ├── prompt <text>
    └── rank [--band <band>] [--unscored] [--all]

joelclaw vault adr purpose

  • list — inventory ADR metadata with optional status filter
  • collisions — detect duplicate ADR numeric prefixes
  • audit — full ADR hygiene check:
    • missing/non-canonical status values
    • number collisions
    • missing superseded-by targets
    • broken or ambiguous wiki links inside ADR bodies (skips custom directive tags like [[tts:text]])
    • README index alignment against ADR files
  • locate — section/file lookup across the ADR corpus using ADR numbers, slugs, titles, and heading names
  • refs — backlink discovery for a resolved ADR file or section id
  • prompt — expand [[ADR refs]] in prompt text into canonical ids and append an <adr-context> block for agents
  • rank — score + rank ADRs by NRC+novelty rubric for daily prioritization:
    • default scope: open ADRs (accepted + proposed)
    • --all includes shipped/superseded/deprecated/rejected ADRs
    • --band <band> filters ranked rows (do-now|do-next|de-risk|park; alias nextdo-next)
    • --unscored returns ADRs missing priority-score
    • required axes: priority-need, priority-readiness, priority-confidence
    • novelty facet: priority-novelty (or alias priority-interest), defaults to neutral 3 when missing
    • score formula: clamp(round(20*(0.5*Need + 0.3*Readiness + 0.2*Confidence)) + round((Novelty-3)*5), 0, 100)
    • bands: do-now (80-100), do-next (60-79), de-risk (40-59), park (0-39)
    • emits CLI OTEL via component=vault-cli:
      • vault.adr.rank.started
      • vault.adr.rank.completed
      • vault.adr.rank.failed

Canonical statuses:

  • proposed
  • accepted
  • shipped
  • superseded
  • deprecated
  • rejected

Content command tree (ADR-0168)

joelclaw content
├── seed
├── verify
└── prune [--apply]

Semantics:

  • seed — full Vault ADR sync to Convex for canonical ADR filenames only (NNNN-*.md).
  • verify — strict ADR drift check against canonical ADR files (fails healthy state on both missing and extra ADR records in Convex).
  • prune — dry-run report of Convex ADR extras (status: dry_run).
  • prune --apply — removes ADR extras from Convex (status: pruned) and should be followed by joelclaw content verify.

Inngest source guard (ADR-0089)

joelclaw inngest source [--repair]

Semantics:

  • Verifies launchd binding for com.joel.system-bus-worker against the canonical infra/launchd/com.joel.system-bus-worker.plist values (program + working directory).
  • --repair copies canonical plist into ~/Library/LaunchAgents, performs launchctl bootout, then bootstrap with retry for transient Bootstrap failed: 5 launchd races.
  • Use before joelclaw inngest restart-worker when host runtime/source drift is suspected.

Inngest stale-run sweep (ADR-0194)

joelclaw inngest sweep-stale-runs
joelclaw inngest sweep-stale-runs --apply

Semantics:

  • preview-first by default (--apply required for mutation).
  • scope defaults to stale health checks older than 30 minutes:
    • check/o11y-triage
    • check/system-health
  • runtime target defaults:
    • namespace joelclaw
    • pod inngest-0
    • sqlite path /data/main.db
  • apply mode safety gates:
    • refuses if age threshold is too young (<5m)
    • refuses when candidate count exceeds --max-apply-candidates
    • refuses when function_runs rows are missing (cannot insert terminal history safely)
    • always creates and verifies point-in-time backup before transaction:
      • /data/main.db.pre-sweep-<UTC-stamp>.sqlite
  • terminalization contract (single transaction):
    1. insert missing history.type = FunctionCancelled
    2. insert missing function_finishes
    3. set trace_runs.status = 500 + terminal ended_at

This command exists for cases where Inngest API cancellation returns not found for stale RUNNING ghosts after SDK reachability failures.

Status command

joelclaw status [--agent-dispatch-canary]

Semantics:

  • default joelclaw status remains the fast base worker/server health surface
  • default output now also includes latestAgentDispatchCanary when a persisted deterministic canary snapshot exists, so operators can see the last proof result without spelunking runs or inbox files
  • --agent-dispatch-canary runs scripts/verify-agent-dispatch-timeout.ts and folds the deterministic non-LLM system/agent-dispatch timeout proof into the returned envelope
  • when requested, the command only reports healthy if both the base health probes and the canary pass
  • this is the canonical on-demand proof surface for the live outer-timeout closeout path; it exists so operators do not have to run the verifier script manually

Build and verify

bunx tsc --noEmit
pnpm biome check packages/ apps/
bun test packages/cli/src/commands/*.test.ts
bun build packages/cli/src/cli.ts --compile --outfile ~/.bun/bin/joelclaw
joelclaw status
joelclaw vault
joelclaw vault adr audit
joelclaw vault adr locate "gateway guardrails"
joelclaw vault adr refs 0189-gateway-guardrails
joelclaw vault adr prompt "compare [[0189-gateway-guardrails]] with [[0218-gateway-availability-lifecycle-qol-improvements]]"

Add a command

  1. Create command module in packages/cli/src/commands/.
  2. Return envelopes with respond/respondError only.
  3. Include useful next_actions with param hints.
  4. Wire command in packages/cli/src/cli.ts.
  5. Add/extend tests in packages/cli/src/commands/*.test.ts.
  6. Update this file when command tree or contracts change.