Move NAS placement behind shared configuration, update docs paths, and remove stale storage defaults.
Focused tests, CLI/docs-api typechecks, and changed-file Biome checks pass. CI failures are pre-existing workflow package-manager duplication and an unrelated Vercel ADR auth failure.
SR 🐀
57 KiB
joelclaw CLI
Canonical operator interface for joelclaw.
Contract
- JSON envelope output (
ok,command,result,next_actions) - Deterministic error codes via
respondError - HATEOAS navigation in every command response
- Heavy dependencies and privileged service credentials loaded lazily when possible; commands that do not need Inngest must not crash when
INNGEST_EVENT_KEYis absent on thin Machines - Capability adapter registry with typed command contracts (
packages/cli/src/capabilities/; otel/recall/deploy/log/secrets/notify/mail/subscribe/heal adapter implementations live in@joelclaw/sdk)
SDK surface (@joelclaw/sdk)
@joelclaw/sdk is the programmatic wrapper for software integrations that need CLI parity without rebuilding adapters.
Current contract:
- transport modes:
subprocess— shell tojoelclawinprocess— run SDK capability adapters directly (no shell) for supported capabilities (deploy,heal,log,mail,notify,otel,recall,secrets,subscribe)hybrid(default) — inprocess first, subprocess fallback
- parses canonical JSON envelopes (
ok,command,result,next_actions) - provides typed convenience methods for common routes:
status()deployWorker(options)logWrite({ action, tool, detail, reason? })notifySend({ message, channel?, priority?, context?, type?, source?, telegramOnly? })— producer-facts surface; every field is evidence for the gateway agent, never routing instructionsecretsStatus/lease/revoke/audit/envotelList/search/stats/emitrecall/recallRawvaultRead/search/ls/tree+vaultAdrList/collisions/audit/rank
- exposes structured errors:
JoelclawProcessError(spawn/exit/parse failures)JoelclawEnvelopeError(ok:falseenvelopes viarunOrThrow)JoelclawCapabilityError(inprocess capability failures)
Example:
import { createJoelclawClient } from "@joelclaw/sdk";
const client = createJoelclawClient({
timeoutMs: 15_000,
transport: "inprocess",
});
const otel = await client.otelSearch("gateway", { hours: 1, limit: 20 });
Health endpoint fallback (ADR-0182)
CLI health probes for Inngest and worker resolve endpoints in this order:
localhost- discovered Colima VM IP (
JOELCLAW_COLIMA_VM_IP, fallback192.168.64.2) - k8s service DNS (
*.joelclaw.svc.cluster.local)
Probe detail strings include the selected endpoint class (localhost|vm|svc_dns) and skipped-candidate counts.
Inngest status truth model (ADR-0187 + ADR-0159)
joelclaw inngest status [--heal] [--wait-ms 1500]
Semantics:
- reports both raw and normalized checks:
checks_raw: direct probe outcomeschecks: normalized truth surface used forok
- worker truth is route-aware:
- if direct worker endpoint probe fails but
deployment/system-bus-workeris ready, worker is reported as healthy withsummary.route = "k8s-only"
- if direct worker endpoint probe fails but
- k8s truth uses core workload readiness (not every pod phase):
statefulset/inngest,statefulset/redis,statefulset/typesense,deployment/system-bus-worker
- the lightweight
joelclaw statuspod summary now ignores terminalSucceeded/Completedpods from old ReplicaSets so reboot rollouts don't stay falsely red after replacement pods are healthy. - includes Talon watchdog health (
http://127.0.0.1:9999/health) plus launchd state to guide escalation.
--heal performs targeted remediation before re-check:
- enforce worker single-source binding
- restart/register worker when worker checks fail
- kickstart Talon + run
talon --checkwhen server/k8s checks fail - re-collect status and return before/after snapshots
Brain and observation recall
joelclaw recall <query> queries the disposable observations and brain_graph_nodes Typesense projections. It preserves the existing recall result envelope, including hits[].observation, so voice and other callers do not need a coordinated cutover. Brain .svx files remain canonical; Typesense is rebuildable.
The legacy memory_observations health/e2e probes are retired with that collection. Use joelclaw observations index to rebuild observation pages and the Brain graph index owner to rebuild brain_graph_nodes.
Inngest Connect WebSocket auth probe
joelclaw inngest connect-auth [--start-only] [--timeout-ms 8000] [--instance-id joelclaw-cli-probe] [--url http://localhost:8288]
Semantics:
- uses
INNGEST_SIGNING_KEYas Bearer auth against/v0/connect/start. - decodes the protobuf
StartResponse(connection_id,gateway_endpoint,gateway_group, session/sync token presence). - full mode runs the websocket handshake on the returned gateway endpoint:
- connects with subprotocol
v0.connect.inngest.com - expects
GATEWAY_HELLO - sends
WORKER_CONNECTwithsession_token+sync_token - expects
GATEWAY_CONNECTION_READY
- connects with subprotocol
- output never includes raw tokens (metadata only: length/format/JWT timestamps).
--start-onlyvalidates HTTP auth path without websocket handshake.
Command roots
joelclaw status--agent-dispatch-canaryruns the deterministic non-LLM timeout canary and folds its terminal truth into the status envelope
joelclaw summary [--hours N] [--format json|text]joelclaw runsjoelclaw runjoelclaw agentjoelclaw contentjoelclaw gatewayjoelclaw loopjoelclaw docsjoelclaw vaultjoelclaw skillsjoelclaw mailjoelclaw secretsjoelclaw logjoelclaw notifyjoelclaw oteljoelclaw o11yjoelclaw recalljoelclaw memoryjoelclaw subscribejoelclaw webhookjoelclaw inngestjoelclaw restatejoelclaw pdsjoelclaw knowledgejoelclaw capabilitiesjoelclaw queuejoelclaw workloadjoelclaw jobsjoelclaw fleet
Fleet command tree
joelclaw fleet
├── status [--host <alias>] [--config <private-manifest>]
└── diff [--host <alias>] [--config <private-manifest>]
Both commands are read-only and return the standard JSON envelope. status gathers declared-host facts; diff classifies declared expectations as in_sync, expected_difference, drift, or unavailable. The manifest is local private configuration: never commit or print its contents. Fleet plan/apply commands do not exist yet; use a reviewed, host-scoped change path only after explicit approval.
PDS command tree
joelclaw pds
├── health
├── describe
├── collections
├── records <collection> [--limit <n>]
├── write <collection> --data <json>
├── delete <collection> <rkey>
└── session [--refresh]
Semantics:
- targets the rebuilt host-published PDS at
http://localhost:9627 - caches sessions at
~/.joelclaw/pds-session.json - resolves the login identifier from
pds_joel_didby callingdescribeRepoand using the repo handle beforecreateSession, because the rebuilt PDS accepted the handle while rejecting raw DID login - returns token metadata in
sessionoutput without dumping raw JWTs writeauto-adds$typeandcreatedAt
Docs command tree (ADR-0234 docs pipeline v2)
joelclaw docs
├── add <path> [--title <title>] [--tags a,b,c] [--category <storage-category>]
├── search <query> [--limit <n>] [--category <category>] [--concept <concept-id>] [--chunk-type <section|snippet>] [--doc <doc-id>] [--semantic]
├── context <chunk-id> [--mode snippet-window|parent-section|section-neighborhood] [--before <n>] [--after <n>] [--neighbors <n>]
├── list [--category <category>] [--limit <n>]
├── show <doc-id>
├── markdown <doc-id>
├── summary <doc-id>
├── status
├── reconcile [--manifest <path>] [--sample <n>]
├── enrich <doc-id>
├── reindex [--doc <doc-id>]
├── reindex-v2 <path> [--title <title>] [--skip-existing]
└── batch-reindex [--from-collection] [--skip-existing]
Semantics:
- active chunk search collection is switchable via
DOCS_CHUNKS_COLLECTIONand now defaults todocs_chunks_v2 - markdown + summary read durable artifacts from
DOCS_ARTIFACTS_DIR(default/Volumes/services/joelclaw/docs-artifacts) docs statusreports bothdocs_chunks(v1) anddocs_chunks_v2plus artifact directory availability- agentic expansion flow is
docs search→docs context→docs markdown/docs summary
Skills command tree
joelclaw skills
├── ensure <skill>
│ [--source-root <repo>]
│ [--consumer all|agents|pi|claude]
└── audit [--deep] [--wait-ms <wait-ms>] [--poll-ms <poll-ms>]
joelclaw skills ensure semantics:
- compatibility-shim maintenance surface for skills that already live in a repo
skills/directory - resolves
skills/<name>/SKILL.mdfrom--source-root, cwd/ancestor repos, or the joelclaw repo fallback - creates missing flat per-skill symlinks and repairs wrong symlinks in
~/.agents/skills/,~/.pi/agent/skills/, and~/.claude/skills/ - expects those consumer roots to be real directories; whole-root symlinks are a layering error
- fails loudly if a consumer target exists as a real file/dir instead of a symlink
- returns the installed skill path so agents can
readit immediately - for external third-party skill packages, the CLI points at the upstream installer:
npx skills add -y -g <source>
Workload command tree
joelclaw workload
├── plan "<intent>"
│ [--preset docs-truth|research-compare|refactor-handoff]
│ [--kind auto|repo.patch|repo.refactor|repo.docs|repo.review|research.spike|runtime.proof|cross-repo.integration]
│ [--shape auto|serial|parallel|chained]
│ [--autonomy inline|supervised|afk|blocked]
│ [--proof none|dry-run|canary|soak|full]
│ [--risk reversible-only,host-okay]
│ [--artifacts patch,verification,summary]
│ [--acceptance "criterion one|criterion two"]
│ [--repo /abs/path/or/owner/repo]
│ [--paths docs/workloads.md,docs/cli.md]
│ [--paths-from status|head|recent:<n>]
│ [--stages-from /abs/path/to/stages.json]
│ [--write-plan ~/.joelclaw/workloads/]
│ [--requested-by Joel]
├── dispatch <plan-artifact>
│ [--stage <stage-id>]
│ [--to <to>]
│ [--from <from>]
│ [--send-mail]
│ [--write-dispatch ~/.joelclaw/workloads/]
├── run <plan-artifact>
│ [--stage <stage-id>]
│ [--tool pi|codex|claude]
│ [--execution-mode auto|host|sandbox]
│ [--sandbox-backend local|k8s]
│ [--sandbox-mode minimal|full]
│ [--skip-dep-check]
│ [--repo-url <repo-url>]
│ [--dry-run]
└── sandboxes
├── list [--state active|completed|failed|cancelled] [--mode minimal|full] [--expired] [--limit <n>]
├── cleanup [--request-id <id> | --sandbox-id <id> | --expired | --all-terminal] [--dry-run] [--force]
└── janitor [--dry-run]
joelclaw workload plan semantics:
- planner surface for ADR-0217 Phase 4.3
- returns a canonical
request+planenvelope usingdocs/workloads.md - also returns
guidancewith:recommendedExecution(execute-inline-now,tighten-scope-first,dispatch-after-health-check, etc.)operatorSummaryso the CLI says what to do next instead of shruggingadrCoverageto show which ADRs likely govern the slice already; on fresh repo-local ADR clusters it remains best-effort guidance and may still need human reconciliationrecommendedSkillswith install/read readiness, includingjoelclaw skills ensure <name>for local repo skills andnpx skills add -y -g <source>for external skillsexecutionExamplesfor serial / parallel / chained coding workloads, including setup + execution few-shot patternsexecutionLoopso the agent gets the honest plan → approve → execute/watch → summarize posture after the operator says yes
- infers
kind,shape,mode, andbackendwhen the caller leaves them open - supports reusable planner presets for common docs/research/refactor shapes
- preserves
Acceptance:clauses embedded in the prompt when--acceptanceis omitted - prefers implementation intent over docs follow-through, so mixed intents like
refactor ... then update docsorextend ... then update READMEstay implementation-shaped - validates known
riskandartifactsvalues and emits warnings for unknown ones instead of silently inventing vocabulary - mentioning sandboxes as the topic of a comparison does not force sandbox mode by itself; isolation has to be explicit or implied by AFK autonomy
deploy-allowedis inferred only from explicit release/deploy intent; nouns likepublished skillsdo not count as deploy requestsproof=canary|soakno longer forces supervised repo work ontodurable/restateby itself--paths-fromcan seed file scope from local git activity, and--write-planwrites the full envelope to a reusable JSON artifact--stages-from <file>loads an explicit JSON stage DAG, validates dependencies/cycles, carries per-stage acceptance into plan verification, and adds DAG metadata to the plan result- when
--shape autois still in effect, an explicit stage DAG now decides whether the plan isserial,parallel, orchained - chained repo.patch/refactor work can decompose a
Goal:section into explicit milestones and add a reflection/update stage when the prompt asks for it - defaults
--repoto the current working directory and infersbranch/baseShawhen that target is a local git repo; if the cwd is not a git repo, it warns and points the caller at--repo - does not execute code or mutate repos
joelclaw workload dispatch semantics:
- reads a saved plan artifact from
joelclaw workload plan --write-plan ... - turns it into a stage-specific dispatch/handoff contract with canonical
handoffdata plus a clawmail-ready subject/body - also returns dispatch
guidanceso the CLI can say whether the right move is to execute the stage now, keep the slice inline, or pause for a health check/recipient clarification - dispatch guidance also carries
executionLoopso the receiving agent knows the approval, progress-reporting, and closeout posture instead of inventing workflow theatre - defaults to the first stage, but
--stagecan target a later stage explicitly - preserves scoped file boundaries through
selectedStage.reservedPaths/handoff.reservedPaths - carries forward ADR coverage + recommended skill setup/readiness for the receiving agent
--write-dispatchwrites the dispatch contract as a reusable JSON artifact--send-mail --to <to> --from <from>sends that contract throughjoelclaw mail- does not execute code or mutate repos
joelclaw workload run semantics:
- reads a saved plan artifact and normalizes it into the canonical queue-backed runtime request
- emits the queue family
workload/requested, which the registry maps tosystem/agent.requested - current durable runtime path is
Redis queue → Restate dagOrchestrator → dagWorker dagWorkerhandlers currently covershell,infer, andmicrovm- defaults to
--tool pi, withcodex|claudeas explicit opt-ins - supports
--sandbox-backend local|k8splus--sandbox-mode minimal|fullwhen sandbox execution is the point - explicit-stage plans now gate stage execution on dependency inbox truth; use
--skip-dep-checkonly for deliberate manual recovery or replay - supports
--dry-runfor request inspection before queue admission - returns queue admission details once the request is enqueued
- if queue admission fails before the runtime request is accepted,
workload runnow writes a terminal inbox snapshot for thatrequestIdimmediately instead of leaving operators with no truth artifact to inspect status|explain|cancelremain planned, not shippedjoelclaw workload sandboxesis now the operator surface for ADR-0221 local sandbox state:listreconciles the registry against per-sandboxsandbox.jsonmetadata before reporting retention + filesystem truth, so operator output stops lying about terminal state after older partial writeback failurescleanupis the bounded manual deletion path with--dry-runand--force, and it performs the same reconciliation before deciding whether a sandbox is still activejanitoris the dedicated expired-sandbox cleanup path instead of waiting for startup-time opportunistic pruning, and it also reconciles registry drift before computing candidates- scheduled janitoring now lives in the repo-managed launchd service
com.joel.local-sandbox-janitor, which runsjoelclaw workload sandboxes janitorat load and every 30 minutes
Restate command tree
joelclaw restate
├── status [--namespace <namespace>] [--admin-url <url>]
├── deployments [--admin-url <url>] [--cli-bin <bin>]
├── smoke [--script <path>]
├── enrich "<name>" [--github <user>] [--twitter <user>] [--depth quick|full] [--sync]
├── pi-mono-sync [--repo <owner/repo>] [--full-backfill] [--max-pages <n>] [--sync]
└── cron
├── status [--namespace <namespace>] [--service-name <service>] [--base-url <url>]
├── list [--namespace <namespace>] [--service-name <service>] [--base-url <url>]
├── enable-health [--schedule "0 7 * * * *"] [--run-now] [--restate-url <url>]
└── delete <job>
joelclaw restate smoke semantics:
- resolves the smoke script path in this order:
- exact absolute path
- relative to current working directory
- relative to
JOELCLAW_ROOT - relative to
~/Code/joelhooks/joelclaw
- runs
scripts/restate/test-workflow.shby default. - default smoke validates
deployGateend-to-end. - DAG smoke is available via script override:
joelclaw restate smoke --script scripts/restate/test-dag-workflow.sh
joelclaw restate pi-mono-sync semantics:
- triggers the
pi-mono-syncRestate DAG pipeline, which runs the host-side direct task runnerscripts/restate/run-tier1-task.ts --task pi-mono-artifacts-sync. - syncs repo docs, issues, issue comments, pull requests, pull-request review comments, commits, and releases into the Typesense collection
pi_mono_artifacts. - writes two materialized documents into the same collection:
- maintainer profile (
kind=maintainer_profile, currently forbadlogic) - sync checkpoint (
kind=sync_state) so later runs can stay incremental unless--full-backfillis set.
- maintainer profile (
- default repo is
badlogic/pi-mono. --syncwaits for the DAG result; async mode returns a workflow ID and lets Restate finish in the background.- the new collection is queryable through
joelclaw search --collection pi_mono_artifacts.
joelclaw restate cron semantics:
- manages Dkron scheduler jobs for Restate pipelines.
- default access path is a short-lived CLI-managed
kubectl port-forwardtosvc/dkron-svc. - pass
--base-urlonly when you already have a direct Dkron API endpoint. enable-healthseeds the health proof job:restate-health-check.sync-tier1upserts the full ADR-0216 tier-1 set:restate-health-checkrestate-skill-gardenrestate-typesense-full-syncrestate-daily-digestrestate-subscription-check-feeds
listincludesmigratedFrom,successCount,errorCount,lastSuccess, andlastErrorso the soak is visible from the CLI without spelunking Dkron by hand.- the jobs use Dkron's shell executor plus
wget; it appends epoch seconds to each workflow ID prefix so every scheduled run is a fresh Restate workflow. - Dkron cron expressions are six-field by default (
sec min hour dom month dow), so hourly-at-minute-7 is0 7 * * * *, not7 * * * *.
Discover command
joelclaw discover <url> [-c <context>] [--site <site>] [--visibility <visibility>]
Semantics:
--sitechoices:joelclaw,wizardshit,shared--visibilitychoices:public,private,archived,migration-only- sensible defaults when omitted:
site=joelclaw,visibility=public - default path still emits
discovery/noteddirectly to Inngest. - when
QUEUE_PILOTS=discovery,joelclaw discovernow posts raw event intent to the worker admission endpoint (POST /internal/queue/enqueue) instead of writing Redis directly:- the worker owns queue admission, static registry lookup, and bounded triage mode resolution
- returns queue metadata (
streamId,eventId,priority) instead of Inngest run ids - includes
triageMode+triagemetadata whenever the family is enabled for shadow or enforce - relies on the Restate queue drainer to forward the event onward
- this keeps discovery pilot clients thin while the server remains the only queue policy surface.
joelclaw discoveris the thin fire-and-forget shortcut. If you need the final link for the created piece in the same turn, use the canonical follow path instead:
joelclaw send discovery/noted --data '{"url":"<url>","context":"<optional>","site":"joelclaw","visibility":"public"}' --follow
The terminal result from discovery-capture now includes finalLink. send --follow treats transient Inngest run lookup misses (for example immediate sql: no rows in result set after event send) as polling gaps, not fatal send failures: it keeps the event-sent result, emits a warning if final state cannot be confirmed, and points operators to joelclaw run / joelclaw runs for inspection.
Subscribe check queue pilot
joelclaw subscribe check [--id <id>]
Semantics:
- scoped checks (
--id <id>) still emitsubscription/check.requesteddirectly to Inngest. - all-subscription checks keep the legacy Inngest path by default.
- when
QUEUE_PILOTS=subscriptions,joelclaw subscribe checkwithout--idpostssubscription/check-feeds.requestedto the worker admission endpoint instead of writing Redis directly:- returns queue metadata (
streamId,eventId,priority) - includes
triageMode+triagemetadata whenever the family is enabled for shadow or enforce - points next actions at
joelclaw queue inspect/joelclaw queue depth - relies on the Restate queue drainer to forward the actual
subscription/check-feeds.requestedevent name onward
- returns queue metadata (
Jobs command tree (ADR-0217 runtime monitor)
joelclaw jobs
└── status [--hours <n>] [--count <n>] [--namespace <namespace>] [--restate-admin-url <url>] [--dkron-service-name <service>] [--dkron-base-url <url>]
Semantics:
jobs statusis the first operator glance for real workloads during the ADR-0217 transition. It aggregates the queue/Redis substrate, Restate runtime, Dkron scheduler, and still-live Inngest surfaces into one JSON snapshot.overall.statusis a bounded truth surface (healthy|degraded|down) derived from those four components, not a raw dump of every underlying health probe.- queue section:
- reads the canonical Redis queue directly through
@joelclaw/queue - reports depth, priority buckets, oldest age, and active deterministic pauses
- reads the canonical Redis queue directly through
- Restate section:
- mirrors
joelclaw restate statusin-place so the operator can see statefulset readiness + admin health without command hopping
- mirrors
- Dkron section:
- mirrors
joelclaw restate cron statusin-place and includes the count of Restate-tagged scheduler jobs when the API is reachable
- mirrors
- Inngest section:
- stays visible during migration, but its status is scoped to the transitional job path that still matters here: server/worker health plus recent run outcomes
- broad informational checks (for example unrelated k8s pod drift) stay in the payload without poisoning the top-level job monitor status
- next actions point directly at the management surfaces that actually move workload state:
queue control status,queue observe,queue resume,restate status,restate cron status, andruns
Pi async jobs monitor (runtime_jobs_monitor)
The loaded pi extension at packages/pi-extensions/inngest-monitor/index.ts now does two jobs:
- tracks followed Inngest runs (
inngest_send,inngest_runs) - exposes
runtime_jobs_monitorfor the ADR-0217 runtime substrate
runtime_jobs_monitor semantics:
action=start|status|stop(defaultstart)- on
start, it pollsjoelclaw jobs statusin the background, paints a persistent widget, emits OTEL on severity changes and meaningful workload-state changes, and sends hidden follow-up messages for async report-back - on
status, it returns the latest runtime snapshot (overall status, queue depth, active pause count, Restate/Dkron/Inngest state) - on
stop, it stops the poller and sends a final follow-up summary - widget posture is intentionally operator-first: current runtime state on top, active followed runs underneath
This is the canonical async monitoring path when you want a pi session to keep an eye on real workloads while you do other things.
Queue command tree (ADR-0217 Phase 1)
joelclaw queue
├── emit <event> [-d <json>] [-p P0|P1|P2|P3]
├── depth
├── stats [--hours <n>] [--limit <n>]
├── observe [--hours <n>] [--limit <n>] [--since <iso|ms>]
├── pause <family> [--ttl <duration>] [--reason <text>]
├── resume <family> [--reason <text>]
├── control
│ └── status [--hours <n>] [--limit <n>] [--since <iso|ms>]
├── list [--limit <n>]
└── inspect <stream-id>
Semantics:
- all queue subcommands return a clean JSON envelope; read-oriented subcommands close their Redis client before exit and
emitstays a thin worker client instead of writing Redis directly - queue state/control commands resolve Redis from the canonical joelclaw CLI config (
~/.config/system-bus.env→REDIS_URL) before considering ambient shell env so the installed operator surface stays pointed at the same localhost queue as the worker and Restate drainer emitposts queue admission intent to the worker endpoint (POST /internal/queue/enqueue).- accepts event name (e.g.,
discovery/noted,content/updated) - accepts optional JSON payload via
-d - optional priority override via
-pis normalized client-side, but static registry routing and bounded triage stay server-side - the worker generates the canonical
QueueEventEnvelope, adds trace metadata, resolves shadow/enforce mode for the event family, evaluates bounded triage, and persists the queue record - returns the Redis stream ID, priority, and any
triageMode/triagemetadata from admission
- accepts event name (e.g.,
depthreports queue depth, priority distribution (P0/P1/P2/P3 counts), oldest/newest message timestampsstatssummarizes recent Restate queue-drainer behavior plus Phase 2 triage behavior from OTEL over a lookback window.- dispatch section reports sampled/found dispatch events, live queue depth, started/completed/failed counts, success rate, queue wait-time percentiles (
p50/p95), dispatch-duration percentiles, promotion count, top event families, and recent failures - triage section reports attempts, completed/failed/fallback counts, fallback counts by reason, disagreement count, applied-change count, suggested-not-applied count, route mismatches, latency percentiles, per-family rollups, and recent mismatch/fallback samples
- uses
metadata.waitTimeMsfromqueue.dispatch.startedas the Story 5 queue-to-dispatch latency signal - uses
queue.triage.*OTEL metadata as the Story 3 source of truth for queue-admission disagreements and fallback behavior --since <iso|ms>overrides the lower bound so operators can anchor soak evidence to a known clean point (for example the supervisedqueue.drainer.startedafter a rollout) instead of mixing fresh traffic with a dirty pre-fix window- keeps the operator in CLI-land; no raw Redis keys or manual OTEL spelunking required for the first sanity pass
- dispatch section reports sampled/found dispatch events, live queue depth, started/completed/failed counts, success rate, queue wait-time percentiles (
jobs statusis the first unified runtime view; drop toqueuesubcommands when the aggregated surface says the queue itself needs attention.observeis now the Phase 3 Story 2-4 dry-run Sonnet operator surface.- builds a canonical live snapshot from current queue depth + queued messages + recent drainer OTEL + recent triage OTEL + gateway sleep/muted-channel state + active deterministic pauses
- runs the bounded Sonnet observer in
dry-runmode only and returns the currentsnapshotplus the currentdecision - long Sonnet summaries are trimmed instead of turning otherwise-useful observations into schema-only fallbacks
- if all queued work is intentionally held behind fresh active manual pauses and no recent failures suggest downstream trouble, it short-circuits to a deterministic
noopinstead of wasting a 60s Sonnet call on an obvious hold state historysummarizes recentqueue.observe.*OTEL for the same window so operators can compare the latest dry-run against raw history without spelunking Typesense by handcontrolnow reflects the shipped deterministic queue-control plane: active manual pauses,queue.control.applied|expired|rejectedcounts, and recent control events come from the same Redis + OTEL truth the drainer uses--since <iso|ms>anchors the related OTEL history window the same wayqueue statsdoes
pauseapplies a deterministic manual family pause with bounded TTL and emitsqueue.control.appliedtelemetryresumeclears a deterministic family pause and emits eitherqueue.control.appliedorqueue.control.rejectedwhen the family was not pausedcontrol statusis the dedicated deterministic control-plane operator surface.- reports active pauses (family, reason, TTL, applied/expiry timestamps, actor)
- summarizes
queue.control.applied|expired|rejectedOTEL for the same window - is the first CLI answer to “what queue controls are active right now?” before any automatic Sonnet mutation ships
listlists recent messages in priority order (highest priority first), does not ack/removeinspectloads a message by Redis stream ID and returns full payload + metadata- if the message is already acked/expired, it now returns a structured
QUEUE_MESSAGE_MISSINGerror envelope with queue-state next actions instead of crashing the CLI
- if the message is already acked/expired, it now returns a structured
Queue configuration:
- Stream key:
joelclaw:queue:events - Priority index:
joelclaw:queue:priority - Consumer group:
joelclaw:queue:cli - Phase 1 pilot events:
discovery/noted,discovery/captured,content/updated,subscription/check-feeds.requested,github/workflow_run.completed
OTEL telemetry under queue.* namespace:
- CLI queue commands forward queue package telemetry to OTEL with
source=cliandcomponent=queue queue.enqueue— message enqueuedqueue.lease— message leased for processing (includes wait time, priority, promotion metadata)queue.ack— message acknowledgedqueue.replay— unacked messages loaded for replay
Daily summary command
joelclaw summary [--hours 24] [--format json|text]
Semantics:
- aggregates recent git activity (joelclaw + Vault), Inngest run rollups, k8s pod health, OTEL stats, slog deploy/config events, and ADR churn into one response envelope.
- default output is JSON summary payload;
--format textadds a compact text rendition underresult.textfor downstream chat/mobile surfaces.
Memory command group
joelclaw recall "<query>"
joelclaw recall --request-file -
joelclaw memory review --since 48h
joelclaw memory search "<legacy query>" # typed retirement response; no search occurs
joelclaw memory write "<legacy input>" # typed retirement response; no write occurs
Semantics:
recall "<query>"is the canonical interactive composed-recall surface.recall --request-file -accepts an exact composed-recall request on stdin.memory reviewreturns recent accepted-session, Git, Brain, OTEL, and flowing evidence in separate lanes.memory searchis a fail-closed compatibility pointer torecall; it returnsMEMORY_SEARCH_RETIREDwith exit3.memory writeis a fail-closed compatibility pointer. Agent work enters through accepted Runs; durable curation belongs in Brain.svxpages.
Knowledge turn-write command
joelclaw knowledge note \
--source gateway \
--agent gateway-daemon \
--session <session-id> \
--turn <turn-number> \
--summary "<what changed>"
Use --skip-reason routine-heartbeat|duplicate-signal|no-new-information when a turn is eligible but has no durable signal to capture.
Gateway channel control
joelclaw gateway channel list
joelclaw gateway channel status [channel]
joelclaw gateway channel disable <channel> [--restart] [--force]
joelclaw gateway channel enable <channel> [--restart]
Semantics:
- edits
~/.joelclaw/scripts/gateway-start.shfor the known runtime channels:telegram,discord,imessage,slack. - writes a
/tmp/joelclaw/gateway-start.sh.*backup before changing the script. - never prints secret values; output reports variable names plus source classes (
secret,derived,literal,blank,missing). --restartapplies the change immediately viajoelclaw gateway restart; otherwise next actions tell the agent to restart.- disabling
telegramrequires--forcebecause it is a primary operator channel.
Gateway known issues / muted channels
joelclaw gateway known-issues
joelclaw gateway mute <channel> [--reason "<why muted>"]
joelclaw gateway unmute <channel>
Semantics:
- stores muted channel IDs at Redis key
gateway:health:muted-channels(JSON array). - stores optional mute reasons at Redis key
gateway:health:mute-reasons(JSON object). - muted channels remain in probe telemetry but are excluded from
gateway.channels.degradedalerts.
Gateway process-layer diagnostics (joelclaw gateway diagnose) now inspect exact launchd state for com.joel.gateway and report disabled services explicitly. joelclaw gateway restart now re-enables the launch agent before bootstrap/kickstart to avoid restart failures when launchd has the service disabled.
Use joelclaw gateway enable for direct launch-agent recovery (enable + bootstrap + kickstart) without manual launchctl usage.
Gateway status/diagnose now separate daemon availability from Redis bridge health:
joelclaw gateway statusprefers daemon/healthwhen available, so Redis loss no longer makes the gateway look fully dead.- status now returns
mode(normalorredis_degraded),degradedCapabilities,sessionPressure(context %, compaction age, session age, next action), andguardrails(current turn tool budget state + pending deploy verifications). joelclaw gateway diagnosetreatsredis_degradedas a degraded runtime, not a process failure, skips Redis-dependent E2E checks in that mode, and surfaces runtime guardrail findings when a checkpoint/deploy verification is active.
Gateway behavior control plane (ADR-0211)
joelclaw gateway behavior add --type keep|more|less|stop|start --text "..."
joelclaw gateway behavior list
joelclaw gateway behavior promote --id <candidate-id>
joelclaw gateway behavior remove --id <directive-id>
joelclaw gateway behavior apply
joelclaw gateway behavior stats
Semantics:
- Single write authority is CLI. Extensions must call these commands; no direct Redis/Typesense writes.
- Active runtime contract lives in Redis key
joelclaw:gateway:behavior:contract. - Directive/candidate history lives in Typesense collection
gateway_behavior_history. addnormalizes directives, enforces conflict + dedupe + cap rules, and updates Redis + Typesense.promotemoves a pending daily-review candidate into the active contract (manual gate; no auto-activation).applyre-runs governance over the active contract (dedupe, conflict cleanup, cap enforcement) and expires stale candidates.statsreports contract hash/version, candidate lifecycle counts, and governance settings.
Run listing semantics
joelclaw runs [--status RUNNING|FAILED|COMPLETED|QUEUED|CANCELLED]
Semantics:
- applies backend status filtering and a local status guard so mixed-status payloads cannot leak through.
- the runs list query uses a longer timeout budget than generic Inngest GraphQL calls because self-hosted
runs(...)can take ~50s even for small filtered windows; timeout scales upward with requestedcount, and timeout aborts get one bounded retry with a larger budget before the CLI gives up. - for suspicious
RUNNINGrows (endedAt present while running, or long-running health checks), runs performs bounded detail reconciliation (max 5 lookups) to detect stale SDK-unreachable ghosts. - runs with stale indicators include
staleSignalwithlikely,confidence, and machine-readablereasons. - response includes
staleSignalssummary (detected,likely,detailChecked) and suggestsjoelclaw inngest sweep-stale-runswhen likely ghosts are present. countin response reflects post-filter rows (what the operator actually sees).
Run inspection + cancellation
joelclaw run <run-id> [--cancel] [--wait-ms 3000]
Semantics:
- default mode returns run detail, trigger event, trace, and step errors.
- run detail and
joelclaw event <event-id>now use a much larger timeout/retry budget than generic GraphQL calls because self-hosted detail lookups can take more than a minute under load even while the runtime itself is healthy. --cancelissues Inngest GraphQLcancelRunfor active runs, then polls status up to--wait-ms.- deterministic error envelopes:
RUN_CANCEL_FAILEDwhencancelRunmutation fails.RUN_CANCEL_TIMEOUTwhen run remainsRUNNING|QUEUEDafter the wait window.RUN_STALE_SDK_UNREACHABLEwhen a run appearsRUNNINGbut trace errors showUnable to reach SDK URLand cancellation has no live execution to target.
- terminal runs are never re-cancelled; response includes
cancellation.skipped = "already_terminal".
Capability adapter config precedence (ADR-0169 phase 0)
Resolution order is deterministic:
- CLI flags (e.g.
--adapter) - Environment variables
- Project config (
.joelclaw/config.toml) - User config (
~/.joelclaw/config.toml) - Built-in defaults
Current env keys:
JOELCLAW_CAPABILITY_<CAPABILITY>_ADAPTERJOELCLAW_CAPABILITY_<CAPABILITY>_ENABLED
Capability-backed command roots (ADR-0169 through phase 4)
joelclaw secrets status
joelclaw secrets lease <name> --ttl 15m
joelclaw secrets revoke <lease-id>
joelclaw secrets revoke --all
joelclaw secrets audit --tail 50
joelclaw secrets env --dry-run [--ttl 1h] [--force]
joelclaw log write --action <action> --tool <tool> --detail <detail> [--reason <reason>] [--session <session>] [--system <system>]
joelclaw notify send "<message>" [--kind memory|alert|digest|ask|receipt] [--priority low|normal|high|urgent|critical] [--channel gateway|main|all] [--context '{"k":"v"}']
joelclaw notify wait <event-id> --source <source> [--timeout 15s]
joelclaw messages trace <flowId>
joelclaw heal {list|run}
joelclaw mail {status|register|send|inbox|read|reserve|renew|release|locks|search}
joelclaw otel {list|search|stats|emit}
joelclaw o11y {session|system}
joelclaw recall <query> [--limit N] [--min-score F] [--raw] [--include-hold] [--include-discard] [--budget auto|lean|balanced|deep] [--category <id|alias>]
joelclaw sessions search <query> [--source typesense|local|ssh|both] [--runtime pi|codex|claude-code|all] [--extract] [--machine dark-wizard] [--ssh-target joel@dark-wizard] [--limit N]
joelclaw session search <query> ... # singular alias
joelclaw sessions extract <session-id-or-path> --query <topic> [--format json|markdown]
joelclaw sessions chunks <query> [--source typesense|local|both] [--limit N] [--context-before N] [--context-after N]
joelclaw sessions inspect <session-id-or-path> --around <regex> [--before N] [--after N]
joelclaw sessions signals [--kind friction|preference|decision|praise|mode-mismatch|any] [--source local|ssh|both] [--machine dark-wizard] [--since 14d] [--limit N] [--sample N] [--review-out path.jsonl] [--evaluate] [--format json|ndjson]
joelclaw sessions friction [--source local|ssh|both] [--machine dark-wizard] [--since 14d] [--limit N]
joelclaw satellite health [--notify] [--central-ssh joel@panda] [--priority high]
joelclaw satellite repair-request [--central-ssh joel@panda] [--priority high]
joelclaw subscribe {list|add|remove|check|summary}
Semantics:
logwrites structured system entries (slog backend).log writenow accepts explicit--session/--systemprovenance flags and also falls back toSLOG_SESSION_ID/SLOG_SYSTEM_IDenv vars before handing off to the slog backend.logsreads/analyzes runtime logs.notifyis the canonical producer command for messages to Joel;gateway pushremains transport/debug. Plain text is a complete payload. Producers do not need to add or change flags for the Agent Comms Gateway.- Message text,
--kind,--priority,--channel,--telegram-only, and structured evidence are evidence for the gateway agent. They never instruct routing, urgency, timing, formatting, batching, suppression, or platform choice. Old flags remain accepted without deprecation nagging. - The current CLI accepts structured JSON evidence through
--context. The producer-facts contract reserves--data <json>and repeated--ref, butjoelclaw notify senddoes not implement those flags yet. Add run IDs, receipts, links, or available actions only when they help the agent make or verify its decision. - The gateway agent records one
gateway.decision.recordedreceipt for each consumed external event before advancing its stream cursor. Recordeddeliverandaggregate/close-deliverdecisions are executed mechanically by the slim transport. The current decision executor delivers only to Telegram. messages trace <flowId>reads the stream and delivery lifecycle. Correlate replies, reactions, button taps, decisions, and platform receipts byflowId.deploy,heal,log,notify,secrets,mail,otel,recall, andsubscribekeep their existing UX/envelopes while executing through capability registry adapters (scripted-deploy,runbook-heal,slog-cli,gateway-redis,agent-secrets-cli,mcp-agent-mail,typesense-otel,typesense-recall,redis-subscriptions).typesense-otel,typesense-recall,scripted-deploy,runbook-heal,slog-cli,agent-secrets-cli,gateway-redis,mcp-agent-mail, andredis-subscriptionsadapter logic is canonical in@joelclaw/sdk(packages/sdk/src/capabilities/adapters/*); CLI adapter files are thin re-exports.otel emitaccepts stdin JSON payloads (or convenience args/positional action), normalizes defaults (id,timestamp,level=info,success=true), and forwards to the worker ingest endpoint (/observability/emit). Provenance defaults are now role-aware: explicit event/session fields win, thenSLOG_SESSION_ID/SLOG_SYSTEM_ID, then session handle envs when present, then role/hostname fallbacks (gatewayorJOELCLAW_ROLE, system hostname stripped topanda).otel listandotel searchaccept exact--session/--systemfilters, mapped tosessionId/systemIdin Typesense.knowledge searchnow auto-heals the common post-rebuild failure mode wheresystem_knowledgeis missing: on a404 Collection not foundresponse it creates the collection, reindexes ADRs + skills once, retries the query, and reports the repair in-band.o11y session/o11y systemrun a unified multi-search acrossotel_eventsandsystem_log, merge both timelines bytimestamp, and tag each hit with its source collection.sessions searchis explicit raw-history drill-down. It reads native transcript adapters and the current local SQLite projection when available;--source localnever depends on retired Typesense Run collections. Use--runtime pi|codex|claude-code|alland verify the returned native path/runtime before claiming capture success. Remote index pointers are compatibility evidence, not a memory lane.sessions search --extractruns bounded deterministic extraction for top raw hits so recovery prompts can include decisions, commands, files, receipts, verification, blockers, next actions, and exact transcript line pointers without dumping the full JSONL. Each emitted hit carries its ownextractionwhen available;.result.extractionsremains as a backward-compatible convenience list. Large session envelopes are written through direct stdout, not Effect Console, so| jqremains safe above 64KB.sessions extractreads one raw local transcript by path or session-id substring, redacts likely secrets, and returns JSON extraction fields.--format markdownstill uses the standard JSON envelope and places rendered markdown at.result.markdown; raw markdown stdout would violate the CLI JSON contract.sessions chunksexposes matching Typesense chunks and/or raw local neighboring transcript context. Use top-level.result.chunks/.result.hits; source-specific mirrors remain under.result.local.chunksand.result.typesense.chunks.- Source metadata uses
rawReturnedfor pre-dedupe raw source count andemittedHits/emittedChunksfor top-level emitted counts. joelclaw session ...is a singular alias forjoelclaw sessions ....sessions signalsmines role-aware high-signal user turns from raw local/remote Pi transcripts. It is ADR-0247-backed and starts deterministic: user turns are the primary signal source; assistant/tool turns are bounded evidence context.--kind frictiondetects correction/frustration/output-quality signals, including Joel's strong-emphasis profanity where it is attached to critique.sessions frictionis an alias forsessions signals --kind friction.--sample Nbalances candidates across signal categories;--review-out path.jsonlwrites a tiny golden-set review file with stablehitId, turnKind, prediction, improvement route, evidence, and empty review fields.--evaluateadds a small routing summary by improvement surface/kind/turnKind/confidence/review priority. V1 excludes task payloads, source material, and handoffs by default so friction analysis focuses on operator intent, review feedback, and approvals.--kind mode-mismatchfinds cases where agents picked the wrong execution shape: inline vs background, visual vs response, durable Inngest vs ad-hoc script, or feedback-blocking vs async.--format ndjsonstreamsmeta,hit, andsummaryrows for deep scans without building one huge envelope.satellite healthis the thin-Machine self-check surface. It verifies local prerequisites likepython3,jq,joelclaw, raw Pi session storage, and local session search JSON. With--notify, degraded health is relayed to Central via SSH (--central-ssh, defaultjoel@panda) usingjoelclaw notify send --type satellite/repair.requested; this lets a satellite ask the gateway for repair without local Redis/secrets.satellite repair-requestalways sends the Central gateway a repair request with the local probe payload.- New satellite Machines should use
scripts/setup-satellite-rig.shplusdocs/runbooks/satellite-rig-setup.md. The bootstrap keeps satellites thin: repo + CLI + skills + raw session search + Central repair relay. It writes~/.local/bin/joelclawas a wrapper that sources~/.config/system-bus.env, so Typesense and PDF Brain docs-api credentials work over non-interactive SSH without installing Panda's agent-secrets stack. Do not install Panda's k8s/Inngest/Redis/gateway stack on a satellite by default. - Software surfaces should route OTEL through this command contract (or shared CLI ingest helper), not ad-hoc raw HTTP calls.
mail searchauto-falls back to/mail/api/unified-inboxfiltering when MCPsearch_messagesreturns transient DB/tool errors, so steering signals remain usable.mail reservenow sends explicit lease TTL (--ttl-seconds, default900) and enforces a minimum of 60s.mail renewextends active file reservations without releasing/reacquiring (--extend-seconds, default900, optional--paths).mail locksnow prefers the local git-mailboxfile_reservations/artifact store when available because/mail/api/lockscan under-report advisory file reservations while still reporting mailbox internals like archive/commit locks. Responses exposesourceandfallback_reasonwhen artifact fallback was required.subscribe checkemits Inngest request events for scoped checks and for all-subscription checks when the queue pilot is off;response.idsare event/request IDs (inspect viajoelclaw event <event-id>), not run IDs unless explicitly returned asrunIds.- when
QUEUE_PILOTS=subscriptions, unscopedsubscribe checkreturns queue metadata instead of Inngest response ids because the request first lands in Redis and is forwarded by the Restate drainer. recallrewrite telemetry now exposesrewrite.strategy(disabled|skipped|haiku|openai|fallback) andrewrite.reasonso low-ROI rewrite skips/fallbacks are queryable; short/literal/direct-id queries skip LLM rewrite by design. The default rewrite model isopenai-codex/gpt-5.6-sol; setJOELCLAW_RECALL_REWRITE_MODELonly to an approved model.
Webhook command tree (ADR-0185)
joelclaw webhook
├── subscribe <provider> <event>
│ [--repo <owner/repo>] [--workflow <name>] [--branch <name>] [--conclusion <status>]
│ [--session <session-id>] [--ttl <duration>] [--stream] [--timeout <seconds>] [--replay <count>]
├── unsubscribe <subscription-id>
├── list [--provider <provider>] [--event <event>] [--session <session-id>]
└── stream <subscription-id> [--timeout <seconds>] [--replay <count>]
Semantics:
- Subscriptions are Redis-backed and session-scoped (
joelclaw:webhook:*). subscribe --streamstarts an NDJSON stream immediately after creation.streamemits ADR-0058 NDJSON (start,log,event, terminalresult|error).- Default session target is
gatewayfor central gateway role, otherwisepid-<ppid>. - TTL defaults to
24hand is enforced at match time.
Skills command tree (ADR-0179)
joelclaw skills
└── audit [--deep] [--wait-ms <wait-ms>] [--poll-ms <poll-ms>]
joelclaw skills audit purpose
- triggers the
skill-garden/checkevent on-demand - waits for the corresponding run and returns the findings report in-envelope
- supports
--deepfor LLM staleness checks
Agent command tree (ADR-0180 phases 2-4)
joelclaw agent
├── list
├── show <name>
├── run <name> <task> [--cwd <cwd>] [--timeout <seconds>]
├── chain <steps> --task <task> [--cwd <cwd>] [--fail-fast]
└── watch <id> [--timeout <seconds>]
Semantics:
runemitsagent/task.runfor single roster agent execution and returnstaskIdpluseventIdsfrom the Inngest send response.runnext_actionsare truthful: usejoelclaw event <event-id>when an event ID exists (orjoelclaw events ...fallback), and never assumetaskIdis a run ID.chainemitsagent/chain.runwith comma-separated sequential steps and+parallel groups (e.g.scout,planner+reviewer,coder).watchstreams NDJSON progress for a task (at-...) or chain (ac-...) by subscribing tojoelclaw:notify:gateway, replayingjoelclaw:events:gateway, and falling back to Inngest polling.watchdefault timeout is 300 seconds for tasks and 900 seconds for chains; terminal events always includenext_actionson completion, timeout, or interrupt.- Runtime-proof recipe (ADR-0180):
joelclaw agent list(expect builtincoder/designer/ops/story-executor)joelclaw agent run coder "reply with OK" --timeout 20joelclaw event <event-id>(expectAgent Task RunstatusCOMPLETEDwith output payload)
- If
Unknown agent roster entry: coderappears, treat it as worker-runtime drift: deploy latestsystem-bus-worker, restart the host worker, then rerun the three-step proof.
Vault command tree
joelclaw vault
├── read <ref>
├── search <query> [--semantic] [--limit <limit>]
├── ls [section]
├── tree
└── adr
├── list [--status <status>] [--limit <limit>]
├── collisions
├── audit
├── locate <query> [--limit <limit>]
├── refs <query>
├── prompt <text>
└── rank [--band <band>] [--unscored] [--all]
joelclaw vault adr purpose
list— inventory ADR metadata with optional status filtercollisions— detect duplicate ADR numeric prefixesaudit— full ADR hygiene check:- missing/non-canonical status values
- number collisions
- missing
superseded-bytargets - broken or ambiguous wiki links inside ADR bodies (skips custom directive tags like
[[tts:text]]) - README index alignment against ADR files
locate— section/file lookup across the ADR corpus using ADR numbers, slugs, titles, and heading namesrefs— backlink discovery for a resolved ADR file or section idprompt— expand[[ADR refs]]in prompt text into canonical ids and append an<adr-context>block for agentsrank— score + rank ADRs by NRC+novelty rubric for daily prioritization:- default scope: open ADRs (
accepted+proposed) --allincludes shipped/superseded/deprecated/rejected ADRs--band <band>filters ranked rows (do-now|do-next|de-risk|park; aliasnext→do-next)--unscoredreturns ADRs missingpriority-score- required axes:
priority-need,priority-readiness,priority-confidence - novelty facet:
priority-novelty(or aliaspriority-interest), defaults to neutral3when missing - score formula:
clamp(round(20*(0.5*Need + 0.3*Readiness + 0.2*Confidence)) + round((Novelty-3)*5), 0, 100) - bands:
do-now(80-100),do-next(60-79),de-risk(40-59),park(0-39) - emits CLI OTEL via
component=vault-cli:vault.adr.rank.startedvault.adr.rank.completedvault.adr.rank.failed
- default scope: open ADRs (
Canonical statuses:
proposedacceptedshippedsupersededdeprecatedrejected
Content command tree (ADR-0168)
joelclaw content
├── seed
├── verify
└── prune [--apply]
Semantics:
seed— full Vault ADR sync to Convex for canonical ADR filenames only (NNNN-*.md).verify— strict ADR drift check against canonical ADR files (fails healthy state on both missing and extra ADR records in Convex).prune— dry-run report of Convex ADR extras (status: dry_run).prune --apply— removes ADR extras from Convex (status: pruned) and should be followed byjoelclaw content verify.
Inngest source guard (ADR-0089)
joelclaw inngest source [--repair]
Semantics:
- Verifies launchd binding for
com.joel.system-bus-workeragainst the canonicalinfra/launchd/com.joel.system-bus-worker.plistvalues (program + working directory). --repaircopies canonical plist into~/Library/LaunchAgents, performslaunchctl bootout, thenbootstrapwith retry for transientBootstrap failed: 5launchd races.- Use before
joelclaw inngest restart-workerwhen host runtime/source drift is suspected.
Inngest stale-run sweep (ADR-0194)
joelclaw inngest sweep-stale-runs
joelclaw inngest sweep-stale-runs --apply
Semantics:
- preview-first by default (
--applyrequired for mutation). - scope defaults to stale health checks older than 30 minutes:
check/o11y-triagecheck/system-health
- runtime target defaults:
- namespace
joelclaw - pod
inngest-0 - sqlite path
/data/main.db
- namespace
- apply mode safety gates:
- refuses if age threshold is too young (
<5m) - refuses when candidate count exceeds
--max-apply-candidates - refuses when
function_runsrows are missing (cannot insert terminal history safely) - always creates and verifies point-in-time backup before transaction:
/data/main.db.pre-sweep-<UTC-stamp>.sqlite
- refuses if age threshold is too young (
- terminalization contract (single transaction):
- insert missing
history.type = FunctionCancelled - insert missing
function_finishes - set
trace_runs.status = 500+ terminalended_at
- insert missing
This command exists for cases where Inngest API cancellation returns not found for stale RUNNING ghosts after SDK reachability failures.
Status command
joelclaw status [--agent-dispatch-canary]
Semantics:
- default
joelclaw statusremains the fast base worker/server health surface - default output now also includes
latestAgentDispatchCanarywhen a persisted deterministic canary snapshot exists, so operators can see the last proof result without spelunking runs or inbox files --agent-dispatch-canaryrunsscripts/verify-agent-dispatch-timeout.tsand folds the deterministic non-LLMsystem/agent-dispatchtimeout proof into the returned envelope- when requested, the command only reports healthy if both the base health probes and the canary pass
- this is the canonical on-demand proof surface for the live outer-timeout closeout path; it exists so operators do not have to run the verifier script manually
Build and verify
bunx tsc --noEmit
pnpm biome check packages/ apps/
bun test packages/cli/src/commands/*.test.ts
bun build packages/cli/src/cli.ts --compile --outfile ~/.bun/bin/joelclaw
joelclaw status
joelclaw vault
joelclaw vault adr audit
joelclaw vault adr locate "gateway guardrails"
joelclaw vault adr refs 0189-gateway-guardrails
joelclaw vault adr prompt "compare [[0189-gateway-guardrails]] with [[0218-gateway-availability-lifecycle-qol-improvements]]"
Add a command
- Create command module in
packages/cli/src/commands/. - Return envelopes with
respond/respondErroronly. - Include useful
next_actionswith param hints. - Wire command in
packages/cli/src/cli.ts. - Add/extend tests in
packages/cli/src/commands/*.test.ts. - Update this file when command tree or contracts change.