mirror of
https://github.com/ghostsecurity/skills.git
synced 2026-09-14 20:00:39 +08:00
GHO-11947: Add the ghost-exo skill for the exo workflow lifecycle (#19)
This commit is contained in:
Brad Geesaman
@@ -42,6 +42,7 @@ Full documentation, tutorials, and video guides at [ghostsecurity.ai](https://gh
|
||||
| `ghost-report` | Combined security report across all scan results |
|
||||
| `ghost-validate` | Dynamic validation of findings against a live application (DAST) |
|
||||
| `ghost-proxy` | HTTP proxy for the `ghost-validate` skill |
|
||||
| `ghost-exo` | Build, improve, and debug workflows on the exo agent orchestration platform |
|
||||
|
||||
### ghost-repo-context
|
||||
<div align="center">
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "ghost",
|
||||
"description": "Enterprise grade AI-native application security scanning, validation, and remediation",
|
||||
"version": "1.1.3",
|
||||
"version": "1.2.0",
|
||||
"author": {
|
||||
"name": "Ghost Security",
|
||||
"email": "oss@ghost.security",
|
||||
@@ -28,6 +28,9 @@
|
||||
"wraith",
|
||||
"poltergeist",
|
||||
"ghost-security",
|
||||
"ghost"
|
||||
"ghost",
|
||||
"exo",
|
||||
"workflow",
|
||||
"orchestration"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
# Changelog
|
||||
|
||||
## 1.2.0 - 2026-08-26
|
||||
|
||||
Adds the `ghost-exo` skill for the exo workflow lifecycle.
|
||||
|
||||
### Skills
|
||||
|
||||
- **ghost-exo** - One front door for building, improving, and debugging workflows on [exo](https://ghostsecurity.ai), an agent orchestration platform. Routes a request to one of three intents. BUILD takes a rough idea through interrogation, assessment, and resource creation in dependency order. IMPROVE runs the observe-and-iterate loop over recent runs behind two approval gates. DEBUG diagnoses a single failed run by walking the dependency graph of everything it touched. Bundles `scripts/exo-skill.py`, which moves skill bundles over REST so file contents never serialize as tool-call arguments. Harness-neutral: the bundle names no harness, and on first use it registers the exo MCP server for whichever harness is running it.
|
||||
|
||||
## 1.1.0 - 2026-02-17
|
||||
|
||||
Plugin naming convention and compliance/passing scores according to tessl best practices. Instead of **plugin:skill-name** it's **plugin-skill-name**. When installed as a plugin in Claude Code, it will be invocable by **ghost-skill-name** as well as **plugin:skill-name**
|
||||
|
||||
@@ -45,6 +45,14 @@ Full documentation, tutorials, and video usage guides are available at [ghostsec
|
||||
/ghost-validate # Dynamic/live validation against a live application (DAST)
|
||||
```
|
||||
|
||||
## Exo Workflows
|
||||
|
||||
`ghost-exo` drives [exo](https://ghostsecurity.ai), an agent orchestration platform. Connect your exo MCP server, then:
|
||||
|
||||
```
|
||||
/ghost-exo # Turn an idea into a workflow, iterate on an existing one, or diagnose a failed run
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please open a pull request or issue on this repository.
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: "ghost-exo"
|
||||
description: The single interface for building, improving, and debugging exo workflows. Routes to one of three intents. BUILD takes a rough idea through interrogation, assessment, resource creation in dependency order, and one manual run, then hands off. IMPROVE runs the observe-and-iterate loop over recent runs, proposing and applying changes behind two gates. DEBUG diagnoses one failed or misbehaving run by walking the dependency graph of everything it touched. Use whenever the user wants to create a new exo workflow, iterate on or improve an existing one, or find out why a specific run failed.
|
||||
license: apache-2.0
|
||||
metadata:
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
# exo
|
||||
|
||||
One front door for the exo workflow lifecycle. This skill is a router. It classifies the request into one of three intents, loads the shared substrate, then loads the matching intent recipe and follows it.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Exo is an agent workflow platform. Every intent drives it through an exo MCP server.
|
||||
|
||||
## Pick the connection
|
||||
|
||||
Several exo connections coexist normally, one per workspace, sometimes several on one deployment. Enumerate the exo MCP servers this session has before doing anything else.
|
||||
|
||||
With exactly one, use it. With more than one, ask which through the structured question tool and stop until the user answers, because a wrong guess writes to the wrong workspace. Do not switch connections partway through an intent. Start over if the target changes.
|
||||
|
||||
Call `whoami` on the chosen server. It confirms the connection and returns the workspace ID. Report that ID so the user can see which workspace they are about to change. If no exo MCP tools are present or `whoami` fails, read `resources/bootstrap.md` and follow it, then call `whoami` again. Do not classify an intent until it succeeds.
|
||||
|
||||
`scripts/exo-skill.py` reaches the same workspace over REST, and it has to reach the same one. It reads `EXO_API_URL`, `EXO_API_KEY`, and `EXO_WORKSPACE_ID` from the process environment, falling back to the profile named by `--profile`, which is the file `${XDG_CONFIG_HOME:-~/.config}/exo/<name>.env`. Pass `--profile` with the name of the MCP server you chose on every call, so the two cannot point at different workspaces.
|
||||
|
||||
## Always read first
|
||||
|
||||
Read `resources/common.md`. It holds the shared substrate every intent uses: the run-walking read primitives, the DTO discovery discipline, the resource write primitives, the unprobeable nodes, and the note on why the improve intent reads the debug recipe inline rather than invoking it.
|
||||
|
||||
## Classify the intent
|
||||
|
||||
| If the request is about | Intent | Read |
|
||||
|---|---|---|
|
||||
| Why a specific run failed, what went wrong with a run_id, diagnosing one run | debug | `intents/debug.md` |
|
||||
| Improving, iterating on, tightening, or speeding up an existing workflow over its recent runs | improve | `intents/improve.md` |
|
||||
| Turning an idea into a new workflow, building, creating, or scaffolding a workflow | build | `intents/build.md` |
|
||||
|
||||
Pick exactly one. If the request is genuinely ambiguous between intents, ask the user which one in a single question rather than guessing.
|
||||
|
||||
## User interaction
|
||||
|
||||
Put every question to the user through the harness's structured question tool, whatever it is called here. Free-text prose questions with bullet lists or "Q1/Q2/Q3" prompts are not allowed, even when the question feels open-ended. Bucket open areas into concrete options and let the user type a custom answer instead. Batch related questions into one call so the user answers a structured form rather than a thread of replies. Respect the current limits of the tool you have, and ask directly in prose only when no structured tool is available or the answer is inherently free-form, such as a name, a metric, or a path. This applies to intent disambiguation, the build interrogation in `intents/build.md`, the proposal and rerun gates in `intents/improve.md`, and any candidate-disambiguation prompt in `intents/debug.md`.
|
||||
|
||||
Require an explicit answer at every approval, production write, credential, and rerun gate. Never attach auto-resolution to those questions.
|
||||
|
||||
The boundaries between intents are intentional gates, not friction to remove. A build that ends in a first run does not auto-continue into improve, because the user owns when to cross from constructing to iterating. An improve pass that finds a failed run reads the debug walk inline rather than switching intents, because the diagnosis is a sub-procedure of the loop, not a separate request.
|
||||
|
||||
## Paths
|
||||
|
||||
All paths in the intent files are relative to this skill's root directory: `scripts/` for executables, `resources/` for shared docs and templates, `intents/` for the three recipes. `agents/` holds per-harness interface metadata that no recipe reads.
|
||||
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Exo Workflow Lifecycle"
|
||||
short_description: "Build, improve, and debug exo workflows"
|
||||
default_prompt: "Use $ghost-exo to build, improve, or debug an exo workflow."
|
||||
@@ -0,0 +1,86 @@
|
||||
# Intent: build
|
||||
|
||||
Take a rough idea and turn it into a live, well-formed workflow: orient, interrogate outcome-first, assess and steer, gate on a blueprint, create resources in dependency order, run once manually, and hand off. This intent owns research, plan, create, and one clean manual run. It does not own the iterate loop. When the first run is in hand, route the user to the improve intent, and route any failed run to the debug intent.
|
||||
|
||||
The shared substrate, namely the read and write primitives and the DTO discovery discipline, is in `resources/common.md`. The scoring rubric is in `resources/workflow-assessment.md`. The blueprint shape is in `resources/blueprint.template.md`. Read common.md first.
|
||||
|
||||
## Phase 0: Orient
|
||||
|
||||
Before talking to the user, call `whoami` and list the workspace resources: workflows, skills, environments, models, credentials, and tools. Hold a reuse catalog and a set of candidate template workflows. When a structurally similar workflow exists, you may offer to clone and adapt it as a skeleton, but only from Stage 3 onward, never to seed the outcome.
|
||||
|
||||
## Phase 1: Interrogate (research)
|
||||
|
||||
A fixed coverage checklist with adaptive phrasing. You may not leave this phase until every area is resolved, but skip what the user already answered and phrase each question in context. The order is outcome-first.
|
||||
|
||||
Every question in this phase goes through the harness's structured question tool (see the User interaction section of `SKILL.md`). Bucket each item below into concrete options with a recommended default, batch related items into a single call, and use multi-select where the choices are not mutually exclusive and the tool allows it. Do not ask any interrogation item as free-text prose.
|
||||
|
||||
### Stage 1: Outcome and measurement (the spine)
|
||||
|
||||
1. The one-line idea and the security outcome it advances.
|
||||
2. The concrete metric the workflow will emit to feed that outcome's SLI. This is a hard prerequisite. If the user cannot name one, stay here and help derive a measurable metric from the outcome. Do not advance to decomposition until a metric exists.
|
||||
3. The full ladder, recorded in the blueprint: the emitted metric key, the SLI it feeds, the SLO target, the current baseline, and the good direction. The workflow emits only the raw metric. The rollup lives in a dashboard elsewhere.
|
||||
|
||||
### Stage 2: Trigger and the unit of work
|
||||
|
||||
4. What fires the workflow and on what cadence, with the cadence justified against the outcome's measurement window rather than picked arbitrarily.
|
||||
5. The definition of done for one run and the durable artifact it leaves.
|
||||
|
||||
### Stage 3: Decomposition into linear steps
|
||||
|
||||
6. Narrate the work from trigger to metric emission as a linear sequence. Workflows are linear step chains, not branching graphs.
|
||||
7. Per step, the single responsibility and the measurement it emits.
|
||||
8. The metric chain: show how the per-step metrics roll up to the headline metric, the way a leading-indicator count should equal an outcome count by construction. A chain that does not close, where a step emits a number nothing downstream consumes or reconciles against, is a design smell to surface here, before any wiring.
|
||||
9. The handoff seam at every boundary, down to the exact file path and metric keys, since that seam is the only contract between steps.
|
||||
|
||||
### Stage 4: Per-step realization
|
||||
|
||||
10. The judgment-versus-deterministic split per step, applying the remove-thought lens, which decides how much is scripted skill versus prompt.
|
||||
11. The skill per step, resolved interactively: reuse an existing skill by skill_id, or author a new one with `scripts/exo-skill.py --profile <name> create` (which auto-activates the first version). Discover existing skills first and offer reuse before authoring.
|
||||
12. The model, credentials, env vars, and tools per step, reusing from the Phase 0 catalog by ID wherever a fit exists, and creating new only with the paste-through warning for secrets.
|
||||
|
||||
When a template was chosen in Phase 0, Stages 1 and 2 run identically. The template seeds step structure and wiring only from Stage 3 onward.
|
||||
|
||||
## Phase 2: Assess and steer
|
||||
|
||||
Score each step against `resources/workflow-assessment.md`. Reachable is the gate. Repeatable, Valuable, Verifiable, and Concrete are advisory. This never blocks, but wherever a dimension lands below the threshold, actively reshape the design toward something that would pass: split an overloaded step, move sequencing or parsing into a script, tighten vague inputs to lift Concrete, or place a human-review gate after a step whose Verifiable is weak. Fold the revised scores and reasoning into the blueprint. Weave this into Phase 1 so the design is already close to passing before the user sees the blueprint.
|
||||
|
||||
## Phase 3: Plan gate
|
||||
|
||||
Write `blueprint.md` under the working directory, for example `/tmp/exo-build/<slug>/blueprint.md`, using `resources/blueprint.template.md`, and present it. This is the one hard approval gate before any writes. It shows the outcome ladder, the step graph with per-step prompt, skill, model, environment, creds, vars, and metrics, the metric chain, the assessment scorecard, the reuse-versus-create plan, the handoff seams, and the dependency-ordered build sequence. The user approves once here, and only then do you touch the workspace.
|
||||
|
||||
## Phase 4: Build in dependency order
|
||||
|
||||
Create or wire resources from the leaves up, recording every resulting ID into `manifest.json` in the same working directory immediately after each create, so an interrupted build resumes without double-creating. Consult the manifest before every create. The order is credentials, models, skills, tool bindings, environments, tasks, then workflow. The write path for each is in common.md. Leave the workflow's cron schedule unset.
|
||||
|
||||
## Phase 5: First run
|
||||
|
||||
Trigger one manual run with `trigger_workflow_run`, wait for terminal status, and summarize by walking the child runs and their event summaries, using the read primitives in common.md. The cron schedule stays unset through this phase.
|
||||
|
||||
## Phase 6: Review gate and handoff
|
||||
|
||||
Present the run result and route. A failed or ugly run points the user to the debug intent on that run_id. A working but mediocre one points to the improve intent. When the user says it is good, hand them the exact call that enables the schedule and stop, leaving that final go-live action to them. Build that call with the update procedure in `resources/common.md`, so the body is a full replacement carrying the existing steps and not a lone `cron_schedule`.
|
||||
|
||||
## Report
|
||||
|
||||
- Outcome: built_ran_handed_off, built_no_run, blueprint_only, or stopped.
|
||||
- Blueprint path and the manifest of every created or reused resource ID, by type.
|
||||
- Run trail: the manual run_id with status and duration, if a run happened.
|
||||
- Next step: the pointer to the improve or debug intent, and the exact call to enable the cron when ready.
|
||||
|
||||
## Stop conditions
|
||||
|
||||
- No nameable outcome metric: stay in Stage 1 until one exists.
|
||||
- User declines the blueprint at the plan gate: blueprint_only, with the blueprint saved.
|
||||
- A build write fails partway: the manifest holds what was created. Exit stopped and report the resume point.
|
||||
- The manual run does not reach terminal status within a bounded wait: report the in-flight run_id.
|
||||
|
||||
## Sharp edges
|
||||
|
||||
- The cron is left off until the user enables it. The build never schedules a workflow.
|
||||
- Secret material passes through context only with the warning, and only when no existing credential fits. Prefer reuse by ID.
|
||||
- Route to the debug and improve intents as user-driven hops at the end. Within this intent you do not run their loops.
|
||||
- Workflows are linear step chains, not DAGs. The decomposition must be a sequence.
|
||||
- The manifest is the resume key. Consult it before every create, and never create a resource whose ID already sits in it.
|
||||
- Template cloning rebinds every referenced ID. A cloned workflow must not inherit the source's creds, env, model, or skill_refs.
|
||||
- The assessment steers but never blocks. The metric is the only hard prerequisite.
|
||||
- The metric chain must close. A step emitting a number nothing downstream consumes is a smell to surface, not to wire.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Intent: debug
|
||||
|
||||
Diagnose one failed or misbehaving run end to end by walking the dependency graph of everything it touched until every reachable node is ruled out or named as the cause. The run event tape is one node, not the whole picture. This walk is read-only. It diagnoses and stops. It does not propose or apply fixes, which is the improve intent's job.
|
||||
|
||||
The shared substrate, namely the read primitives, the pivot IDs, the dependency graph, and the unprobeable nodes, is in `resources/common.md`. Read that first if you have not.
|
||||
|
||||
## Inputs
|
||||
|
||||
- A `run_id`, or a workflow name to resolve to one. The run_id wins when both are given.
|
||||
- An optional user hypothesis or symptom, which biases which nodes to probe first but does not let you skip the walk.
|
||||
|
||||
## Nodes and their probes
|
||||
|
||||
Each node can fail and surface as a run-level error. Each has a probe that says whether it is healthy in the run's started_at..finished_at window. Walk all of them.
|
||||
|
||||
| Node | What it owns | Probe |
|
||||
|---|---|---|
|
||||
| Workflow definition | step order, step-to-task binding, concurrency, schedule | `get_resource('workflow', id)` |
|
||||
| Step / child run | per-step environment, task, runner | `list_run_children(run_id)`, then recurse this whole procedure into each child |
|
||||
| Task | instruction text or entrypoint command, skill binding, agent, model, environment binding | `get_resource('task', id)` |
|
||||
| Skill + active version | SKILL.md entrypoint, prompt, scripts, required outputs | `get_resource('skill', id)` then `get_skill_version(skill_id, version_id)` |
|
||||
| Environment | env vars, credential bindings, model defaults | `get_resource('environment', id)` |
|
||||
| Credential | secret material, OAuth expiry, scope | `get_resource('credential', id)`, and `query_observability` for matching credential_uses rows in the window |
|
||||
| Agent | binary version, prompt baseline | No MCP surface. There is no agent resource type, and run.agent_id is a free-form string. Infer health from whether the events show any agent activity at all |
|
||||
| Model / LLM provider | provider, auth, rate limits, 5xx | `get_resource('model', id)` for the row, and error-event payloads in `get_run_events` for runtime failures |
|
||||
| Runner identity | mTLS cert serial/CN, TTL, renewal | No MCP surface. Operator-side only. Capture run.runner_id for the operator |
|
||||
| Runner lifecycle | heartbeats, WS connect/disconnect, OOM, restarts, slot assignment | No MCP surface. Runner host and gateway logs only |
|
||||
| Cert revocation | revoked serials | No MCP surface. cert_revocations is an operator-side collection with no read path |
|
||||
| Credential proxy | DNS allowlist hits, MITM token swap, upstream errors | `query_observability` filtered to the run_id over proxy logs |
|
||||
| Gateway dispatch | run-queue assignment, runner-pool selection across step boundaries | `query_observability` filtered to the run_id and step transitions |
|
||||
| Workflow event tape | step lifecycle, status, errors | `summarize_run_events(run_id)` then `get_run_events(run_id, event_types=[...])` |
|
||||
| Tool surface | per-tool args/results, exit codes | `get_run_events(run_id, event_types=['tool_use','error'])` |
|
||||
| Approvals | pending and resolved approval gates | `list_approval_requests` filtered by run_id |
|
||||
|
||||
Nodes marked No MCP surface cannot be probed from here. Record their pivot IDs in the report and mark them unprobed rather than guessing.
|
||||
|
||||
## Procedure
|
||||
|
||||
0. Resolve to a run_id if only a workflow name was given. `list_resources('workflow')`, exact-then-substring match. Multiple matches means list candidates and stop. Then `list_runs(workflow_id, limit=1, status='failed')`, falling back to the most recent of any status, and narrate the fallback.
|
||||
|
||||
1. `get_run(run_id)`. Capture every pivot ID. If the status is still active, stop and say so.
|
||||
|
||||
2. `summarize_run_events(run_id)`. Read the shape.
|
||||
|
||||
3. `get_run_events(run_id, event_types=[...])`. Include at least error and step_failed, plus whatever the summary flagged. Page with after_sequence until the failure events and their immediate predecessors are in hand. Harvest any new IDs from the failure payloads.
|
||||
|
||||
4. Walk the graph. For every node for which you now have an ID, run its probe over the window. For every step, recurse into its child run with this whole procedure. Do not stop at the first plausible cause, because failures layer, for example a content bug masking a credential rotation or a credential expiry masking a runner restart, so keep going until every reachable node is touched. Record an implicated node and keep walking.
|
||||
|
||||
5. Report:
|
||||
- One-line diagnosis, or a plain statement that the issue is not fully diagnosable here if it converges on an unprobeable node.
|
||||
- Failing nodes, ordered by when they fired in the timeline.
|
||||
- Citations: event sequence numbers, resource fields, observability results.
|
||||
- A suggested fix at the primitive level ONLY when the culprit is a node you can probe and act on. Do not apply it, because that is the improve intent. When the culprit is an unprobeable node, skip the fix.
|
||||
- Coverage list: every node marked clean, culprit, contributing, or unprobed with a reason. An honest unprobed beats a confident wrong answer.
|
||||
|
||||
## Stop conditions
|
||||
|
||||
- Run still active: stop after step 1.
|
||||
- A node has no probe surface: mark it unprobed and continue. Never abandon the rest of the walk for one opaque node.
|
||||
- Multiple contributors: report all of them, ordered by which fired first.
|
||||
|
||||
## Sharp edges
|
||||
|
||||
- Do not page raw events without an event_types filter.
|
||||
- The bound environment at run time may differ from the workflow's current environment if it was rebound since. Cite the run's own IDs, not the workflow's current state.
|
||||
- Child runs are first-class. Each has its own runner, environment, task, agent, and model. Recurse rather than treating them as opaque.
|
||||
- If a node's probe shows it never received work, meaning zero tool calls, zero tokens, or sub-100ms on a step that should take seconds, the failure is upstream. Walk what was supposed to provision it before what it contains.
|
||||
- Different step indices may bind to different environments and runner pools. Compare per-step environment_id when failures cluster on step boundaries.
|
||||
- Probed-and-clean and did-not-probe are different states. Do not conflate them in the report.
|
||||
@@ -0,0 +1,107 @@
|
||||
# Intent: improve
|
||||
|
||||
The observe-and-iterate loop on an existing workflow. Read the last N runs (default 3), walk their step and tool events, find 1 to 2 things worth changing across the whole run set, propose them in plain language, and on explicit agreement apply each through the matching write primitive. On a second explicit agreement, trigger a rerun, fold it into the run set, and reevaluate. Iterate until the user says the workflow is good. Failures in the run set are signal, not an exception, so fold their diagnosis into the proposal set rather than handing the user off.
|
||||
|
||||
The shared substrate, namely the read and write primitives, the DTO discovery discipline, the inline-debug note, and the production and secret safety rules, is in `resources/common.md`. Read it first if you have not.
|
||||
|
||||
## Inputs
|
||||
|
||||
- A workflow name, resolved to a workflow_id via `list_resources('workflow')`. Never ask the user for an ID, because they have a name. Exact-then-substring, and multiple matches means list candidates and stop.
|
||||
- Optional N, the number of recent runs, default 3.
|
||||
- An optional user hypothesis such as "feels slow on step 2" or "the output keeps drifting", which biases which signals to weight without letting you skip the walk.
|
||||
|
||||
## What improvement means
|
||||
|
||||
Four signal categories, three across the run set and one within a single run.
|
||||
|
||||
Correctness, from failed runs: any run whose terminal status is failure, or whose events show errors that resolved only after retries. For each, run the debug walk inline (see step 4). A failure in 1 of 3 runs is also a consistency signal, pointing at flakiness rather than a universal break.
|
||||
|
||||
Efficiency, from cross-run aggregates: total tokens per run, total duration, per-step tokens and duration, tool-call count per step, retry or error counts that still resolved, and idle gaps between steps that change no state.
|
||||
|
||||
Consistency, from cross-run variance: whether each run used the same tools in roughly the same order, whether per-step durations sit in a tight band, whether each run hit the same skill versions, whether output shapes match, and whether any run took a path the others did not.
|
||||
|
||||
Within-run inefficiency: repeated tool calls with identical arguments, long stretches of model output with no tool use and no state change, steps whose token budget is disproportionate to the work, and prompts dragged up by context the task does not need.
|
||||
|
||||
Outliers are where proposals come from. Two runs that look identical and a third that diverges is a stronger signal than three that are uniformly mediocre, because the divergence points at something the workflow does not control.
|
||||
|
||||
## Where the biggest wins come from
|
||||
|
||||
### Remove thought from the LLM
|
||||
|
||||
The single most valuable transformation is moving work out of the model and into deterministic code, because the model is the most expensive and least reliable component. Patterns to look for, roughly by frequency:
|
||||
|
||||
- A step running two or more CLI commands back to back as separate tool calls, where one shell script would do it in one call.
|
||||
- A workflow chaining two or more scripts across steps, where one script taking the right arguments collapses them into one step.
|
||||
- A model turn that exists mainly to parse a blob of command output, where filtering at the previous step leaves only the fields the next turn needs.
|
||||
- A model turn that tracks variables, paths, or state across the run, where externalizing that into a file or an env var removes the bookkeeping.
|
||||
- A model turn that picks between options a script could pick with a conditional.
|
||||
- Long repeated context blocks in prompts that exist only because the previous step did not extract the needed part.
|
||||
|
||||
Name which lever a proposal pulls. When a proposal pulls none but still matters, such as a missing handoff, a failed-run fix, or a consistency tightening, say so explicitly.
|
||||
|
||||
### Good, Better, Best
|
||||
|
||||
Every proposal names the rung the workflow is on and the rung it moves toward.
|
||||
|
||||
- Good: works roughly 80% of the time, spends turns and tokens on sequencing, parsing, and state-tracking the model should not be doing, and leans on a strong model to paper over its own ambiguity.
|
||||
- Better: works roughly 90% of the time, hands the deterministic parts to purpose-built scripts, and runs on a modest model because the structure does the work.
|
||||
- Best: works roughly 98% of the time, is token-efficient, composes scripts cleanly, and reserves the model for judgment, open-ended synthesis, and natural-language interaction.
|
||||
|
||||
The rubric is directional. A workflow can be Good on step 1 and Best on step 2, so frame proposals at the step level when the evidence supports it.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Resolve the workflow name to an ID. Multiple matches means list candidates and stop.
|
||||
|
||||
2. Gather the run set via `list_runs(workflow_id, limit=N)`. Fewer than N means work with what exists and note the smaller sample. Fewer than 2 means consistency signals are unavailable, so fall back to within-run analysis and say the proposal is weaker.
|
||||
|
||||
3. Walk each run with `get_run` and `summarize_run_events` for each, plus `list_run_children`, and the same pair on each child. The goal is a per-run, per-step shape naming tokens, duration, tool-call count, errors-resolved-to-success, and the distinct tools used.
|
||||
|
||||
4. Diagnose failed runs inline. For each failed run in the set, read the debug walk in `intents/debug.md` and run it against that run_id, then fold the resulting diagnosis into the proposal set. Do not invoke debug as a skill. Read the file and execute it inline, per the inline-debug note in common.md. If you choose to skip a diagnosis, that run stays an opaque failure: it still informs the consistency signal but cannot motivate a failure-fix proposal.
|
||||
|
||||
5. Drill where surviving runs look interesting via `get_run_events` on the outliers the step 3 summary flagged. Filter to tool_use and text for efficiency, to step_started, step_finished, and error for composition or consistency. Page with after_sequence.
|
||||
|
||||
6. Synthesize 1 to 2 proposals from any category. Each names a specific target (skill, task, workflow step, env binding, model on a task, credential), a specific change, the evidence from the run set, the current and target rung on Good/Better/Best, and which thought-removal lever it pulls or why it still matters if none. When proposals tie on evidence, prefer the one that pulls a lever. If nothing meaningful surfaces, say so and stop, because a made-up low-value change is worse than none.
|
||||
|
||||
7. First gate, agreement on the change. One message with the proposals, their evidence, the target type and ID, and a before-and-after for the field. Ask for explicit go-ahead on which to apply. Treat ambiguity as no, and a partial yes as exactly that. For a credential write, fold the LLM-context exposure warning into this same message and request the value in the reply.
|
||||
|
||||
8. Apply the agreed change through the write primitive from common.md. If both were agreed, apply in the order the user listed and capture each result. Capture every resource ID and version ID for the change log. One write per proposal, even when bundled.
|
||||
|
||||
9. Second gate, agreement on the rerun. A short message naming what was applied, asking whether to trigger a rerun now. Wait for explicit yes. No, or wants-to-inspect-first, exits applied_no_rerun.
|
||||
|
||||
10. Trigger the rerun via `trigger_workflow_run(workflow_id)`. Capture the new run_id. Wait for terminal status with a bounded poll.
|
||||
|
||||
11. Fold the new run into the set and return to step 3, dropping the oldest if the set is larger than N. The newest run matters most for judging whether the change helped.
|
||||
|
||||
12. Stop when the user says so.
|
||||
|
||||
## Report
|
||||
|
||||
At the end of every loop, including a mid-iteration stop:
|
||||
|
||||
- Outcome: user_satisfied, user_declined, applied_no_rerun, no_proposal, or cap_reached.
|
||||
- Change log: one row per applied write, in order, naming the iteration, target, primitive, resource ID, and for skill content the new version_id plus bundle size.
|
||||
- Run trail: every run_id analyzed and every run_id triggered, in order, with status and duration.
|
||||
- Last evaluation: the most recent run set's efficiency and consistency signals, which is what the user judges when they say good enough.
|
||||
|
||||
## Stop conditions
|
||||
|
||||
- Declines all proposals at the first gate with no qualifier: user_declined.
|
||||
- Accepts changes but declines the rerun: applied_no_rerun.
|
||||
- Says the workflow is good: user_satisfied.
|
||||
- Nothing worth changing: no_proposal, with the run-set summary as the rationale.
|
||||
- Rerun not terminal within the bounded poll: cap_reached, reporting the in-flight run_id.
|
||||
- Workflow name resolves to multiple candidates: stop after step 1 and ask which.
|
||||
|
||||
## Sharp edges
|
||||
|
||||
- Two gates per iteration, never one. The change gate is the user owning whether the change is right, and the rerun gate is the user owning when their workflow runs against real resources. Collapsing them surprises the user.
|
||||
- One write per proposal, even when bundled, so the change log attributes the next run's behavior to the right change.
|
||||
- Skill edits go through `scripts/exo-skill.py --profile <name>` so file contents never serialize as tool-call arguments and land in the workspace you are already on. The MCP `create_skill_version({skill_id, files, base_version_id?})` path is the fallback and carries the full-bundle token cost. Skill versions are not reachable through the generic CRUD tools. Either way no skill is deleted and no environment is rebound.
|
||||
- Failures are signal. Diagnose them inline via intents/debug.md and fold the report into the proposal set rather than aborting or handing the user off.
|
||||
- Do not propose without evidence from the run set. "This prompt could be tighter" is not a proposal. "Step 2 averaged 4200 tokens, roughly 3000 of them the unchanged context block from step 1, and dropping that block cuts step 2 input by about 70%" is a proposal.
|
||||
- A proposal that moves nothing up a rung is polish, not improvement. Label it polish at the first gate if the user asked for one anyway.
|
||||
- Prefer thought-removal proposals, because moving work out of the model cuts tokens, latency, and variance and usually opens a cheaper model, which other improvements rarely all do.
|
||||
- Outlier-driven beats average-driven. Two consistent runs and one divergent is sharper than three uniformly varying.
|
||||
- The bound environment at run time may differ from the workflow's current binding if rebound since. Write to the one the analyzed runs used, captured in step 3.
|
||||
- "The rerun looks better" is not the user being satisfied. Wait for explicit confirmation before user_satisfied.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Workflow blueprint: {{WORKFLOW_NAME}}
|
||||
|
||||
## Outcome and measurement
|
||||
|
||||
- Security outcome: {{OUTCOME}}
|
||||
- Headline metric emitted: `{{METRIC_KEY}}`
|
||||
- SLI it feeds: {{SLI}}
|
||||
- SLO target: {{SLO_TARGET}}
|
||||
- Current baseline: {{BASELINE}}
|
||||
- Good direction: {{higher or lower}} is better
|
||||
|
||||
## Trigger and unit of work
|
||||
|
||||
- Trigger: {{cron | webhook | manual}}, cadence {{CADENCE}}
|
||||
- Cadence justified against the measurement window: {{WHY}}
|
||||
- Definition of done for one run: {{DONE}}
|
||||
- Durable artifact a run leaves: {{ARTIFACT}}
|
||||
|
||||
## Steps
|
||||
|
||||
For each step, in order:
|
||||
|
||||
### Step {{N}}: {{STEP_NAME}}
|
||||
|
||||
- Single responsibility: {{WHAT}}
|
||||
- Emits metric: `{{STEP_METRIC}}`
|
||||
- Inputs: {{INPUTS}}
|
||||
- Outputs and handoff seam to next step: file `{{PATH}}`, metric keys `{{KEYS}}`
|
||||
- Skill: {{reuse skill_id ... | author new ...}}
|
||||
- Model: {{MODEL}}
|
||||
- Credentials: {{CREDS}}
|
||||
- Env vars: {{VARS}}
|
||||
- Tools: {{TOOLS}}
|
||||
- Judgment versus deterministic split: {{what the LLM does versus what a script does}}
|
||||
|
||||
## Metric chain
|
||||
|
||||
How the per-step metrics roll up to the headline SLI metric, and the by-construction check between them:
|
||||
|
||||
{{CHAIN}}
|
||||
|
||||
## Assessment scorecard
|
||||
|
||||
| Step | Reachable | Repeatable | Valuable | Verifiable | Concrete | Notes and steering applied |
|
||||
|---|:--:|:--:|:--:|:--:|:--:|---|
|
||||
| {{N}} | | | | | | |
|
||||
|
||||
Reachable is the gate. The others are advisory. Record any reshaping done to lift a weak dimension, and any human-review gate placed where Verifiable is weak.
|
||||
|
||||
## Resource plan (reuse versus create)
|
||||
|
||||
| Resource | Type | Reuse id or CREATE | Notes |
|
||||
|---|---|---|---|
|
||||
| | | | |
|
||||
|
||||
## Build sequence (dependency order)
|
||||
|
||||
1. Credentials
|
||||
2. Models
|
||||
3. Skills
|
||||
4. Tool bindings
|
||||
5. Environments
|
||||
6. Tasks
|
||||
7. Workflow, with the cron schedule left unset
|
||||
@@ -0,0 +1,59 @@
|
||||
# Bootstrap: connecting the exo MCP server
|
||||
|
||||
Run this when the exo MCP tools are absent, or when `whoami` fails to reach the workspace. Skip it whenever `whoami` already returns a workspace ID.
|
||||
|
||||
## What the connection needs
|
||||
|
||||
Register a streamable HTTP MCP server in whatever way this harness registers one. It needs three things:
|
||||
|
||||
These key names are the same in a profile file and in the environment:
|
||||
|
||||
| Field | Key | Notes |
|
||||
|---|---|---|
|
||||
| Endpoint | `EXO_API_URL` | The workspace base URL. Append `/api/v1/mcp`. |
|
||||
| `Authorization` header | `EXO_API_KEY` | Sent as `Bearer <key>`. |
|
||||
| `X-Workspace-Id` header | `EXO_WORKSPACE_ID` | The `ws_` identifier the key belongs to. |
|
||||
|
||||
Those values arrive by whatever route set this instance up, so look before concluding they are absent. A setup script may have written a connection description file holding a complete `mcpServers` entry to copy verbatim, plus a matching credential profile under `${XDG_CONFIG_HOME:-~/.config}/exo/`. Those are commonly named for the connection, as in `exo-demo.mcp.json` and `exo-demo.env`. Failing that, the user may have exported the variables. Search the working directory and that config directory first, then fall back to the environment.
|
||||
|
||||
Report what you could not find, by name, and stop. Never ask the user to paste the key into the conversation, because it would then sit in the transcript. Point them at whichever source is missing instead.
|
||||
|
||||
## Name the entry, and leave the others alone
|
||||
|
||||
Several exo connections coexist normally. One workspace per entry, and a single deployment often serves more than one. Adding another is the usual case rather than a problem.
|
||||
|
||||
When a setup script wrote a connection description file, register under the exact name its `mcpServers` key uses, and never rename it. A credential profile of the same name sits beside it, and renaming one breaks the pair.
|
||||
|
||||
Otherwise name the entry after the deployment and the workspace it points at, following whatever convention the existing entries use. Never reuse a bare name like `exo`, because the next connection collides with it.
|
||||
|
||||
Never modify or remove an entry that points at a different endpoint or a different workspace. Rewrite an entry only when it already points at the same workspace you are configuring now.
|
||||
|
||||
Write the credentials as literal values in this entry. A variable reference cannot work here, because one process holds one value per variable name and several connections need several keys at once. The CLI profile described in `SKILL.md` is where variable indirection belongs.
|
||||
|
||||
## Find the mechanism before concluding there is none
|
||||
|
||||
Every harness that runs MCP tools can register an MCP server. Do the discovery rather than assuming:
|
||||
|
||||
1. Look at how this harness already stores MCP servers, and copy that shape. An existing entry is the most reliable template available, and one is usually there. Copy the structure only. Take no URL, no header value, and no credential reference from another entry, because pointing exo at another service's token sends that token to the exo endpoint.
|
||||
2. Check the harness CLI for an MCP subcommand and read its help for the flags that set a URL, headers, and a bearer token.
|
||||
3. Failing both, edit the config file directly in the format the existing entries use. Merge the new entry in. Never rewrite the file from scratch, and never remove or reorder anything you did not add.
|
||||
|
||||
Report that registration is impossible only after all three come up empty. Say which you tried.
|
||||
|
||||
Write to the narrowest scope the harness offers. Reach for a global or user-wide config only when there is no project-level or session-level alternative.
|
||||
|
||||
## The rules that constrain how
|
||||
|
||||
Write the key and the workspace ID as literal header values, and use the harness's field for literal headers rather than the one that names environment variables. Those two fields are separate, and the literal field expands nothing. Putting `${VAR}` in it sends that text to the server as the header, which reads as an invalid credential rather than as a mistake in the config.
|
||||
|
||||
Resolve the endpoint to a literal URL at write time. Do not template it. Harnesses differ here and at least one drops a server whose URL is not a valid absolute URL, silently and with no error, which produces a config that looks written and yields no tools.
|
||||
|
||||
## Gate the write
|
||||
|
||||
Editing a config file changes the user's machine outside this workspace, so it gets the same gate as a production write. Show the exact entry and the exact file it lands in, then require an explicit answer before writing. Proceed on approval, and stop on refusal without writing a partial entry.
|
||||
|
||||
## Verify
|
||||
|
||||
Call `whoami` and confirm it returns a workspace ID. Report that ID and continue to intent classification. An authentication failure here usually means a header carried template text instead of a resolved value, or the key belongs to a different workspace than the `X-Workspace-Id` header names. Recheck both before anything else.
|
||||
|
||||
A harness usually loads MCP servers once at session start, so a server registered mid-session may not appear until the user restarts. If `whoami` is still unavailable right after a successful write, say the config is in place and ask the user to restart the session. Do not rewrite the entry or try a different endpoint.
|
||||
@@ -0,0 +1,69 @@
|
||||
# common.md: the exo MCP substrate
|
||||
|
||||
Shared by all three intents: the read primitives for walking runs, the DTO discovery discipline, the write primitives for changing resources, the unprobeable nodes, the in-bundle invocation note, and the production and secret safety rules. Read this before any intent recipe.
|
||||
|
||||
## Orientation
|
||||
|
||||
Every intent talks to one exo workspace through that workspace's MCP server. Several connections may be present, so confirm which one you are on before acting, per the connection rules in `SKILL.md`. If you do not know the workspace ID, call `whoami` first. It is the only tool callable without a workspace context. If the exo MCP tools are absent or `whoami` fails, read `resources/bootstrap.md` and follow it before going further. The `workspace_id` it returns travels as the `X-Workspace-Id` header, which the server configuration supplies. Later tool calls do not accept it as an argument.
|
||||
|
||||
## Read primitives (walking runs)
|
||||
|
||||
- `list_resources(type, parent_id?)` returns every resource of a type in the workspace. The response key is the plural of the type.
|
||||
- `get_resource(type, id, parent_id?)` returns one resource. For a skill, this is the cheapest way to find its `active_version_id`.
|
||||
- Nested resources such as `task_metric` require `parent_id` on both calls.
|
||||
- `list_runs(workflow_id, ...)` returns a workflow's runs, most recent first, with optional status and since filters.
|
||||
- `get_run(run_id)` returns one run's metadata: status, timing, token usage, environment, effective model, and whether it is still active or terminal.
|
||||
- `summarize_run_events(run_id)` returns a compact aggregate of one run: counts per event type, distinct tool calls with counts, distinct errors with one sample each, and token totals. Call this BEFORE get_run_events to decide whether to drill in at all and which event types to fetch.
|
||||
- `get_run_events(run_id, event_types=[...])` returns the raw event stream, filtered. Never page this without an event_types filter. Page with after_sequence.
|
||||
- `list_run_children(run_id)` returns the child runs of a multi-step run, one per executed step. Each child is a first-class run with its own runner, environment, task, agent, and model. Recurse into children rather than treating them as opaque payloads on a step event.
|
||||
- `get_skill_version(skill_id, version_id, with_content)` returns a skill bundle's files.
|
||||
- `list_approval_requests(run_id)` returns the pending and resolved approval gates for a run.
|
||||
- `query_observability(category='events', run_id, ...)` gives a cross-run log view. Bucketed `traffic` observability cannot be filtered by run ID.
|
||||
|
||||
## The dependency graph
|
||||
|
||||
Every run depends on a chain of resources, and a run-level failure can originate in any of them:
|
||||
|
||||
workflow definition, then step or child run, then task, then skill and its active version, then environment, then credential, then model, with the runner, agent, credential proxy, and gateway dispatch alongside.
|
||||
|
||||
Pivot IDs to harvest from `get_run` and from failure-event payloads: `workflow_id`, `task_id`, `source_id`, `execution_id`, `parent_run_id`, every child via `list_run_children`, `runner_id`, `agent_id`, `environment_id`, the effective model, and `started_at` and `finished_at` for the observability window.
|
||||
|
||||
Proxy requests attributed to a run surface in `get_run_events` as `traffic_accepted` or `traffic_blocked`, and the payloads can carry `credential_id`, host, path, status, and error code. There is no MCP query for raw credential-use rows. Read credential metadata with `get_resource` and mark secret validity unprobed unless the events prove the result.
|
||||
|
||||
## Write primitives (the fix and build surface)
|
||||
|
||||
Each resource layer maps to one write path. A build creates these from the leaves of the graph upward. An improvement changes exactly one of them per pass.
|
||||
|
||||
Discover the DTO before you write. `describe_resources` is authoritative for the resource-type slugs the generic CRUD tools accept, so call it when a slug below does not resolve rather than guessing. `describe_resource(type)` returns the accepted create and update fields for one type. Call it before every unfamiliar create and before every update.
|
||||
|
||||
To create: call `describe_resource(type)`, build the body from `create_body` fields only, include every required field, then call `create_resource(type, body, parent_id?)`.
|
||||
|
||||
To update: call `describe_resource(type)`, read the current resource with `get_resource(type, id, parent_id?)`, build a full replacement body from `update_body` fields only that carries every required existing value, change the one field you mean to change, show that semantic diff at the approval gate, then call `update_resource(type, id, body, parent_id?)`.
|
||||
|
||||
`update_resource` is a full-replacement PUT, not a partial patch. A one-field body wipes every field it omits. Do not send one unless the discovered DTO permits it, and do not round-trip computed response fields back into the body. Nested resources such as `task_metric` take a `parent_id` and are not reachable through the parent's DTO.
|
||||
|
||||
| Target | Write path |
|
||||
|---|---|
|
||||
| Credential | Reuse by ID where one fits. Otherwise create through the discovered DTO. New secret values pass through the LLM context, so warn in the same message that requests the paste. Prefer having the user set the value in the UI. |
|
||||
| Model | Reuse the workspace default unless a new provider is genuinely needed. Otherwise create through the discovered DTO. |
|
||||
| Skill content | Use the bundled CLI, passing `--profile <mcp-server-name>` on every call so it targets the workspace you are already on. `python3 scripts/exo-skill.py --profile <name> download <skill> --out <dir>`, edit files, then `python3 scripts/exo-skill.py --profile <name> upload <skill> --folder <dir> --activate`. The CLI moves bundles over REST so file contents never serialize as tool-call arguments. A brand-new skill uses `python3 scripts/exo-skill.py --profile <name> create --folder <dir>`. The MCP fallback is `create_skill_version({skill_id, files: [{path, content}], base_version_id?})`, which auto-activates the new version. There is no separate `activate_skill_version` call, because that tool is rollback-only. Passing `base_version_id` sends only the changed files as an overlay, and omitting it requires the complete bundle. Either way the file contents serialize as tool-call arguments, which is the token cost the CLI exists to avoid. Either path keeps the environment's skill_ref pointing at the same skill_id with no rebind. |
|
||||
| Tool binding | Update the owning `environment` through the discovered DTO. Tool bindings are a field on the environment, not a resource of their own. |
|
||||
| Environment | Create or update through the discovered DTO. References creds, model, skill_refs, tool_bindings, and env vars. Surface only the keys being added or changed, and never print values for keys whose names suggest secret material. |
|
||||
| Task | Create or update through the discovered DTO. The execution mode lives here, and metric definitions live on the nested `task_metric` resource. Set exactly one of `instruction`, which pairs an agent with a model, or `entrypoint`, where the runner execs the command via `sh -c` from the workspace dir with no LLM. Under `entrypoint`, exit 0 completes the run, stdout is the run output, and `model_id` must be unset. Entrypoint paths differ by source: skill scripts are workspace-relative, as in `python ./.agents/skills/<skill>/resources/script.py`, and environment files are read-only under `$EXO_RESOURCES_DIR` and need an interpreter prefix. |
|
||||
| Workflow | Create or update through the discovered DTO. Composes steps in order. When changing composition, surface a structural diff at the gate, not just the new array. |
|
||||
|
||||
Skill iteration is strictly additive in v1: create a new skill version and activate it, never delete a skill or a version, and never unbind and rebind an environment.
|
||||
|
||||
## Unprobeable nodes
|
||||
|
||||
Some nodes have no MCP probe or write path: runner identity and lifecycle, cert revocations, raw credential-use rows, and agent internals. When a diagnosis converges on one of these, name it, record the pivot IDs for the operator, and mark it unprobed rather than guessing. There is no workspace write surface for them.
|
||||
|
||||
## Why improve reads debug inline
|
||||
|
||||
Invoking another skill from inside a skill does not reliably return control to the caller. That is the reason the three intents live in one bundle rather than three skills. When the improve recipe needs a failed-run diagnosis, it reads `intents/debug.md` and runs that walk inline, which is a file read and not a skill invocation, so control never leaves the recipe. Do not invoke any skill from within an intent. Read the file and execute it.
|
||||
|
||||
## Production and secret safety
|
||||
|
||||
- Require an explicit answer before any production write and before any production run trigger. A blueprint approval counts only when it enumerates both the writes and the one manual run.
|
||||
- Keep credential values out of logs, diffs, manifests, and reports.
|
||||
- Do not switch workspaces after approval. Restart orientation if the target changes.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Assessing a Workflow for Agent Fit
|
||||
|
||||
A one-page guide for judging whether a piece of work is a good candidate to hand to an AI agent, meaning an LLM with tools.
|
||||
|
||||
## The core idea
|
||||
|
||||
Two questions decide whether work belongs with an agent. The first is whether the work is even within an agent's reach, since an agent perceives and acts through text and tools rather than the physical world. The second, which only matters once the first is satisfied, is whether the work is worth handing off and shaped so the agent can succeed. We capture the first as a gate and the rest as four scored dimensions, and the best-fit work is where all of them hold at once.
|
||||
|
||||
## The five criteria
|
||||
|
||||
| Criterion | What it measures |
|
||||
|---|---|
|
||||
| **Reachable** | How much of what the work needs to read and to change is available through the agent's tools rather than the physical world or undocumented knowledge. This is the gate. If it fails, nothing else matters. |
|
||||
| **Repeatable** | How closely the work follows a pattern the model has seen many times. Patterned, conventional work is in-distribution and is what models are reliably good at. |
|
||||
| **Valuable** | How worth doing it is to offload the work, counting both how often it recurs and how much expensive human time it currently burns. |
|
||||
| **Verifiable** | How cheaply and objectively the result can be checked once produced, whether by a test, a tool, a quick diff, or a human reviewer downstream. This is the linchpin, because a probabilistic agent needs a check to iterate against and to be trusted. |
|
||||
| **Concrete** | How clearly the inputs and the definition of done are specified, so the agent can start cleanly and knows when it is finished. |
|
||||
|
||||
## The scoring scale
|
||||
|
||||
Each criterion is scored on a single unipolar scale that runs in one direction from none to total.
|
||||
|
||||
| Anchor | Score |
|
||||
|---|:---:|
|
||||
| Not at all | 0 |
|
||||
| Slightly | 1 |
|
||||
| Moderately | 2 |
|
||||
| Very | 3 |
|
||||
| Completely | 4 |
|
||||
|
||||
## Scoring a workflow step
|
||||
|
||||
Break the workflow into its steps and give each step its own five-by-five grid, marking where each criterion lands. Read it left to right. The further right the marks, the better the fit, and any mark in the 0 or 1 column is a red flag that points to the part of the work a human still needs to own.
|
||||
|
||||
| Attribute | 0 None | 1 Slight | 2 Mod | 3 Very | 4 Full |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|
|
||||
| Reachable | | | | ● | |
|
||||
| Repeatable | | | | ● | |
|
||||
| Valuable | | | | | ● |
|
||||
| Verifiable | | | | | ● |
|
||||
| Concrete | | | | | ● |
|
||||
|
||||
## Reading the result
|
||||
|
||||
The hand-off threshold is Very, meaning a score of 3 on every criterion. A step whose marks all sit at or beyond the 3 column is one you can delegate with confidence. A step with a mark at Moderately or below has a weak spot worth addressing before you trust it.
|
||||
|
||||
A low verifiability score does not disqualify a step on its own, because verification can be supplied externally. A downstream human review of the step's output is itself a cheap, objective check, so placing a single human-review gate at the end of a chain often rescues the judgment-heavy steps and turns the whole workflow into a safe agent-drafts and human-approves pattern.
|
||||
|
||||
```
|
||||
step 1 → step 2 → step 3 → step 4 ──▶ [ HUMAN REVIEW ] ──▶ approve / send back
|
||||
(agent) (agent) (agent) (agent) verification gate
|
||||
```
|
||||
+456
@@ -0,0 +1,456 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
exo-skill: download and upload exo skill bundles by folder.
|
||||
|
||||
Sidesteps the token cost of serializing skill content as MCP tool-call
|
||||
arguments. The improvement loop downloads the active version into a local
|
||||
folder, edits files with normal text-editing tools, and uploads the folder
|
||||
as a new SkillVersion (optionally activating it).
|
||||
|
||||
Settings, read from the process environment first and then from the
|
||||
profile named by --profile (or EXO_PROFILE), which is the file
|
||||
$XDG_CONFIG_HOME/exo/<name>.env, defaulting to ~/.config. Name a
|
||||
profile after the MCP server holding
|
||||
the same workspace, so the two cannot drift apart:
|
||||
|
||||
EXO_API_URL required, the workspace endpoint without a path
|
||||
EXO_API_KEY required, the same bearer key used by the MCP server
|
||||
EXO_WORKSPACE_ID sent as X-Workspace-Id. Remote gateways require it, and
|
||||
a local dev gateway may infer it from the key instead.
|
||||
|
||||
Subcommands:
|
||||
create --folder DIR [--name NAME] [--force]
|
||||
download SKILL [--version VID] [--out DIR]
|
||||
upload SKILL --folder DIR [--activate] [--no-base]
|
||||
|
||||
create bootstraps a brand-new skill from a folder via the multipart
|
||||
import route (POST /skills/upload). The skill name defaults to the
|
||||
folder's basename; pass --name to override. The first version is
|
||||
created and auto-activated, and a meta file is written into the folder
|
||||
so a later `upload <id> --folder DIR` chains as a patch. Unlike upload,
|
||||
create can carry binary files because the import route is multipart.
|
||||
The import route upserts by name, so create refuses an existing name
|
||||
unless --force is passed.
|
||||
|
||||
SKILL may be a skill ID or a skill name. When a name is supplied and
|
||||
multiple skills match by case-insensitive exact-or-substring rules, the
|
||||
candidates are printed and the command exits non-zero so the caller can
|
||||
disambiguate.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import binascii
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
|
||||
META_FILE = ".exo-skill-meta.json"
|
||||
API_PREFIX = "/api/v1"
|
||||
|
||||
|
||||
CONFIG_DIR = pathlib.Path(
|
||||
os.environ.get("XDG_CONFIG_HOME") or pathlib.Path.home() / ".config"
|
||||
) / "exo"
|
||||
|
||||
_profile: dict[str, str] = {}
|
||||
|
||||
|
||||
def load_profile(name: str | None) -> None:
|
||||
"""Populate _profile from CONFIG_DIR/<name>.env. Process env still wins."""
|
||||
if not name:
|
||||
return
|
||||
path = CONFIG_DIR / f"{name}.env"
|
||||
if not path.is_file():
|
||||
available = sorted(p.stem for p in CONFIG_DIR.glob("*.env"))
|
||||
sys.exit(
|
||||
f"error: no profile {name!r} at {path}"
|
||||
+ (f"\navailable: {', '.join(available)}" if available else "")
|
||||
)
|
||||
for line in path.read_text().splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, val = line.partition("=")
|
||||
_profile[key.strip()] = val.strip().strip("\"'")
|
||||
|
||||
|
||||
def setting(name: str) -> str | None:
|
||||
return os.environ.get(name) or _profile.get(name)
|
||||
|
||||
|
||||
def setting_or_die(name: str) -> str:
|
||||
val = setting(name)
|
||||
if not val:
|
||||
sys.exit(f"error: {name} is required. Set it, or pass --profile.")
|
||||
return val
|
||||
|
||||
|
||||
def env_or_die(name: str) -> str:
|
||||
return setting_or_die(name)
|
||||
|
||||
|
||||
def api_url() -> str:
|
||||
return setting_or_die("EXO_API_URL").rstrip("/")
|
||||
|
||||
|
||||
def request(method: str, path: str, body: dict | None = None) -> dict:
|
||||
url = api_url() + API_PREFIX + path
|
||||
data = None
|
||||
headers = {"Authorization": "Bearer " + env_or_die("EXO_API_KEY")}
|
||||
workspace_id = setting("EXO_WORKSPACE_ID")
|
||||
if workspace_id:
|
||||
headers["X-Workspace-Id"] = workspace_id
|
||||
if body is not None:
|
||||
data = json.dumps(body).encode("utf-8")
|
||||
headers["Content-Type"] = "application/json"
|
||||
req = urllib.request.Request(url, data=data, method=method, headers=headers)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
raw = resp.read()
|
||||
if not raw:
|
||||
return {}
|
||||
return json.loads(raw)
|
||||
except urllib.error.HTTPError as e:
|
||||
detail = e.read().decode("utf-8", errors="replace")
|
||||
sys.exit(f"error: {method} {path} -> HTTP {e.code}: {detail}")
|
||||
except urllib.error.URLError as e:
|
||||
sys.exit(f"error: {method} {path} -> {e.reason}")
|
||||
|
||||
|
||||
def post_multipart(path: str, fields: list[tuple], file_parts: list[tuple]) -> dict:
|
||||
"""POST a multipart/form-data body.
|
||||
|
||||
fields is a list of (name, str_value) pairs; file_parts is a list of
|
||||
(field_name, filename, content_bytes, content_type) tuples. Mirrors
|
||||
request()'s auth headers and error handling.
|
||||
"""
|
||||
boundary = "----exo-skill-" + binascii.hexlify(os.urandom(16)).decode()
|
||||
crlf = b"\r\n"
|
||||
buf = bytearray()
|
||||
|
||||
def emit(text: str) -> None:
|
||||
buf.extend(text.encode("utf-8"))
|
||||
|
||||
for name, value in fields:
|
||||
emit(f"--{boundary}\r\n")
|
||||
emit(f'Content-Disposition: form-data; name="{name}"\r\n\r\n')
|
||||
buf.extend(value.encode("utf-8"))
|
||||
buf.extend(crlf)
|
||||
for field_name, filename, content, content_type in file_parts:
|
||||
emit(f"--{boundary}\r\n")
|
||||
emit(f'Content-Disposition: form-data; name="{field_name}"; filename="{filename}"\r\n')
|
||||
emit(f"Content-Type: {content_type}\r\n\r\n")
|
||||
buf.extend(content)
|
||||
buf.extend(crlf)
|
||||
emit(f"--{boundary}--\r\n")
|
||||
|
||||
url = api_url() + API_PREFIX + path
|
||||
headers = {
|
||||
"Authorization": "Bearer " + env_or_die("EXO_API_KEY"),
|
||||
"Content-Type": f"multipart/form-data; boundary={boundary}",
|
||||
}
|
||||
workspace_id = setting("EXO_WORKSPACE_ID")
|
||||
if workspace_id:
|
||||
headers["X-Workspace-Id"] = workspace_id
|
||||
req = urllib.request.Request(url, data=bytes(buf), method="POST", headers=headers)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
raw = resp.read()
|
||||
if not raw:
|
||||
return {}
|
||||
return json.loads(raw)
|
||||
except urllib.error.HTTPError as e:
|
||||
detail = e.read().decode("utf-8", errors="replace")
|
||||
sys.exit(f"error: POST {path} -> HTTP {e.code}: {detail}")
|
||||
except urllib.error.URLError as e:
|
||||
sys.exit(f"error: POST {path} -> {e.reason}")
|
||||
|
||||
|
||||
def list_skills() -> list[dict]:
|
||||
return request("GET", "/skills").get("skills", [])
|
||||
|
||||
|
||||
def resolve_skill(skill_ref: str) -> dict:
|
||||
"""Resolve a skill name or ID to its summary dict."""
|
||||
skills = list_skills()
|
||||
by_id = next((s for s in skills if s["id"] == skill_ref), None)
|
||||
if by_id:
|
||||
return by_id
|
||||
|
||||
ref_lower = skill_ref.lower()
|
||||
exact = [s for s in skills if s["name"].lower() == ref_lower]
|
||||
if len(exact) == 1:
|
||||
return exact[0]
|
||||
if len(exact) > 1:
|
||||
_print_candidates(exact, skill_ref)
|
||||
|
||||
substr = [s for s in skills if ref_lower in s["name"].lower()]
|
||||
if len(substr) == 1:
|
||||
return substr[0]
|
||||
if len(substr) > 1:
|
||||
_print_candidates(substr, skill_ref)
|
||||
|
||||
sys.exit(f"error: no skill matched '{skill_ref}'")
|
||||
|
||||
|
||||
def _print_candidates(candidates: list[dict], skill_ref: str) -> None:
|
||||
print(f"error: multiple skills matched '{skill_ref}':", file=sys.stderr)
|
||||
for s in candidates:
|
||||
print(f" {s['id']} {s['name']}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def get_version(skill_id: str, version_id: str) -> dict:
|
||||
versions = request("GET", f"/skills/{skill_id}/versions").get("versions", [])
|
||||
match = next((v for v in versions if v["id"] == version_id), None)
|
||||
if not match:
|
||||
sys.exit(f"error: version {version_id} not found for skill {skill_id}")
|
||||
return match
|
||||
|
||||
|
||||
def _collect_folder_files(folder: pathlib.Path) -> list[tuple]:
|
||||
"""Walk a folder into (rel_path, content_bytes, content_type) tuples.
|
||||
|
||||
Skips the meta file and any dotted path component. UTF-8-decodable
|
||||
files are sent as text/plain so the runtime classifies them as
|
||||
text-editable regardless of extension; everything else is binary.
|
||||
"""
|
||||
parts: list[tuple] = []
|
||||
for entry in sorted(folder.rglob("*")):
|
||||
if entry.is_dir():
|
||||
continue
|
||||
if entry.name == META_FILE:
|
||||
continue
|
||||
rel_parts = entry.relative_to(folder).parts
|
||||
if any(part.startswith(".") for part in rel_parts):
|
||||
continue
|
||||
rel = entry.relative_to(folder).as_posix()
|
||||
raw = entry.read_bytes()
|
||||
try:
|
||||
raw.decode("utf-8")
|
||||
content_type = "text/plain; charset=utf-8"
|
||||
except UnicodeDecodeError:
|
||||
content_type = "application/octet-stream"
|
||||
parts.append((rel, raw, content_type))
|
||||
return parts
|
||||
|
||||
|
||||
def cmd_create(args: argparse.Namespace) -> None:
|
||||
folder = pathlib.Path(args.folder)
|
||||
if not folder.is_dir():
|
||||
sys.exit(f"error: {folder} is not a directory")
|
||||
|
||||
name = (args.name or folder.resolve().name).strip()
|
||||
if not name:
|
||||
sys.exit("error: could not derive a skill name from the folder; pass --name")
|
||||
|
||||
existing = next(
|
||||
(s for s in list_skills() if s["name"].lower() == name.lower()), None
|
||||
)
|
||||
if existing and not args.force:
|
||||
sys.exit(
|
||||
f"error: a skill named '{existing['name']}' already exists ({existing['id']}). "
|
||||
f"Use 'upload {existing['id']} --folder {folder}' to add a version, "
|
||||
"or pass --force to upsert a new version under it via the import key."
|
||||
)
|
||||
|
||||
file_parts = _collect_folder_files(folder)
|
||||
if not file_parts:
|
||||
sys.exit(f"error: no files to upload under {folder}")
|
||||
if not any(rel == "SKILL.md" for rel, _, _ in file_parts):
|
||||
print(
|
||||
"note: no SKILL.md at the folder root; the runtime expects "
|
||||
"'SKILL.md' as the skill entrypoint.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
fields = [("folder_name", name)]
|
||||
fields += [("paths", rel) for rel, _, _ in file_parts]
|
||||
files = [("files", rel, content, content_type) for rel, content, content_type in file_parts]
|
||||
|
||||
resp = post_multipart("/skills/upload", fields, files)
|
||||
skill_id = resp.get("id")
|
||||
if not skill_id:
|
||||
sys.exit(f"error: create response missing skill id: {resp}")
|
||||
|
||||
version_id = resp.get("active_version_id") or resp.get("latest_version_id")
|
||||
print(f"created skill {skill_id} ('{resp.get('name', name)}') with {len(file_parts)} file(s)")
|
||||
if version_id:
|
||||
print(f"active version {version_id}")
|
||||
|
||||
meta = {
|
||||
"skill_id": skill_id,
|
||||
"skill_name": resp.get("name", name),
|
||||
"version_id": version_id,
|
||||
"binary_files": [
|
||||
rel for rel, _, content_type in file_parts
|
||||
if content_type == "application/octet-stream"
|
||||
],
|
||||
}
|
||||
(folder / META_FILE).write_text(json.dumps(meta, indent=2), encoding="utf-8")
|
||||
|
||||
|
||||
def cmd_download(args: argparse.Namespace) -> None:
|
||||
skill = resolve_skill(args.skill)
|
||||
skill_id = skill["id"]
|
||||
version_id = args.version or skill.get("active_version_id")
|
||||
if not version_id:
|
||||
sys.exit(f"error: skill {skill_id} has no active version; pass --version")
|
||||
|
||||
version = get_version(skill_id, version_id)
|
||||
out = pathlib.Path(args.out or f"./{skill['name']}-{version_id[-8:]}")
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
text_files: list[dict] = []
|
||||
binary_files: list[dict] = []
|
||||
for f in version.get("files", []):
|
||||
if f.get("previewable_as_text"):
|
||||
text_files.append(f)
|
||||
else:
|
||||
binary_files.append(f)
|
||||
|
||||
for f in text_files:
|
||||
path = f["path"]
|
||||
query = urllib.parse.urlencode({"path": path})
|
||||
body = request("GET", f"/skills/{skill_id}/versions/{version_id}/content?{query}")
|
||||
dest = out / path
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(body.get("content", ""), encoding="utf-8")
|
||||
print(f" wrote {dest}")
|
||||
|
||||
meta = {
|
||||
"skill_id": skill_id,
|
||||
"skill_name": skill["name"],
|
||||
"version_id": version_id,
|
||||
"binary_files": [f["path"] for f in binary_files],
|
||||
}
|
||||
(out / META_FILE).write_text(json.dumps(meta, indent=2), encoding="utf-8")
|
||||
|
||||
print(f"downloaded {len(text_files)} text file(s) to {out}")
|
||||
if binary_files:
|
||||
paths = ", ".join(f["path"] for f in binary_files)
|
||||
print(
|
||||
f"note: {len(binary_files)} binary file(s) skipped ({paths}). "
|
||||
"Upload will preserve them via base_version_id unless --no-base is passed.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
|
||||
def cmd_upload(args: argparse.Namespace) -> None:
|
||||
folder = pathlib.Path(args.folder)
|
||||
if not folder.is_dir():
|
||||
sys.exit(f"error: {folder} is not a directory")
|
||||
|
||||
skill = resolve_skill(args.skill)
|
||||
skill_id = skill["id"]
|
||||
|
||||
base_version_id = None
|
||||
meta_path = folder / META_FILE
|
||||
if not args.no_base and meta_path.is_file():
|
||||
meta = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
if meta.get("skill_id") and meta["skill_id"] != skill_id:
|
||||
sys.exit(
|
||||
f"error: folder's meta file is for skill {meta['skill_id']}, "
|
||||
f"but upload target is {skill_id}"
|
||||
)
|
||||
base_version_id = meta.get("version_id")
|
||||
|
||||
files = []
|
||||
for entry in sorted(folder.rglob("*")):
|
||||
if entry.is_dir():
|
||||
continue
|
||||
if entry.name == META_FILE:
|
||||
continue
|
||||
rel_parts = entry.relative_to(folder).parts
|
||||
if any(part.startswith(".") for part in rel_parts):
|
||||
continue
|
||||
rel = entry.relative_to(folder).as_posix()
|
||||
try:
|
||||
content = entry.read_text(encoding="utf-8")
|
||||
except UnicodeDecodeError:
|
||||
sys.exit(
|
||||
f"error: {rel} is not text. The /skills/:id/versions endpoint "
|
||||
"only accepts text content; binary files must be preserved via "
|
||||
"base_version_id (do not pass --no-base)."
|
||||
)
|
||||
files.append({"path": rel, "content": content})
|
||||
|
||||
if not files:
|
||||
sys.exit(f"error: no files to upload under {folder}")
|
||||
|
||||
body: dict = {"files": files}
|
||||
if base_version_id:
|
||||
body["base_version_id"] = base_version_id
|
||||
|
||||
resp = request("POST", f"/skills/{skill_id}/versions", body=body)
|
||||
new_version_id = resp.get("new_version_id")
|
||||
if not new_version_id:
|
||||
sys.exit(f"error: upload response missing new_version_id: {resp}")
|
||||
|
||||
print(f"created version {new_version_id} (base: {base_version_id or 'none'})")
|
||||
|
||||
if args.activate:
|
||||
request(
|
||||
"PUT",
|
||||
f"/skills/{skill_id}/active-version",
|
||||
body={"version_id": new_version_id},
|
||||
)
|
||||
print(f"activated version {new_version_id}")
|
||||
|
||||
# Refresh local meta so a subsequent edit + upload chains correctly.
|
||||
if meta_path.is_file():
|
||||
meta = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
meta["version_id"] = new_version_id
|
||||
meta_path.write_text(json.dumps(meta, indent=2), encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(prog="exo-skill")
|
||||
parser.add_argument(
|
||||
"--profile",
|
||||
default=os.environ.get("EXO_PROFILE"),
|
||||
help="credential profile in $XDG_CONFIG_HOME/exo (default ~/.config), named after the MCP server",
|
||||
)
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
cr = sub.add_parser("create", help="create a new skill from a folder")
|
||||
cr.add_argument("--folder", required=True, help="folder to upload as the new skill")
|
||||
cr.add_argument("--name", help="skill name; defaults to the folder's basename")
|
||||
cr.add_argument(
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="upsert a new version even if a skill with this name already exists",
|
||||
)
|
||||
cr.set_defaults(func=cmd_create)
|
||||
|
||||
dl = sub.add_parser("download", help="download a skill version's bundle into a folder")
|
||||
dl.add_argument("skill", help="skill ID or name")
|
||||
dl.add_argument("--version", help="version ID; defaults to the active version")
|
||||
dl.add_argument("--out", help="output folder; defaults to ./<name>-<version-suffix>")
|
||||
dl.set_defaults(func=cmd_download)
|
||||
|
||||
up = sub.add_parser("upload", help="upload a folder as a new skill version")
|
||||
up.add_argument("skill", help="skill ID or name")
|
||||
up.add_argument("--folder", required=True, help="folder to upload")
|
||||
up.add_argument("--activate", action="store_true", help="set the new version active")
|
||||
up.add_argument(
|
||||
"--no-base",
|
||||
action="store_true",
|
||||
help="upload without using the folder's meta file as base_version_id; drops any binary files that were not redownloaded",
|
||||
)
|
||||
up.set_defaults(func=cmd_upload)
|
||||
|
||||
args = parser.parse_args()
|
||||
load_profile(args.profile)
|
||||
args.func(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user