When the user's first message is vague, exploratory, or asks what you can do ("make me a video", "what can you do?", "help me create something", "I want to make content"), read the onboarding skill **before** doing anything else:
**Read:** `skills/meta/onboarding.md`
This skill teaches you to run discovery, classify the user's setup, present capabilities in plain language, and offer starter prompts tailored to their available tools. The goal: get the user from "curious" to "making a video" in under 60 seconds.
**Skip onboarding** when the user arrives with a specific, actionable request (e.g., "Make a 60-second explainer about black holes"). Go directly to Rule Zero.
## Rule Zero — All Production Goes Through a Pipeline
**Every video production request MUST go through the pipeline system. No exceptions.**
When the user asks to make, create, produce, or generate any video content — a trailer, explainer, clip, animation, or any other video — the agent must:
1.**Identify the pipeline.** Match the request to one of the pipelines in `pipeline_defs/`. If unclear, ask the user.
2.**Read the pipeline manifest.**`pipeline_defs/<pipeline>.yaml` — know the stages, tools, and quality gates.
3.**Run preflight.** Discover available tools via the registry. Present the capability menu.
4.**Execute stage by stage.** For EACH stage, read the stage director skill (`skills/pipelines/<pipeline>/<stage>-director.md`) BEFORE doing any work in that stage.
5.**Read Layer 3 skills before calling tools.** Before using any tool with an `agent_skills` field, read the referenced skill in `.agents/skills/`. These contain provider-specific prompting guidance, parameter optimization, and quality techniques that dramatically improve output.
**Do NOT:**
- Write ad-hoc Python scripts to call tools directly
- Skip the pipeline and go straight to API calls
- Generate assets without reading the stage director skill first
- Use a tool without checking its Layer 3 skill for prompting guidance
- Bypass preflight, checkpoints, or review
The intelligence is in the skills, not in improvised code. An agent that reads the director skills and Layer 3 knowledge will produce significantly better output than one that calls tools directly with generic prompts.
OpenMontage is an instruction-driven video production system. The AI agent IS the intelligence — it reads instructions (pipeline manifests + stage director skills + meta skills) and drives the pipeline using tools.
-> checkpoints (Python utility) -> presents to human for approval
```
**Python = tools + persistence.** No orchestration logic, creative decisions, review logic, or checkpoint policy in Python code. The agent makes those decisions guided by instructions.
Core loop:
1. Select a pipeline.
2. Run preflight.
3. Discover real tools from the registry.
4. Present the user with concepts, tool plan, production plan, and cost.
For any meaningful production decision, the agent must communicate the decision before acting. The user should never have to infer which provider, model, or render path was chosen after the fact.
### Announce Before Execution
Before any paid or consequential generation call, state:
- the exact tool name,
- the provider,
- the model or provider variant,
- the reason it was chosen,
- whether it is a sample or a batch run.
### Ask Before Major Changes
The agent must ask the user before changing any major production choice, including:
- switching provider,
- switching model family or provider variant,
- switching from video-led to still-led treatment,
- switching composition engine when that changes the output character,
- dropping narration, music, or other approved creative elements,
- changing from sample mode to batch mode.
Minor prompt refinements inside an already approved provider/model path do not require separate approval unless they materially change the creative direction.
### Escalate Blockers Explicitly
When a blocker occurs, the agent must surface it immediately using this structure:
1. What was attempted
2. What failed
3. Whether the issue is auth, provider access, tool bug, or prompt/design quality
4. What options exist next
5. Which option the agent recommends, with reasoning
Do not continue with a substitute path until the user approves.
### Recommendation Style
When asking the user to choose, do not just list options. The agent should:
- provide the shortlist,
- explain the tradeoffs briefly,
- recommend one option,
- wait for approval before proceeding.
### No Unilateral Substitutions
If the approved path is blocked, the agent may investigate and prepare alternatives, but may not execute those alternatives without user approval.
This applies especially to:
- provider swaps,
- model swaps,
- fallback tools,
- prompt-only substitutes for reference-driven generation,
Every production run creates a project workspace under `projects/`. This directory is gitignored — all generated assets are regenerable.
```
projects/<project-name>/
├── artifacts/ # JSON artifacts from each stage (research_brief, script, scene_plan, etc.)
├── assets/
│ ├── images/ # Generated images (PNG)
│ ├── video/ # Generated video clips (MP4)
│ ├── audio/ # Narration segments + final mix (MP3/WAV)
│ ├── music/ # Background music track (MP3)
│ └── subtitles.srt # Generated subtitles
└── renders/
└── final.mp4 # Final rendered video (the deliverable)
```
**Naming convention**: Use kebab-case derived from the video title (e.g., `hidden-math-of-nature`, `how-music-rewires-brain`).
Create the project directory at pipeline initialization, before any stage runs. All tools and agents should write outputs to these paths — never to the repo root or ad-hoc locations.
## Music Library
Users can place royalty-free music tracks in `music_library/` (gitignored). The asset director will check this folder before falling back to API-based music generation.
```
music_library/
├── ambient_track.mp3
├── cinematic_epic.mp3
└── ...
```
If the folder has tracks, the proposal and asset stages should present them as options alongside generated music. See the proposal-director and asset-director skills for details.
This returns every capability grouped by status — how many providers the user has configured vs. how many exist. **Present this to the user as a capability menu**, not as a flat tool list.
You can produce videos now with images + TTS + FFmpeg.
Quick upgrades available — see below.
```
For EACH capability with unavailable providers, read the `install_instructions` field from the menu output and present setup options grouped by effort:
```
QUICK SETUP OPTIONS (1-minute each — set an env var in .env)
Video Generation (0/12 -> unlock the biggest upgrade):
Each unavailable provider lists its own install_instructions.
Read them from the provider_menu output and present grouped by env var.
Example: if 3 tools need FAL_KEY, group them: "FAL_KEY unlocks 3 providers"
Image Generation (1/7 -> more style options):
Same pattern — read install_instructions from each unavailable tool.
Text-to-Speech (1/3):
Same pattern.
LOCAL OPTIONS (free, needs hardware):
Tools with runtime=LOCAL or runtime=LOCAL_GPU — read from the menu.
Already Available:
List what's working. The user should feel good about what they have.
```
**Rules:**
- Do NOT hardcode provider names, API key names, or setup URLs in your prompts.
Read them from the registry's `install_instructions` field on each tool.
- Always show the ratio: "X of Y configured" — this makes breadth visible.
- Group by capability, not by individual tool.
- Show what they CAN do now, then what they COULD unlock.
- If the user declines setup, proceed with the best available path — no nagging.
- If a tool shares an env var with others, group them (read from `dependencies` field).
### Setup Offer Protocol
When tools are `UNAVAILABLE` but can be fixed with simple configuration, **offer the user setup help instead of silently working around the limitation.** Many tools are one env var away from working.
| Fix Complexity | Action |
|----------------|--------|
| **1-minute fix** (env var) | Offer to help configure now — read `install_instructions` from the tool |
| **5-minute fix** (install) | Explain what to install and why — read `install_instructions` from the tool |
| **Complex fix** (GPU, model download) | Note the limitation, explain what it would unlock, move on |
**Rules:**
- Always tell the user what they're missing AND what they'd gain
- Show the cost difference (free local vs. paid API)
- If the user declines setup, proceed with the best available path — no nagging
- Group related fixes (tools sharing the same env var dependency)
### Remotion Rendering (Inside video_compose)
`video_compose` has two render engines. Check which are available:
```bash
python -c "
from tools.tool_registry import registry
registry.discover()
info = registry._tools['video_compose'].get_info()
For any request where the deliverable inherently depends on motion rather than static coverage, treat motion as a hard requirement. Examples:
- sci-fi trailers,
- cinematic teasers built from generated clips,
- hype edits,
- avatar or agent videos,
- any brief whose promise depends on moving shots rather than still frames.
For these requests:
-`Remotion` availability must be confirmed up front if the planned visual treatment depends on Remotion rendering.
- Still-image fallback is forbidden. Do not quietly convert the job into a Ken Burns teaser, animatic, or slide-based video.
- FFmpeg-only fallback is forbidden when it changes the approved deliverable from motion-led video to still-led video.
- Bubble critical issues immediately. If Remotion is unavailable, fails to render, or provider clip generation fails in a way that blocks the approved treatment, stop and tell the user before proceeding.
- Do not spend more tokens or time on downgraded output unless the user explicitly approves the downgrade as an animatic or proof-of-concept.
**When Remotion is available**, the agent should design production plans around it:
- Explainer videos with `flat-motion-graphics` playbook -> Remotion animated scenes, not Ken Burns
- Data-driven videos -> Remotion stat cards and charts, not static image screenshots
- Any pipeline using still images -> Remotion spring animations, not FFmpeg pan-and-zoom
**When Remotion is NOT available**, `video_compose` falls back to FFmpeg Ken Burns motion on still images. This still works but produces less engaging visuals. Mention this tradeoff in the proposal.
That fallback is only acceptable when it does not violate the approved delivery shape. If the user asked for a motion-led trailer or comparable clip-driven piece, the project is blocked until Remotion is working or the user explicitly approves a lower-fidelity alternative.
The routing is automatic — the `render` operation in `video_compose` calls `_needs_remotion()` and routes accordingly. But the **agent must know Remotion exists at proposal time** so it can design the visual approach to take advantage of it (animated text cards, component scenes, spring transitions) rather than designing around static images.
## Capability Discovery
OpenMontage uses two layers for capability choice:
- selector tools: capability-level routing such as `tts_selector` and `video_selector`
- provider tools: concrete tools discovered via the registry that call a specific backend
- **analysis** — Transcription, scene detection, frame sampling.
- **avatar** — Talking head and lip sync generation.
- **enhancement** — Upscale, background removal, face enhance, color grading.
Each tool in the registry declares `best_for`, `install_instructions`, `runtime` (LOCAL, API, LOCAL_GPU, HYBRID), and `status`. Read these fields — do not assume tool strengths from memory.
Three selector tools abstract multi-provider capabilities. **Selectors auto-discover providers from the registry.** Adding a new provider tool automatically makes it available through the selector — no selector code changes needed.
Music is a critical part of any video. **Surface the music situation to the user at proposal/idea time** — do not silently defer it to the asset stage where a failure becomes expensive.
Check music availability in this order and present the options:
1.**User music library (`music_library/`):** Check if this folder exists and contains tracks. If so, list available tracks with durations and let the user pick one.
2.**Music generation APIs:** Check which music tools are available via the registry (`registry.get_by_capability("music_generation")`). Report their status honestly — include quota status if known.
3.**Royalty-free sources:** Note if the user can provide their own track (e.g., from YouTube Audio Library, Jamendo, or other free sources). Offer the `music_library/` drop path.
**Always present the user with explicit choices:**
- Use a track from their library (which one?)
- Provide a different track (drop it in `music_library/`)
- Generate one via API (if available — name the provider and cost)
- Proceed without music
**If no music source is available:** Tell the user explicitly. Do NOT let this surface as a surprise at the asset stage.
Record the music decision in the proposal/brief artifact so the asset director knows what to do.
Each pipeline manifest's `tools_available` field declares what tools a stage can use. Use selectors for multi-provider capabilities — the selector handles routing to whatever is available. Read the pipeline manifest for the authoritative list per stage.
## Stage Agents
Each stage produces one canonical artifact that becomes the contract for the next stage. The stage director skill teaches the agent HOW to produce it.
| Stage | Director Skill | Canonical output | Core quality bar |
1. registry / tool contract — discover what's available
2. relevant pipeline or creative skill — know HOW to use it in this context
3. underlying vendor skill — **mandatory before calling any generation tool**
**Layer 3 is not optional.** Every generation tool (video, image, TTS, music) has an `agent_skills` field listing its Layer 3 skills. These skills contain provider-specific prompt engineering, parameter tuning, and quality techniques. Read them before writing prompts. The difference between a generic prompt and a skill-informed prompt is the difference between "usable" and "cinematic."
Example: Before calling `kling_video`, read its `agent_skills` → `ai-video-gen` → get Kling-specific prompt structure, camera direction syntax, and quality keywords that the model responds to best.
- **Do not bypass the pipeline.** Never write ad-hoc scripts to call tools directly. All production goes through pipeline stages with director skills. See Rule Zero.
- **Do not call generation tools without reading their Layer 3 skill.** Check the tool's `agent_skills` field, read the referenced skill, then craft your prompts using that guidance.
- **Do not skip stage director skills.** Before executing any pipeline stage, read its director skill. The skill contains the quality bar, the workflow, and the review criteria.