## Rule Zero — All Production Goes Through a Pipeline
**Every video production request MUST go through the pipeline system. No exceptions.**
When the user asks to make, create, produce, or generate any video content — a trailer, explainer, clip, animation, or any other video — the agent must:
1.**Identify the pipeline.** Match the request to one of the pipelines in `pipeline_defs/`. If unclear, ask the user.
2.**Read the pipeline manifest.**`pipeline_defs/<pipeline>.yaml` — know the stages, tools, and quality gates.
3.**Run preflight.** Discover available tools via the registry. Present the capability menu.
4.**Execute stage by stage.** For EACH stage, read the stage director skill (`skills/pipelines/<pipeline>/<stage>-director.md`) BEFORE doing any work in that stage.
5.**Read Layer 3 skills before calling tools.** Before using any tool with an `agent_skills` field, read the referenced skill in `.agents/skills/`. These contain provider-specific prompting guidance, parameter optimization, and quality techniques that dramatically improve output.
**Do NOT:**
- Write ad-hoc Python scripts to call tools directly
- Skip the pipeline and go straight to API calls
- Generate assets without reading the stage director skill first
- Use a tool without checking its Layer 3 skill for prompting guidance
- Bypass preflight, checkpoints, or review
The intelligence is in the skills, not in improvised code. An agent that reads the director skills and Layer 3 knowledge will produce significantly better output than one that calls tools directly with generic prompts.
OpenMontage is an instruction-driven video production system. The AI agent IS the intelligence — it reads instructions (pipeline manifests + stage director skills + meta skills) and drives the pipeline using tools.
-> checkpoints (Python utility) -> presents to human for approval
```
**Python = tools + persistence.** No orchestration logic, creative decisions, review logic, or checkpoint policy in Python code. The agent makes those decisions guided by instructions.
Core loop:
1. Select a pipeline.
2. Run preflight.
3. Discover real tools from the registry.
4. Present the user with concepts, tool plan, production plan, and cost.
5. Execute stage by stage with checkpoints.
## Orchestrator
The agent itself orchestrates the production state machine:
Every production run creates a project workspace under `projects/`. This directory is gitignored — all generated assets are regenerable.
```
projects/<project-name>/
├── artifacts/ # JSON artifacts from each stage (research_brief, script, scene_plan, etc.)
├── assets/
│ ├── images/ # Generated images (PNG)
│ ├── video/ # Generated video clips (MP4)
│ ├── audio/ # Narration segments + final mix (MP3/WAV)
│ ├── music/ # Background music track (MP3)
│ └── subtitles.srt # Generated subtitles
└── renders/
└── final.mp4 # Final rendered video (the deliverable)
```
**Naming convention**: Use kebab-case derived from the video title (e.g., `hidden-math-of-nature`, `how-music-rewires-brain`).
Create the project directory at pipeline initialization, before any stage runs. All tools and agents should write outputs to these paths — never to the repo root or ad-hoc locations.
## Music Library
Users can place royalty-free music tracks in `music_library/` (gitignored). The asset director will check this folder before falling back to API-based music generation.
```
music_library/
├── ambient_track.mp3
├── cinematic_epic.mp3
└── ...
```
If the folder has tracks, the proposal and asset stages should present them as options alongside generated music. See the proposal-director and asset-director skills for details.
This returns every capability grouped by status — how many providers the user has configured vs. how many exist. **Present this to the user as a capability menu**, not as a flat tool list.
You can produce videos now with images + TTS + FFmpeg.
Quick upgrades available — see below.
```
For EACH capability with unavailable providers, read the `install_instructions` field from the menu output and present setup options grouped by effort:
```
QUICK SETUP OPTIONS (1-minute each — set an env var in .env)
Video Generation (0/12 -> unlock the biggest upgrade):
Each unavailable provider lists its own install_instructions.
Read them from the provider_menu output and present grouped by env var.
Example: if 3 tools need FAL_KEY, group them: "FAL_KEY unlocks 3 providers"
Image Generation (1/7 -> more style options):
Same pattern — read install_instructions from each unavailable tool.
Text-to-Speech (1/3):
Same pattern.
LOCAL OPTIONS (free, needs hardware):
Tools with runtime=LOCAL or runtime=LOCAL_GPU — read from the menu.
Already Available:
List what's working. The user should feel good about what they have.
```
**Rules:**
- Do NOT hardcode provider names, API key names, or setup URLs in your prompts.
Read them from the registry's `install_instructions` field on each tool.
- Always show the ratio: "X of Y configured" — this makes breadth visible.
- Group by capability, not by individual tool.
- Show what they CAN do now, then what they COULD unlock.
- If the user declines setup, proceed with the best available path — no nagging.
- If a tool shares an env var with others, group them (read from `dependencies` field).
### Setup Offer Protocol
When tools are `UNAVAILABLE` but can be fixed with simple configuration, **offer the user setup help instead of silently working around the limitation.** Many tools are one env var away from working.
| Fix Complexity | Action |
|----------------|--------|
| **1-minute fix** (env var) | Offer to help configure now — read `install_instructions` from the tool |
| **5-minute fix** (install) | Explain what to install and why — read `install_instructions` from the tool |
| **Complex fix** (GPU, model download) | Note the limitation, explain what it would unlock, move on |
**Rules:**
- Always tell the user what they're missing AND what they'd gain
- Show the cost difference (free local vs. paid API)
- If the user declines setup, proceed with the best available path — no nagging
- Group related fixes (tools sharing the same env var dependency)
### Remotion Rendering (Inside video_compose)
`video_compose` has two render engines. Check which are available:
```bash
python -c "
from tools.tool_registry import registry
registry.discover()
info = registry._tools['video_compose'].get_info()
| **Remotion** | Still images -> animated video, text cards, stat cards, charts, callouts, comparisons, transitions with spring physics | Node.js (`npx`) + `remotion-composer/` project |
**When Remotion is available**, the agent should design production plans around it:
- Explainer videos with `flat-motion-graphics` playbook -> Remotion animated scenes, not Ken Burns
- Data-driven videos -> Remotion stat cards and charts, not static image screenshots
- Any pipeline using still images -> Remotion spring animations, not FFmpeg pan-and-zoom
**When Remotion is NOT available**, `video_compose` falls back to FFmpeg Ken Burns motion on still images. This still works but produces less engaging visuals. Mention this tradeoff in the proposal.
The routing is automatic — the `render` operation in `video_compose` calls `_needs_remotion()` and routes accordingly. But the **agent must know Remotion exists at proposal time** so it can design the visual approach to take advantage of it (animated text cards, component scenes, spring transitions) rather than designing around static images.
## Capability Discovery
OpenMontage uses two layers for capability choice:
- selector tools: capability-level routing such as `tts_selector` and `video_selector`
- provider tools: concrete tools discovered via the registry that call a specific backend
- **analysis** — Transcription, scene detection, frame sampling.
- **avatar** — Talking head and lip sync generation.
- **enhancement** — Upscale, background removal, face enhance, color grading.
Each tool in the registry declares `best_for`, `install_instructions`, `runtime` (LOCAL, API, LOCAL_GPU, HYBRID), and `status`. Read these fields — do not assume tool strengths from memory.
Three selector tools abstract multi-provider capabilities. **Selectors auto-discover providers from the registry.** Adding a new provider tool automatically makes it available through the selector — no selector code changes needed.
Music is a critical part of any video. **Surface the music situation to the user at proposal/idea time** — do not silently defer it to the asset stage where a failure becomes expensive.
Check music availability in this order and present the options:
1.**User music library (`music_library/`):** Check if this folder exists and contains tracks. If so, list available tracks with durations and let the user pick one.
2.**Music generation APIs:** Check which music tools are available via the registry (`registry.get_by_capability("music_generation")`). Report their status honestly — include quota status if known.
3.**Royalty-free sources:** Note if the user can provide their own track (e.g., from YouTube Audio Library, Jamendo, or other free sources). Offer the `music_library/` drop path.
**Always present the user with explicit choices:**
- Use a track from their library (which one?)
- Provide a different track (drop it in `music_library/`)
- Generate one via API (if available — name the provider and cost)
- Proceed without music
**If no music source is available:** Tell the user explicitly. Do NOT let this surface as a surprise at the asset stage.
Record the music decision in the proposal/brief artifact so the asset director knows what to do.
Each pipeline manifest's `tools_available` field declares what tools a stage can use. Use selectors for multi-provider capabilities — the selector handles routing to whatever is available. Read the pipeline manifest for the authoritative list per stage.
## Stage Agents
Each stage produces one canonical artifact that becomes the contract for the next stage. The stage director skill teaches the agent HOW to produce it.
| Stage | Director Skill | Canonical output | Core quality bar |
1. registry / tool contract — discover what's available
2. relevant pipeline or creative skill — know HOW to use it in this context
3. underlying vendor skill — **mandatory before calling any generation tool**
**Layer 3 is not optional.** Every generation tool (video, image, TTS, music) has an `agent_skills` field listing its Layer 3 skills. These skills contain provider-specific prompt engineering, parameter tuning, and quality techniques. Read them before writing prompts. The difference between a generic prompt and a skill-informed prompt is the difference between "usable" and "cinematic."
Example: Before calling `kling_video`, read its `agent_skills` → `ai-video-gen` → get Kling-specific prompt structure, camera direction syntax, and quality keywords that the model responds to best.
- **Do not bypass the pipeline.** Never write ad-hoc scripts to call tools directly. All production goes through pipeline stages with director skills. See Rule Zero.
- **Do not call generation tools without reading their Layer 3 skill.** Check the tool's `agent_skills` field, read the referenced skill, then craft your prompts using that guidance.
- **Do not skip stage director skills.** Before executing any pipeline stage, read its director skill. The skill contains the quality bar, the workflow, and the review criteria.
