OpenMontage/AGENT_GUIDE.md

OpenMontage - Agent Guide

Start here. This is the complete operating guide and agent contract for OpenMontage.

For architecture, key files, and conventions see PROJECT_CONTEXT.md.

What OpenMontage Is

OpenMontage is an instruction-driven video production system. The AI agent IS the intelligence — it reads instructions (pipeline manifests + stage director skills + meta skills) and drives the pipeline using tools.

Agent reads pipeline manifest (YAML) -> reads stage director skill (MD)
-> uses tools (Python BaseTool subclasses) -> self-reviews (meta skill)
-> checkpoints (Python utility) -> presents to human for approval

Python = tools + persistence. No orchestration logic, creative decisions, review logic, or checkpoint policy in Python code. The agent makes those decisions guided by instructions.

Core loop:

1. Select a pipeline.
2. Run preflight.
3. Discover real tools from the registry.
4. Present the user with concepts, tool plan, production plan, and cost.
5. Execute stage by stage with checkpoints.

Orchestrator

The agent itself orchestrates the production state machine:

research -> proposal -> script -> scene_plan -> assets -> edit -> compose

The agent:

1. Reads the pipeline manifest (pipeline_defs/*.yaml) to know the process
2. Calls checkpoint.get_next_stage() to find where to resume
3. Reads the stage's director skill (skills/pipelines/<pipeline>/<stage>-director.md) to know HOW
4. Uses tools (tools/) for concrete capabilities
5. Self-reviews using the reviewer meta skill (skills/meta/reviewer.md)
6. Checkpoints via the checkpoint protocol (skills/meta/checkpoint-protocol.md)
7. Presents to human for approval when human_approval_default: true

Infrastructure files:

• lib/checkpoint.py — read/write checkpoints, stage validation
• tools/cost_tracker.py — budget governance
• lib/pipeline_loader.py — manifest loading and helpers

Available Pipelines

Pipeline	Best For	Stability
animated-explainer	Topic to fully generated explainer	production
talking-head	Footage-led speaker videos	beta
screen-demo	Screen recordings and walkthroughs	production
clip-factory	Many clips from one long source	beta
podcast-repurpose	Podcast highlights and derivatives	beta
cinematic	Trailer, teaser, and mood-led edits	production
animation	Motion-graphics and animation-first videos	production
hybrid	Source footage plus support visuals	production
avatar-spokesperson	Presenter-led avatar or lip-sync videos	production
localization-dub	Subtitle, dub, and translated variants	beta
framework-smoke	Test: minimal 2-stage smoke test	test

Beta pipelines have not been fully audited. They work, but expect rough edges. Mention this when the user selects one.

Mandatory Preflight

Do this before any creative work:

python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.support_envelope(), indent=2))"

Then:

1. Read the selected manifest in pipeline_defs/.
2. Check every required_tools entry against the registry.
3. Check fallback_tools for unavailable tools.
4. Report one of: passed, degraded, or blocked.
5. Do not start production until the user understands the real capability envelope.

Provider Menu (Mandatory at Preflight)

After running support_envelope(), run the provider menu to see the full picture:

python -c "
from tools.tool_registry import registry
import json
registry.discover()
print(json.dumps(registry.provider_menu(), indent=2))
"

This returns every capability grouped by status — how many providers the user has configured vs. how many exist. Present this to the user as a capability menu, not as a flat tool list.

How to present:

YOUR CAPABILITIES

  Video Generation:  0/12 configured
  Image Generation:  1/7 configured
  Text-to-Speech:    1/3 configured
  Music Generation:  1/1 configured
  Composition:       3/3 configured (FFmpeg, video_stitch, video_trimmer)

  You can produce videos now with images + TTS + FFmpeg.
  Quick upgrades available — see below.

For EACH capability with unavailable providers, read the install_instructions field from the menu output and present setup options grouped by effort:

QUICK SETUP OPTIONS (1-minute each — set an env var in .env)

  Video Generation (0/12 -> unlock the biggest upgrade):
    Each unavailable provider lists its own install_instructions.
    Read them from the provider_menu output and present grouped by env var.
    Example: if 3 tools need FAL_KEY, group them: "FAL_KEY unlocks 3 providers"

  Image Generation (1/7 -> more style options):
    Same pattern — read install_instructions from each unavailable tool.

  Text-to-Speech (1/3):
    Same pattern.

LOCAL OPTIONS (free, needs hardware):
  Tools with runtime=LOCAL or runtime=LOCAL_GPU — read from the menu.

Already Available:
  List what's working. The user should feel good about what they have.

Rules:

• Do NOT hardcode provider names, API key names, or setup URLs in your prompts. Read them from the registry's install_instructions field on each tool.
• Always show the ratio: "X of Y configured" — this makes breadth visible.
• Group by capability, not by individual tool.
• Show what they CAN do now, then what they COULD unlock.
• If the user declines setup, proceed with the best available path — no nagging.
• If a tool shares an env var with others, group them (read from dependencies field).

Setup Offer Protocol

When tools are UNAVAILABLE but can be fixed with simple configuration, offer the user setup help instead of silently working around the limitation. Many tools are one env var away from working.

Fix Complexity	Action
1-minute fix (env var)	Offer to help configure now — read install_instructions from the tool
5-minute fix (install)	Explain what to install and why — read install_instructions from the tool
Complex fix (GPU, model download)	Note the limitation, explain what it would unlock, move on

Rules:

• Always tell the user what they're missing AND what they'd gain
• Show the cost difference (free local vs. paid API)
• If the user declines setup, proceed with the best available path — no nagging
• Group related fixes (tools sharing the same env var dependency)

Remotion Rendering (Inside video_compose)

video_compose has two render engines. Check which are available:

python -c "
from tools.tool_registry import registry
registry.discover()
info = registry._tools['video_compose'].get_info()
print('Render engines:', info.get('render_engines'))
print('Remotion note:', info.get('remotion_note'))
"

Engine	Used For	Requires
FFmpeg	Video-only cuts, concat, trim, subtitle burn	ffmpeg binary (always available)
Remotion	Still images -> animated video, text cards, stat cards, charts, callouts, comparisons, transitions with spring physics	Node.js (npx) + remotion-composer/ project

When Remotion is available, the agent should design production plans around it:

• Explainer videos with flat-motion-graphics playbook -> Remotion animated scenes, not Ken Burns
• Data-driven videos -> Remotion stat cards and charts, not static image screenshots
• Any pipeline using still images -> Remotion spring animations, not FFmpeg pan-and-zoom

When Remotion is NOT available, video_compose falls back to FFmpeg Ken Burns motion on still images. This still works but produces less engaging visuals. Mention this tradeoff in the proposal.

The routing is automatic — the render operation in video_compose calls _needs_remotion() and routes accordingly. But the agent must know Remotion exists at proposal time so it can design the visual approach to take advantage of it (animated text cards, component scenes, spring transitions) rather than designing around static images.

Capability Discovery

OpenMontage uses two layers for capability choice:

• selector tools: capability-level routing such as tts_selector and video_selector
• provider tools: concrete tools discovered via the registry that call a specific backend

Always inspect the registry first:

python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.capability_catalog(), indent=2))"
python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.provider_catalog(), indent=2))"

For finalist tools inspect:

• capability
• provider
• usage_location
• supports
• fallback_tools
• related_skills

Do not rely on memory or old docs when the registry can answer it.

Tool Families

Do not maintain hardcoded tool lists. Always query the registry at runtime:

# See all tools grouped by capability (TTS, video_generation, image_generation, etc.)
python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.capability_catalog(), indent=2))"

# See all tools grouped by provider (elevenlabs, openai, ffmpeg, etc.)
python -c "from tools.tool_registry import registry; import json; registry.discover(); print(json.dumps(registry.provider_catalog(), indent=2))"

Key capability families to look for in the output:

• tts — Text-to-speech providers. Route via tts_selector.
• video_generation — Video generation providers (cloud, local GPU, stock). Route via video_selector.
• image_generation — Image generation providers (cloud, local GPU, stock). Route via image_selector.
• music_generation — Music and sound effect generation.
• video_post — Composition, stitching, trimming (FFmpeg-based, always local).
• audio_processing — Mixing, enhancement (FFmpeg-based, always local).
• analysis — Transcription, scene detection, frame sampling.
• avatar — Talking head and lip sync generation.
• enhancement — Upscale, background removal, face enhance, color grading.

Each tool in the registry declares best_for, install_instructions, runtime (LOCAL, API, LOCAL_GPU, HYBRID), and status. Read these fields — do not assume tool strengths from memory.

Selector Pattern

Three selector tools abstract multi-provider capabilities. Selectors auto-discover providers from the registry. Adding a new provider tool automatically makes it available through the selector — no selector code changes needed.

Selector	Routes to	How it discovers
tts_selector	All tools with capability="tts" (ElevenLabs, Google TTS, OpenAI, Piper)	registry.get_by_capability("tts")
image_selector	All tools with capability="image_generation" (FLUX, Google Imagen, DALL-E, Recraft, etc.)	registry.get_by_capability("image_generation")
video_selector	All tools with capability="video_generation"	registry.get_by_capability("video_generation")

Selectors route based on: user preference > availability > discovery order. They adapt input schemas between providers transparently.

User-Facing Planning Protocol

Before committing to execution, present:

1. 4-5 concept directions when the brief is still open.
2. Recommended pipeline.
3. Recommended tool path.
4. Alternative tool paths that are actually available.
5. Cost estimate and quality tradeoffs.
6. Production plan by stage.
7. Approval gate before asset generation.

If a user prefers a specific vendor and that tool is available, surface it directly. Do not hide provider choice.

Pipeline Asset Expectations

Each pipeline manifest's tools_available field declares what tools a stage can use. Use selectors for multi-provider capabilities — the selector handles routing to whatever is available. Read the pipeline manifest for the authoritative list per stage.

Stage Agents

Each stage produces one canonical artifact that becomes the contract for the next stage. The stage director skill teaches the agent HOW to produce it.

Stage	Director Skill	Canonical output	Core quality bar
idea	*-director.md	brief	Clear hook, target platform, duration, tone, and user intent
script	*-director.md	script	Structured sections, valid timing, coherent narration
scene_plan	*-director.md	scene_plan	Ordered scenes, timings, asset requirements
assets	*-director.md	asset_manifest	Provenance, paths, model/tool metadata, scene linkage
edit	*-director.md	edit_decisions	Concrete cuts, overlays, subtitle/music decisions
compose	*-director.md	render_report	Output paths, encoding profile, verification notes

Stage contract rules:

• A completed or awaiting-human checkpoint must include the stage's canonical artifact.
• Canonical artifacts must validate against the JSON schema in schemas/artifacts/.
• Non-canonical outputs such as media files belong in stage-specific directories.
• Tools should record seeds/model versions for reproducibility.

Reviewer Protocol

The reviewer is a meta skill (skills/meta/reviewer.md) — advisory, never directly blocks progression.

• Self-review after every stage execution, before checkpointing.
• Load review_focus items from the pipeline manifest for the current stage.
• Maximum two review rounds. After that, pass with warnings and move on.
• Findings categorized: critical (must fix), suggestion (should fix), nitpick (nice-to-have).
• Critical findings -> fix and re-review. Suggestions -> note and proceed.
• Check playbook quality_rules as constraints, not suggestions.

Human Checkpoint Protocol

The checkpoint protocol meta skill (skills/meta/checkpoint-protocol.md) teaches the agent when to pause:

• Read human_approval_default from the pipeline manifest per stage
• Creative stages (idea, script, scene_plan) typically require approval
• Technical stages (assets, edit, compose) typically auto-proceed
• When approval is required: present artifact summary, review findings, and cost snapshot
• Wait for human to approve, request revision, or abort

Communication Protocol

Agents coordinate through canonical JSON artifacts, checkpoints, pipeline manifests, and the tool registry.

Primary files:

• Artifact schemas: schemas/artifacts/
• Checkpoint schema: schemas/checkpoints/checkpoint.schema.json
• Pipeline manifest schema: schemas/pipelines/pipeline_manifest.schema.json
• Pipeline manifests: pipeline_defs/
• Style playbooks: styles/*.yaml (validated by schemas/styles/playbook.schema.json)
• Tool contract: tools/base_tool.py
• Tool registry: tools/tool_registry.py
• Stage director skills: skills/pipelines/<pipeline>/<stage>-director.md
• Meta skills: skills/meta/*.md

Checkpoint rules:

• Checkpoints live at pipelines/<project_id>/checkpoint_<stage>.json.
• status may be completed, failed, awaiting_human, or in_progress.
• completed and awaiting_human checkpoints must include the canonical artifact.
• Invalid checkpoints or invalid canonical artifacts are contract violations and should fail fast.

Pipeline manifest rules:

• Pipelines are declarative YAML manifests in pipeline_defs/.
• Stages declare: skill (director skill path), produces, tools_available, review_focus, success_criteria, human_approval_default.
• Adding a new pipeline requires a manifest + stage director skills.

Tool rules:

• Every production tool must inherit from BaseTool.
• Tool discovery flows through the registry, not ad hoc imports.
• Support-envelope reporting is the source of truth for capability, status, and resource requirements.

Style Playbooks

Playbook	Best For
clean-professional	Corporate, educational, SaaS
flat-motion-graphics	Social media, TikTok, startups
minimalist-diagram	Technical deep-dives, architecture

Layer Map

OpenMontage has three instruction layers:

1. tools/ What exists, what is available, cost, runtime, fallback, related skills.
2. skills/ How OpenMontage wants those tools used in pipelines.
3. .agents/skills/ Raw vendor or technology knowledge.

Reading order:

1. registry / tool contract
2. relevant pipeline or creative skill
3. underlying vendor skill only if needed

Quick Lookup

Question	Where to look
What tools exist?	tools/tool_registry.py and registry.support_envelope()
What providers are available for a capability?	registry.capability_catalog()
What tools exist for a vendor?	registry.provider_catalog()
How does a tool actually work?	the tool's usage_location from the registry
How should this pipeline stage behave?	skills/pipelines/<pipeline>/...
What is the checkpoint/review policy?	skills/meta/

What Not To Do

• Do not use deleted legacy names such as tts_cloud, tts_engine, or video_gen.
• Do not hardcode provider names, API key names, or setup URLs. Read them from the registry's install_instructions and dependencies fields.
• Do not begin asset generation before user approval on the production plan.
• Do not hide degraded paths. Record substitutions and blocked options explicitly.
• Do not present a single unavailable tool in isolation. Always show the full capability picture: "X of Y providers configured for this capability."
• Do not skip the Provider Menu at preflight. The user must see what they have AND what they could unlock.