All of these changes came out of running the P1 and P2 documentary-montage
audits end-to-end and watching specific things break. Grouping them into
one commit because they share a theme: making the corpus builder and its
stock source adapters robust enough that a real brief can produce a
real corpus without special-casing.
corpus_builder.py + new clip_cache.py + test_clip_cache.py
Phase 1 of the shared-corpus architecture. Adds
~/.openmontage/clips_cache/ — a process-safe, LRU-evicted cache of
downloaded clip files keyed by clip_id. Before each candidate download,
corpus_builder asks the cache whether the bytes already exist on disk
from a previous run; on a hit it hard-links (or copies on cross-drive)
the blob into the caller's corpus dir and skips the network fetch. On
a miss it downloads as usual and ingests the fresh file. Motivation:
re-running the P1 audit after every tool fix was re-downloading gigs
of archive.org footage that had already been fetched in the prior run.
Cache faults never block the pipeline — they degrade gracefully to
normal downloads. The cache bubbles counters into the corpus_builder
return payload so the production report can show hit/miss/bytes-saved.
Default 20 GB cap, overridable via OPENMONTAGE_CACHE_MAX_GB.
Full test coverage: try_link, ingest, stats, LRU eviction, manifest
persistence, lock behavior. 23 tests, tmp_path-scoped.
stock_sources/archive_org.py
1. Three-strategy query cascade (phrase_prox_10 → distinctive_and →
distinctive_or). Motivation: natural-language documentary queries
against archive.org Solr were zeroing out — "1950s family watching
television" returned 0 hits because Solr's default multi-term AND
over-intersects. Walks strict to loose and returns the first
non-empty strategy. Stop words, source hints ("prelinger",
"archive", "footage"), and year tokens ("1950s") are excluded from
the distinctive-token picks since they don't correlate with
Prelinger title tokens.
2. 150 MB per-rendition size cap. archive.org routinely hosts
multi-hundred-megabyte h.264 masters and one 2 GB Prelinger item
poisons corpus build wall-time and disk. Within a format bucket
we now pick the largest rendition under the cap; if nothing fits
we fall through to the next format rather than dropping the item.
3. 180 s default max-duration ceiling when the caller hasn't set one
— archive.org is the only source that routinely hosts feature-
length material and a naive fan-out pulls them into corpora that
only ever want a few seconds per clip.
stock_sources/wikimedia.py
Parallel 3-strategy cascade (full → top2_or → single_best). Motivation:
Commons CirrusSearch also defaults to AND across multi-word queries;
our first P2 diagnostic pass returned 0 video results for 10/10
queries. Same stop-word / source-hint / year-token stripping as
archive_org so the two adapters stay symmetric.
test_stock_source_adapters.py
Rewrote the wikimedia query-builder tests against the new cascade
API. Added coverage for multi-word fallback + source-hint/year
stripping.
video_compose.py
Two small fixes for the Remotion renderer on Windows:
1. Resolve output_path to absolute before invoking the CLI so the
binary can write wherever the caller asked regardless of cwd.
2. Pass cwd=composer_dir to run_command so npx can find the local
Remotion binary under node_modules/.bin. Without this, Windows
npx returns "could not determine executable to run" because it
resolves .bin relative to the process cwd rather than the
script's parent.
User override during the audit runs made music and an end-tag MANDATORY
defaults for documentary-montage (narration stays optional). This commit
codifies those defaults into the pipeline manifest + director skills and
ships the Remotion end-tag component that the compose stage now
concatenates after the FFmpeg body.
Changes:
- pipeline_defs/documentary-montage.yaml: idea/edit/compose stages now
require music_plan and end_tag_plan to be present; opt-out requires
an explicit user note recorded in metadata.
- skills/pipelines/documentary-montage/idea-director.md: rewrote sections
4 (music MANDATORY), 5 (end-tag MANDATORY with shape
{text,palette,duration_seconds,render_engine:remotion,component:EndTag}),
6 (narration OPTIONAL), 7 (updated brief JSON shape), 8 (quality gate).
Common Pitfalls grew two entries covering silent-contract drift and
end-tag omission.
- skills/pipelines/documentary-montage/compose-director.md: section 4
now stops on music-contract violation; new section 4b documents the
two-engine flow (FFmpeg body -> npx remotion render EndTag -> ffmpeg
concat). Quality gate checks render_report.metadata.music_mixed and
end_tag_rendered.
- remotion-composer/src/components/EndTag.tsx (new): bold uppercase
typographic end-card with animated 0->100% underline draw-in and a
single left-to-right shimmer sweep. Supports cool_offwhite_on_black
and warm_ivory_on_black palettes. Renders at 1920x1080 @ 30fps.
- remotion-composer/src/Root.tsx: register EndTag as a first-class
composition (durationInFrames=165, 5.5s hold) so it can be rendered
standalone via the Remotion CLI with --props overrides.
New end-to-end pipeline for building thematic documentary montages from
a locally-indexed corpus of free stock footage (Pexels, Archive.org,
NASA). The agent builds a project-local corpus, CLIP-ranks candidates
per scene slot, edits with motion-aware arc logic, and composes via
ffmpeg. No paid APIs required for the full path.
Pipeline definition and director skills:
- pipeline_defs/documentary-montage.yaml: 5-stage manifest
(idea -> scene_plan -> assets -> edit -> compose)
- skills/pipelines/documentary-montage/: 6 director skills
(executive-producer + idea/scene/asset/edit/compose directors)
Corpus and retrieval infrastructure:
- tools/video/corpus_builder.py: multi-source stock fan-out with
resumable append-only corpus index
- tools/video/clip_search.py: CLIP ViT-B/32 retrieval —
rank_for_slot, find_similar_set, diversify, stats
- tools/video/stock_sources/: base + pexels + archive_org + nasa
adapters with a pluggable BaseStockSource contract
- lib/clip_embedder.py: CLIP wrapper
- lib/corpus.py: corpus schema, jsonl append/read, motion-score
caching
video_compose fix rolled in because any concat-based pipeline depends
on it:
- Replace ambiguous -to with -t duration (was double-trimming cuts)
- Force re-encode + normalize to 1920x1080 @ 30fps (was keyframe-
snapping with -c copy and breaking concat on mixed-source corpora)
- Add silent-audio anullsrc fallback for clips without an audio
stream
README: add Documentary Montage row to the pipeline table and bump
the pipeline count from 11 to 12.
- New: Higgsfield video provider with multi-model routing (Kling 3.0, Veo 3.1, Sora 2, WAN 2.5, Soul Cinema) and Soul ID character consistency
- Updated: Runway provider with gen4_aleph and gen3a_turbo models, proper pixel-ratio mapping, probe_output, watermark param, RUNWAYML_API_SECRET env var support
- Docs: Updated provider counts (12→13), tool counts (51→52), added Higgsfield setup/pricing sections across README, AGENT_GUIDE, ARCHITECTURE, and PROVIDERS
New capture layer for the screen-demo pipeline: FFmpeg for quick CLI-driven
recording, Cap integration for polished recordings with webcam overlay and
cursor effects. Selector presents both options and routes based on availability.
Route Chirp and Journey voice families to the v1beta1 API endpoint
automatically. Adds beta voice detection, dynamic API version
selection, and Chirp3-HD cost estimation ($30/1M chars).
Compose-director had contradictory instructions: Step 2 described Remotion
captions/audio, but Steps 5/5b gave detailed FFmpeg code that agents followed
instead. This caused three failures in production: FFmpeg subtitles instead of
Remotion CaptionOverlay, missing audio (mixed externally but never embedded in
Remotion props), and skipped audio verification in post-render review.
Changes:
- compose-director: Remotion is now DEFAULT for audio, captions, text overlays;
FFmpeg is labeled FALLBACK only. Post-render review has mandatory ffprobe gate
and audio transcription with explicit stop conditions.
- remotion.md: routing table updated (captions/audio → Remotion), added universal
Post-Render Verification Protocol for all pipelines (only 2/10 had one).
- scene-director, asset-director: added pitfall for AI-generated text in CTA
screens — must use Remotion text_card for verbatim text.
- image-provider-usage: added Recraft V4 caveat (style param causes 422 on fal.ai).
- recraft_image.py: documented the style parameter 422 issue inline.
- AGENT_GUIDE.md: add "never read source code" rule — skills are the
interface, not .py files
- animation.yaml: add Layer 2 skill-first guardrail in assets stage
- video-reference-analyst.md: Step 4b now mandates Layer 2 before Layer 3,
explicitly forbids reading implementation code
- CinematicRenderer: add TikTok-style CaptionOverlay and separate music
track support (narration + music as independent audio layers)
- cinematic/types.ts: add CinematicCaptionConfig and music prop types
Chirp3-HD is near-free ($0.003 vs $0.30+ per script), expressive,
24kHz, and available without quota limits. ElevenLabs demoted to
voice-cloning-only recommendation.
New Step 4b requires the agent to read agent_skills from every tool
before writing generation prompts. Discovered during UAT: agent
generated video clips, images, and TTS without reading provider-specific
prompting guidance, violating AGENT_GUIDE governance.
Proposals must now specify clip duration strategy (prefer 10s over 5s
to halve API costs) and list Layer 3 skills that must be read before
generating assets. Both were governance gaps found during UAT.
The instructions parameter was passed as None when not provided,
causing a 400 error. Now only included when explicitly set. Also
adds speed passthrough support.
test_01_tts.py (ElevenLabs), test_02_image_gen.py (DALL-E + FLUX),
and test_03_music.py (ElevenLabs Music) were burning ~$0.44 per run
against live API keys. These were run repeatedly and exhausted the
ElevenLabs starter plan quota (30K characters).
Agent must run tts_selector preflight to check available providers
instead of defaulting to ElevenLabs. Supports ElevenLabs, Google TTS,
OpenAI TTS, and Piper (offline).
The agent must lock the audio approach (single narrator vs. character
dialogue vs. both) during Step 3, not defer it to script/compose stage.
Proposals now include voice casting with specific voice IDs.
Remotion is now the default composition engine for ALL final renders
when available — video clips, images, mixed content. FFmpeg is only
used as fallback when Remotion is not installed or for standalone
operations (trim, transcode). Also fixes three FFmpeg fallback bugs:
profile + copy codec conflict, stream order mapping, and segment
seeking for audio-first containers.
The reference analyst skill went straight from capability audit to
creative proposals with no research. This caused the agent to propose
concepts based solely on the reference analysis and its own knowledge,
missing content landscape context, technique best practices, and
subject-matter depth.
Added Step 3b between critical questions and creative proposals:
- Content landscape scan (3-5 similar existing videos)
- Style/technique research (AI model strengths, prompting patterns)
- Subject-matter research (facts, tropes, hooks)
- 2-3 minute time budget — lightweight, not full research-director
Four gaps found during user acceptance testing of the reference-video
production workflow:
- video_selector: expose aspect_ratio, duration, reference_image_path,
reference_image_url, and image_url in schema so agents can discover
these critical params. Auto-upload local images when the selected
provider requires a URL.
- _shared.py: add upload_image_fal() for local→URL image bridging via
fal.ai storage. Fix upload_image_heygen() to try v2 presigned upload
before falling back to fal.ai (old /v1/asset endpoint returns 404).
- kling_video: call probe_output() so response includes output_path,
duration_seconds, file_size_mb, video dimensions, and codec info.
The AGENT_GUIDE establishes Remotion as the preferred composition engine
over FFmpeg, but the reference analyst skill was presenting them as peer
options. This caused the agent to default to FFmpeg during proposals.
- Capability audit template now labels Remotion as "preferred" and FFmpeg
as "fallback only"
- Added explicit composition engine priority note
- Proposal template separates Composition and Motion as distinct lines
Video analyzer now uses Farneback dense optical flow to classify each
scene as motion_clip, animated_still, or static_image. This lets the
agent correctly identify whether a reference video uses AI-generated
video clips vs still images with pan/zoom — and plan the right pipeline.
Changes:
- video_analyzer.py: new Step 3b with _classify_scene_motion() and
_read_frame_at() helpers; updated _needs_motion() to use per-scene
motion data instead of pacing heuristic alone
- video-reference-analyst.md: added Motion line to summary template
and instructions to read motion_type field before proposing tools
Entry point files (CLAUDE.md, AGENTS.md, .windsurfrules, .cursorrules,
copilot-instructions.md) previously contained useful snippets that let
agents skip reading the full AGENT_GUIDE. Now they contain zero usable
instructions — only a mandatory redirect — so any assistant is forced
to read AGENT_GUIDE.md before acting on user requests.
Implements the 2026-04-02 transformation spec (Phases 1-8) and fixes all
critical bugs found during 5-pipeline E2E testing.
Governance & Decision Intelligence:
- Pipeline-specific stage order in checkpoint (replaces global STAGES list)
- Provider scoring engine (lib/scoring.py) with 7-dimension weighted ranking
- Decision log artifact enforced at proposal/idea stage across all 10 pipelines
- Delivery promise classifier prevents silent motion-to-still downgrades
- Structured shot language in scene_plan schema (camera, lens, lighting, DOF)
- Variation checker and slideshow risk scorer block samey output before render
- Creative intake, capability extension, and creative-intake meta skills
- Final self-review artifact with 5 mandatory checks before presenting output
- Source media review contract for user-supplied footage
Render & Theme System:
- Remotion AnimatedBackground now derives colors from playbook (no more hardcoded
dark blue fintech gradient on every video)
- video_compose builds custom ThemeConfig from playbook YAML colors/fonts —
custom playbooks flow through to Remotion automatically
- Explainer component wires theme to all child components (charts, cards, etc.)
- resolveAsset() handles absolute paths on Windows/Unix via file:// URIs
- RENDERER_FAMILY_MAP synced with actual Remotion compositions
Critical Bug Fixes:
- Windows npx subprocess: run_command() resolves .cmd wrappers via shutil.which()
- Silent renderer downgrade: Remotion failure now returns explicit error with
options instead of silently falling back to FFmpeg
- .env inline comment parsing strips trailing # comments from API keys
- concat_path UnboundLocalError in video_compose finally block
- audio_mixer and showcase_card capture=True kwarg bug
- Selector estimate_cost() calls fixed (_select_tool -> _select_best_tool)
- asset_manifest schema expanded with provider, license, subtype fields
- screen-demo subtitle_gen moved from required to optional tools
- Duration drift detection in post-render final review (>25% warns)
Add green_screen_processor (auto-detect + rembg fallback) and
green_screen_composite (4 layout presets with alpha compositing) tools
to automate the full keying-to-composite pipeline.
Remotion: add AnimatedBackground with gradient mesh and floating orbs
to Explainer, fix caption burn tool (remove entry point arg, auto-detect
dimensions, extend TalkingHead duration to 300s).
Update scene-director, compose-director, and asset-director skill docs
with green screen workflow steps and component constraints.
Scene-director rewrite: agent now watches footage, understands content, and proposes
creative overlays (charts, stats, key terms, comparisons) before building anything.
Presents enhancement plan to user for approval before proceeding.
Compose-director fixes from Round 1 verification:
- eye_enhance: now explicitly required, not silently skippable
- Caption positioning: explicit MarginV=160 for 9:16, never center
- Final encode: mandatory with target file size table
- ASR corrections: new Step 2b to scan transcript and build corrections dict
- Overlay compositing: new Step 3b for burning approved graphics onto footage
Asset-director rewrite: generates Remotion overlay assets (callouts, stat cards,
charts, comparisons) from scene plan. Includes overlay type → Remotion cut mapping
table and dark theme requirements.
Bug fixes found during subagent verification:
- remotion_caption_burn.py: fix run_command API, add npx.cmd for Windows
- visual_qa.py: fix run_command API (3 places), Windows /dev/null → NUL
New tools: face_tracker, visual_qa, eye_enhance, auto_reframe, remotion_caption_burn, showcase_card, silence_cutter. Updated audio_mixer with segmented_music operation and subtitle_gen with ASR corrections. Registered TalkingHead composition in Root.tsx. Rewrote compose/edit/scene director skills for full enhancement chain, Remotion captions, multi-clip assembly, and visual QA. Gitignore cleanup: exclude test demo-props, downloaded music, and generated images.
Add anime_scene rendering engine (AnimeScene + ParticleOverlay components) with multi-image
crossfade, 9 camera motion types, 5 particle systems, and cinematic lighting overlays.
Fix critical Remotion durationInFrames footgun by passing sceneDurationSeconds from parent.
Add audio offset/loop support in Explainer for skipping quiet music intros.
New tools: audio_energy.py analyzes per-second loudness via ebur128 to find optimal music
offset and detect when looping is needed.
Update all 6 animation pipeline skills (proposal, scene, asset, compose, executive-producer,
remotion.md) with battle-tested image_animation workflow including tool availability scan,
FLUX multi-image generation, composition JSON format, pre-render validation, and post-render
self-review.
Add 3 demo compositions (Candyland, Mori no Seishin, Deep Ocean) and anime-ghibli style
playbook. Update README with 3 anime video showcases and animation prompts. Add Animation
Pipeline section to PROMPT_GALLERY.md.
Replace bash-only commands in Makefile with Python equivalents so
setup, test, lint, and clean work on Windows. Add LUMINA video URL
to README embed and note upcoming Ollama/LM Studio local LLM support.
