mirror of
https://github.com/calesthio/OpenMontage.git
synced 2026-08-05 15:20:40 +08:00
Initial release — OpenMontage: the first open-source agentic video production system
11 production pipelines, 47 tools, 124 agent skills. Supports cloud APIs (fal.ai, OpenAI, ElevenLabs, Suno, HeyGen, Runway) and free local providers (diffusers, Piper TTS, WAN 2.1, Hunyuan, CogVideo). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
calesthio
72
tests/qa/QA_PLAN.md
Normal file
72
tests/qa/QA_PLAN.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# Quality Validation Plan — Phase 3.5 + G3.11
|
||||
|
||||
## Purpose
|
||||
|
||||
Run every tool with real API keys, inspect outputs (see images, listen to audio, watch video), find gaps, fix them. This is the gate before calling Phase 3.5 "Verified."
|
||||
|
||||
## Test Scripts (all ready to run)
|
||||
|
||||
| Script | Tools Tested | API Keys Used | Est. Cost |
|
||||
|--------|-------------|---------------|-----------|
|
||||
| `test_01_tts.py` | `elevenlabs_tts` (ElevenLabs) | ELEVENLABS_API_KEY | ~$0.02 |
|
||||
| `test_02_image_gen.py` | `image_gen` (DALL-E 3 + FLUX) | OPENAI_API_KEY, FAL_AI_API_KEY | ~$0.15 |
|
||||
| `test_03_music.py` | `music_gen` (ElevenLabs) | ELEVENLABS_API_KEY | ~$0.10 |
|
||||
| `test_04_audio_mix.py` | `audio_mixer` | None (ffmpeg only) | $0 |
|
||||
| `test_05_video_compose.py` | `video_compose` | None (ffmpeg only) | $0 |
|
||||
| `test_06_video_stitch.py` | `video_stitch` | None (ffmpeg only) | $0 |
|
||||
| `test_07_playbook_intelligence.py` | `playbook_loader.py` functions | None (pure Python) | $0 |
|
||||
| `test_08_end_to_end.py` | Full animated-explainer pipeline | None (ffmpeg fixtures) | $0 |
|
||||
|
||||
## Inspection Protocol
|
||||
|
||||
For each output:
|
||||
1. **Audio files**: Use `ffprobe` for format/duration/channels, then LISTEN (play in media player or use Whisper to verify content matches prompt)
|
||||
2. **Image files**: Use `ffprobe` for dimensions, then VIEW (open image, check composition, text readability, style match)
|
||||
3. **Video files**: Use `ffprobe` for resolution/fps/duration/codec, then WATCH (check A/V sync, transitions, subtitle timing)
|
||||
4. **Design intelligence**: Run against all 3 playbooks, verify contrast ratios match manual calculation, check CVD warnings are accurate
|
||||
|
||||
## Known Risk Areas
|
||||
|
||||
| Area | Risk | How to Validate |
|
||||
|------|------|-----------------|
|
||||
| TTS voice selection | Default voice may not match playbook mood | Test with multiple voice IDs, compare against playbook `voice_style` |
|
||||
| Image gen consistency | DALL-E/FLUX outputs vary wildly per prompt | Test with playbook `image_prompt_prefix` prepended |
|
||||
| Music duration alignment | Music may not match narration duration | Compare `music.duration` vs `tts.duration`, check padding/looping |
|
||||
| Audio ducking timing | Ducking may cut music too aggressively | Inspect waveform: music should duck ~6dB under speech, recover smoothly |
|
||||
| Video stitch transitions | Crossfade may flicker with mismatched codecs | Test with both matching and mismatched clips, check `auto_normalize` |
|
||||
| Subtitle burn-in | Font size/position may clip on mobile formats | Test with 9:16 (TikTok) and 16:9 (YouTube) profiles |
|
||||
| Remotion render | Components may fail with real data | Build a test composition with all 8 components, render at 1080p |
|
||||
| Playbook contrast | Edge cases in dark-on-dark or light-on-light themes | Test with all 3 playbooks + a deliberately low-contrast custom one |
|
||||
|
||||
## Run Order
|
||||
|
||||
```bash
|
||||
cd C:/Users/ishan/Documents/OpenMontage
|
||||
|
||||
# Phase 1: Individual tools (can run in parallel)
|
||||
python tests/qa/test_01_tts.py
|
||||
python tests/qa/test_02_image_gen.py
|
||||
python tests/qa/test_03_music.py
|
||||
|
||||
# Phase 2: Composition (depends on Phase 1 outputs)
|
||||
python tests/qa/test_04_audio_mix.py
|
||||
python tests/qa/test_05_video_compose.py
|
||||
python tests/qa/test_06_video_stitch.py
|
||||
|
||||
# Phase 3: Intelligence validation (no API calls)
|
||||
python tests/qa/test_07_playbook_intelligence.py
|
||||
|
||||
# Phase 4: Full pipeline
|
||||
python tests/qa/test_08_end_to_end.py
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] All 3 TTS samples: clear speech, correct content, no artifacts, ≥44.1kHz
|
||||
- [ ] All 4 images: match prompt intent, correct dimensions, no watermarks, good composition
|
||||
- [ ] Both music tracks: match mood prompt, correct duration (±2s), no abrupt cuts
|
||||
- [ ] Audio mix: speech clearly above music, ducking smooth, no clipping
|
||||
- [ ] Video compose: A/V sync within 50ms, correct resolution, playable in VLC
|
||||
- [ ] Video stitch: smooth transitions, no frame drops, PIP correctly positioned
|
||||
- [ ] Playbook intelligence: all 3 playbooks pass a11y, contrast ratios within 0.1 of manual calc
|
||||
- [ ] End-to-end: 60-second explainer renders without errors, all stages checkpoint correctly
|
||||
80
tests/qa/test_01_tts.py
Normal file
80
tests/qa/test_01_tts.py
Normal file
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
"""QA Test 01: TTS voice generation via ElevenLabs."""
|
||||
|
||||
import sys, os, json, time
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
|
||||
from lib.env_loader import load_env
|
||||
load_env()
|
||||
|
||||
from tools.audio.elevenlabs_tts import ElevenLabsTTS
|
||||
|
||||
OUT = os.path.join(os.path.dirname(__file__), "output")
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
tool = ElevenLabsTTS()
|
||||
print(f"Tool status: {tool.get_status()}")
|
||||
|
||||
# Test 1: Short narration
|
||||
print("\n--- Test 1: Short narration ---")
|
||||
r1 = tool.execute({
|
||||
"text": "Welcome to OpenMontage. Let's build something amazing together.",
|
||||
"output_path": os.path.join(OUT, "tts_short.mp3"),
|
||||
})
|
||||
print(f"Success: {r1.success}, Cost: ${r1.cost_usd:.4f}, Duration: {r1.duration_seconds:.2f}s")
|
||||
if r1.error: print(f"Error: {r1.error}")
|
||||
if r1.artifacts: print(f"Artifacts: {r1.artifacts}")
|
||||
|
||||
# Test 2: Longer paragraph with technical content
|
||||
print("\n--- Test 2: Technical narration ---")
|
||||
r2 = tool.execute({
|
||||
"text": (
|
||||
"Quantum computing leverages quantum mechanical phenomena like superposition and entanglement "
|
||||
"to process information in fundamentally different ways than classical computers. "
|
||||
"While a classical bit can be either zero or one, a quantum bit, or qubit, can exist in "
|
||||
"a superposition of both states simultaneously. This allows quantum computers to explore "
|
||||
"many possible solutions at once, making them exceptionally powerful for certain types of problems."
|
||||
),
|
||||
"output_path": os.path.join(OUT, "tts_technical.mp3"),
|
||||
})
|
||||
print(f"Success: {r2.success}, Cost: ${r2.cost_usd:.4f}, Duration: {r2.duration_seconds:.2f}s")
|
||||
if r2.error: print(f"Error: {r2.error}")
|
||||
if r2.artifacts: print(f"Artifacts: {r2.artifacts}")
|
||||
|
||||
# Test 3: Emotional / storytelling narration
|
||||
print("\n--- Test 3: Storytelling narration ---")
|
||||
r3 = tool.execute({
|
||||
"text": (
|
||||
"Picture this. You wake up one morning, check your phone, and discover that overnight, "
|
||||
"your side project went viral. Thousands of people are using it. Messages are flooding in. "
|
||||
"This isn't a dream. This is what happens when you build something people actually need."
|
||||
),
|
||||
"output_path": os.path.join(OUT, "tts_story.mp3"),
|
||||
})
|
||||
print(f"Success: {r3.success}, Cost: ${r3.cost_usd:.4f}, Duration: {r3.duration_seconds:.2f}s")
|
||||
if r3.error: print(f"Error: {r3.error}")
|
||||
if r3.artifacts: print(f"Artifacts: {r3.artifacts}")
|
||||
|
||||
# Probe outputs with ffprobe
|
||||
import subprocess
|
||||
for name in ["tts_short.mp3", "tts_technical.mp3", "tts_story.mp3"]:
|
||||
path = os.path.join(OUT, name)
|
||||
if os.path.exists(path):
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "quiet", "-print_format", "json", "-show_format", "-show_streams", path],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
info = json.loads(probe.stdout)
|
||||
fmt = info.get("format", {})
|
||||
streams = info.get("streams", [{}])
|
||||
audio = streams[0] if streams else {}
|
||||
print(f"\n[{name}] Duration: {fmt.get('duration', '?')}s, "
|
||||
f"Sample rate: {audio.get('sample_rate', '?')}Hz, "
|
||||
f"Channels: {audio.get('channels', '?')}, "
|
||||
f"Codec: {audio.get('codec_name', '?')}, "
|
||||
f"Size: {os.path.getsize(path)} bytes")
|
||||
else:
|
||||
print(f"\n[{name}] FILE NOT FOUND")
|
||||
|
||||
print("\n=== TTS TEST COMPLETE ===")
|
||||
88
tests/qa/test_02_image_gen.py
Normal file
88
tests/qa/test_02_image_gen.py
Normal file
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env python3
|
||||
"""QA Test 02: Image generation via OpenAI (DALL-E 3) and fal.ai (FLUX)."""
|
||||
|
||||
import sys, os, json, time
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
|
||||
from lib.env_loader import load_env
|
||||
load_env()
|
||||
|
||||
from tools.graphics.image_gen import ImageGen
|
||||
|
||||
OUT = os.path.join(os.path.dirname(__file__), "output")
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
tool = ImageGen()
|
||||
print(f"Tool status: {tool.get_status()}")
|
||||
|
||||
# Test 1: DALL-E — professional diagram
|
||||
print("\n--- Test 1: DALL-E professional diagram ---")
|
||||
r1 = tool.execute({
|
||||
"prompt": "A clean, professional infographic showing the 5 stages of a video production pipeline: Idea, Script, Assets, Edit, Publish. Flat design, blue and amber color scheme, white background, no text.",
|
||||
"width": 1280,
|
||||
"height": 720,
|
||||
"provider": "openai",
|
||||
"output_path": os.path.join(OUT, "img_dalle_diagram.png"),
|
||||
})
|
||||
print(f"Success: {r1.success}, Cost: ${r1.cost_usd:.4f}")
|
||||
if r1.error: print(f"Error: {r1.error}")
|
||||
if r1.artifacts: print(f"Artifacts: {r1.artifacts}")
|
||||
|
||||
# Test 2: DALL-E — cinematic scene
|
||||
print("\n--- Test 2: DALL-E cinematic scene ---")
|
||||
r2 = tool.execute({
|
||||
"prompt": "A futuristic control room with holographic displays showing data visualizations, cinematic lighting, wide angle, film grain, warm tones",
|
||||
"width": 1280,
|
||||
"height": 720,
|
||||
"provider": "openai",
|
||||
"output_path": os.path.join(OUT, "img_dalle_cinematic.png"),
|
||||
})
|
||||
print(f"Success: {r2.success}, Cost: ${r2.cost_usd:.4f}")
|
||||
if r2.error: print(f"Error: {r2.error}")
|
||||
if r2.artifacts: print(f"Artifacts: {r2.artifacts}")
|
||||
|
||||
# Test 3: FLUX via fal.ai — abstract tech
|
||||
print("\n--- Test 3: FLUX abstract tech ---")
|
||||
r3 = tool.execute({
|
||||
"prompt": "Abstract visualization of neural network connections, glowing nodes and edges, dark background, neon blue and purple, high detail, 8k render",
|
||||
"width": 1280,
|
||||
"height": 720,
|
||||
"provider": "flux",
|
||||
"output_path": os.path.join(OUT, "img_flux_abstract.png"),
|
||||
})
|
||||
print(f"Success: {r3.success}, Cost: ${r3.cost_usd:.4f}")
|
||||
if r3.error: print(f"Error: {r3.error}")
|
||||
if r3.artifacts: print(f"Artifacts: {r3.artifacts}")
|
||||
|
||||
# Test 4: FLUX — character/mascot
|
||||
print("\n--- Test 4: FLUX character illustration ---")
|
||||
r4 = tool.execute({
|
||||
"prompt": "Friendly robot mascot character, simple geometric design, holding a film clapboard, isometric view, clean white background, flat illustration style",
|
||||
"width": 1024,
|
||||
"height": 1024,
|
||||
"provider": "flux",
|
||||
"output_path": os.path.join(OUT, "img_flux_mascot.png"),
|
||||
})
|
||||
print(f"Success: {r4.success}, Cost: ${r4.cost_usd:.4f}")
|
||||
if r4.error: print(f"Error: {r4.error}")
|
||||
if r4.artifacts: print(f"Artifacts: {r4.artifacts}")
|
||||
|
||||
# Probe outputs
|
||||
import subprocess
|
||||
for name in ["img_dalle_diagram.png", "img_dalle_cinematic.png", "img_flux_abstract.png", "img_flux_mascot.png"]:
|
||||
path = os.path.join(OUT, name)
|
||||
if os.path.exists(path):
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "quiet", "-print_format", "json", "-show_streams", path],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
info = json.loads(probe.stdout)
|
||||
stream = info.get("streams", [{}])[0]
|
||||
print(f"\n[{name}] {stream.get('width', '?')}x{stream.get('height', '?')}, "
|
||||
f"Format: {stream.get('codec_name', '?')}, "
|
||||
f"Size: {os.path.getsize(path)} bytes")
|
||||
else:
|
||||
print(f"\n[{name}] FILE NOT FOUND")
|
||||
|
||||
print("\n=== IMAGE GEN TEST COMPLETE ===")
|
||||
62
tests/qa/test_03_music.py
Normal file
62
tests/qa/test_03_music.py
Normal file
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/env python3
|
||||
"""QA Test 03: Music generation via ElevenLabs."""
|
||||
|
||||
import sys, os, json
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
|
||||
from lib.env_loader import load_env
|
||||
load_env()
|
||||
|
||||
from tools.audio.music_gen import MusicGen
|
||||
|
||||
OUT = os.path.join(os.path.dirname(__file__), "output")
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
tool = MusicGen()
|
||||
print(f"Tool status: {tool.get_status()}")
|
||||
|
||||
# Test 1: Upbeat tech background
|
||||
print("\n--- Test 1: Upbeat tech background ---")
|
||||
r1 = tool.execute({
|
||||
"prompt": "Upbeat electronic background music, 120 BPM, energetic but not overwhelming, suitable for a tech explainer video",
|
||||
"duration_seconds": 30,
|
||||
"output_path": os.path.join(OUT, "music_upbeat.mp3"),
|
||||
})
|
||||
print(f"Success: {r1.success}, Cost: ${r1.cost_usd:.4f}")
|
||||
if r1.error: print(f"Error: {r1.error}")
|
||||
if r1.artifacts: print(f"Artifacts: {r1.artifacts}")
|
||||
|
||||
# Test 2: Calm ambient
|
||||
print("\n--- Test 2: Calm ambient ---")
|
||||
r2 = tool.execute({
|
||||
"prompt": "Calm ambient background music, soft piano and strings, 80 BPM, reflective mood, suitable for documentary narration",
|
||||
"duration_seconds": 30,
|
||||
"output_path": os.path.join(OUT, "music_calm.mp3"),
|
||||
})
|
||||
print(f"Success: {r2.success}, Cost: ${r2.cost_usd:.4f}")
|
||||
if r2.error: print(f"Error: {r2.error}")
|
||||
if r2.artifacts: print(f"Artifacts: {r2.artifacts}")
|
||||
|
||||
# Probe outputs
|
||||
import subprocess
|
||||
for name in ["music_upbeat.mp3", "music_calm.mp3"]:
|
||||
path = os.path.join(OUT, name)
|
||||
if os.path.exists(path):
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "quiet", "-print_format", "json", "-show_format", "-show_streams", path],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
info = json.loads(probe.stdout)
|
||||
fmt = info.get("format", {})
|
||||
streams = info.get("streams", [{}])
|
||||
audio = streams[0] if streams else {}
|
||||
print(f"\n[{name}] Duration: {fmt.get('duration', '?')}s, "
|
||||
f"Sample rate: {audio.get('sample_rate', '?')}Hz, "
|
||||
f"Channels: {audio.get('channels', '?')}, "
|
||||
f"Codec: {audio.get('codec_name', '?')}, "
|
||||
f"Size: {os.path.getsize(path)} bytes")
|
||||
else:
|
||||
print(f"\n[{name}] FILE NOT FOUND")
|
||||
|
||||
print("\n=== MUSIC GEN TEST COMPLETE ===")
|
||||
153
tests/qa/test_04_audio_mix.py
Normal file
153
tests/qa/test_04_audio_mix.py
Normal file
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""QA Test 04: Audio mixing — mix TTS + music with ducking, verify levels.
|
||||
|
||||
Depends on test_01 and test_03 outputs (TTS + music files).
|
||||
If those don't exist, generates minimal test fixtures via ffmpeg.
|
||||
"""
|
||||
|
||||
import sys, os, json, subprocess
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
|
||||
from lib.env_loader import load_env
|
||||
load_env()
|
||||
|
||||
from tools.audio.audio_mixer import AudioMixer
|
||||
|
||||
OUT = os.path.join(os.path.dirname(__file__), "output")
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
# --- Fixture generation (if test_01/test_03 outputs don't exist) ---
|
||||
|
||||
SPEECH_FILE = os.path.join(OUT, "tts_short.mp3")
|
||||
MUSIC_FILE = os.path.join(OUT, "music_calm.mp3")
|
||||
|
||||
def generate_fixture(path, description, duration=5):
|
||||
"""Generate a minimal audio fixture with ffmpeg if the file doesn't exist."""
|
||||
if os.path.exists(path):
|
||||
print(f" [fixture] Using existing: {path}")
|
||||
return
|
||||
print(f" [fixture] Generating {description}: {path}")
|
||||
# Sine wave for speech stand-in, pink noise for music stand-in
|
||||
if "speech" in description or "tts" in description:
|
||||
src = f"sine=frequency=440:duration={duration}"
|
||||
else:
|
||||
src = f"anoisesrc=d={duration}:c=pink"
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-f", "lavfi", "-i", src, "-ar", "44100", "-ac", "1", path],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
generate_fixture(SPEECH_FILE, "speech/tts fixture", duration=8)
|
||||
generate_fixture(MUSIC_FILE, "music fixture", duration=15)
|
||||
|
||||
# --- Tool setup ---
|
||||
|
||||
tool = AudioMixer()
|
||||
print(f"Tool status: {tool.get_status()}")
|
||||
|
||||
# --- Test 1: Basic mix (speech + music, no ducking) ---
|
||||
print("\n--- Test 1: Basic mix (speech + music) ---")
|
||||
r1 = tool.execute({
|
||||
"operation": "mix",
|
||||
"tracks": [
|
||||
{"path": SPEECH_FILE, "role": "speech", "volume": 1.0},
|
||||
{"path": MUSIC_FILE, "role": "music", "volume": 0.3},
|
||||
],
|
||||
"normalize": True,
|
||||
"output_path": os.path.join(OUT, "mix_basic.wav"),
|
||||
})
|
||||
print(f"Success: {r1.success}, Duration: {r1.duration_seconds:.2f}s")
|
||||
if r1.error: print(f"Error: {r1.error}")
|
||||
if r1.artifacts: print(f"Artifacts: {r1.artifacts}")
|
||||
|
||||
# --- Test 2: Mix with fades ---
|
||||
print("\n--- Test 2: Mix with fades ---")
|
||||
r2 = tool.execute({
|
||||
"operation": "mix",
|
||||
"tracks": [
|
||||
{"path": SPEECH_FILE, "role": "speech", "volume": 1.0, "fade_in_seconds": 0.5},
|
||||
{"path": MUSIC_FILE, "role": "music", "volume": 0.25, "fade_in_seconds": 1.0, "fade_out_seconds": 2.0},
|
||||
],
|
||||
"normalize": True,
|
||||
"output_path": os.path.join(OUT, "mix_fades.wav"),
|
||||
})
|
||||
print(f"Success: {r2.success}, Duration: {r2.duration_seconds:.2f}s")
|
||||
if r2.error: print(f"Error: {r2.error}")
|
||||
if r2.artifacts: print(f"Artifacts: {r2.artifacts}")
|
||||
|
||||
# --- Test 3: Ducking (sidechain compress music under speech) ---
|
||||
print("\n--- Test 3: Ducking ---")
|
||||
r3 = tool.execute({
|
||||
"operation": "duck",
|
||||
"tracks": [
|
||||
{"path": SPEECH_FILE, "role": "speech"},
|
||||
{"path": MUSIC_FILE, "role": "music"},
|
||||
],
|
||||
"ducking": {
|
||||
"enabled": True,
|
||||
"music_volume_during_speech": 0.15,
|
||||
"attack_ms": 200,
|
||||
"release_ms": 500,
|
||||
},
|
||||
"output_path": os.path.join(OUT, "mix_ducked.wav"),
|
||||
})
|
||||
print(f"Success: {r3.success}, Duration: {r3.duration_seconds:.2f}s")
|
||||
if r3.error: print(f"Error: {r3.error}")
|
||||
if r3.artifacts: print(f"Artifacts: {r3.artifacts}")
|
||||
|
||||
# --- Test 4: Mix with delayed music start ---
|
||||
print("\n--- Test 4: Delayed music start ---")
|
||||
r4 = tool.execute({
|
||||
"operation": "mix",
|
||||
"tracks": [
|
||||
{"path": SPEECH_FILE, "role": "speech", "volume": 1.0},
|
||||
{"path": MUSIC_FILE, "role": "music", "volume": 0.2, "start_seconds": 3.0},
|
||||
],
|
||||
"normalize": False,
|
||||
"output_path": os.path.join(OUT, "mix_delayed.wav"),
|
||||
})
|
||||
print(f"Success: {r4.success}, Duration: {r4.duration_seconds:.2f}s")
|
||||
if r4.error: print(f"Error: {r4.error}")
|
||||
if r4.artifacts: print(f"Artifacts: {r4.artifacts}")
|
||||
|
||||
# --- Probe all outputs ---
|
||||
print("\n--- Output inspection ---")
|
||||
for name in ["mix_basic.wav", "mix_fades.wav", "mix_ducked.wav", "mix_delayed.wav"]:
|
||||
path = os.path.join(OUT, name)
|
||||
if os.path.exists(path):
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "quiet", "-print_format", "json", "-show_format", "-show_streams", path],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
info = json.loads(probe.stdout)
|
||||
fmt = info.get("format", {})
|
||||
streams = info.get("streams", [{}])
|
||||
audio = streams[0] if streams else {}
|
||||
print(f"\n[{name}] Duration: {fmt.get('duration', '?')}s, "
|
||||
f"Sample rate: {audio.get('sample_rate', '?')}Hz, "
|
||||
f"Channels: {audio.get('channels', '?')}, "
|
||||
f"Codec: {audio.get('codec_name', '?')}, "
|
||||
f"Size: {os.path.getsize(path)} bytes")
|
||||
|
||||
# Check for clipping via loudnorm stats
|
||||
loud = subprocess.run(
|
||||
["ffmpeg", "-i", path, "-af", "loudnorm=print_format=json", "-f", "null", "-"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
# Parse loudnorm JSON from stderr (ffmpeg writes it there)
|
||||
stderr = loud.stderr
|
||||
json_start = stderr.rfind("{")
|
||||
json_end = stderr.rfind("}") + 1
|
||||
if json_start >= 0 and json_end > json_start:
|
||||
try:
|
||||
loudness = json.loads(stderr[json_start:json_end])
|
||||
print(f" Loudness: I={loudness.get('input_i', '?')} LUFS, "
|
||||
f"TP={loudness.get('input_tp', '?')} dBTP, "
|
||||
f"LRA={loudness.get('input_lra', '?')} LU")
|
||||
except json.JSONDecodeError:
|
||||
print(" (Could not parse loudness stats)")
|
||||
else:
|
||||
print(f"\n[{name}] FILE NOT FOUND")
|
||||
|
||||
print("\n=== AUDIO MIX TEST COMPLETE ===")
|
||||
230
tests/qa/test_05_video_compose.py
Normal file
230
tests/qa/test_05_video_compose.py
Normal file
@@ -0,0 +1,230 @@
|
||||
#!/usr/bin/env python3
|
||||
"""QA Test 05: Video composition — image + mixed audio → video, verify A/V sync.
|
||||
|
||||
Creates a video from static images with audio and optional subtitles.
|
||||
Uses ffmpeg-generated fixtures if prior test outputs don't exist.
|
||||
"""
|
||||
|
||||
import sys, os, json, subprocess
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
|
||||
from lib.env_loader import load_env
|
||||
load_env()
|
||||
|
||||
from tools.video.video_compose import VideoCompose
|
||||
|
||||
OUT = os.path.join(os.path.dirname(__file__), "output")
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
# --- Fixture generation ---
|
||||
|
||||
def ensure_image(path, width=1280, height=720, color="blue"):
|
||||
"""Generate a test image with ffmpeg if it doesn't exist."""
|
||||
if os.path.exists(path):
|
||||
print(f" [fixture] Using existing: {path}")
|
||||
return
|
||||
print(f" [fixture] Generating {color} image: {path}")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-f", "lavfi", "-i",
|
||||
f"color=c={color}:s={width}x{height}:d=1",
|
||||
"-frames:v", "1", path],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
def ensure_video(path, duration=5, width=1280, height=720, color="blue"):
|
||||
"""Generate a test video clip with ffmpeg if it doesn't exist.
|
||||
|
||||
Uses forced keyframes every 1s so copy-mode trimming retains the video stream.
|
||||
"""
|
||||
if os.path.exists(path):
|
||||
print(f" [fixture] Using existing: {path}")
|
||||
return
|
||||
print(f" [fixture] Generating {duration}s {color} video: {path}")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-f", "lavfi", "-i",
|
||||
f"color=c={color}:s={width}x{height}:d={duration}:r=30",
|
||||
"-f", "lavfi", "-i", f"sine=frequency=440:duration={duration}",
|
||||
"-c:v", "libx264", "-crf", "23", "-pix_fmt", "yuv420p",
|
||||
"-g", "30", "-keyint_min", "30",
|
||||
"-c:a", "aac", "-shortest", path],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
def ensure_audio(path, duration=5):
|
||||
"""Generate a test audio file if it doesn't exist."""
|
||||
if os.path.exists(path):
|
||||
print(f" [fixture] Using existing: {path}")
|
||||
return
|
||||
print(f" [fixture] Generating {duration}s audio: {path}")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-f", "lavfi", "-i",
|
||||
f"sine=frequency=440:duration={duration}",
|
||||
"-ar", "44100", "-ac", "2", path],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
def ensure_subtitle(path):
|
||||
"""Write a minimal SRT subtitle file."""
|
||||
if os.path.exists(path):
|
||||
print(f" [fixture] Using existing: {path}")
|
||||
return
|
||||
print(f" [fixture] Generating subtitle: {path}")
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
f.write("1\n00:00:00,000 --> 00:00:03,000\nWelcome to OpenMontage\n\n")
|
||||
f.write("2\n00:00:03,000 --> 00:00:06,000\nBuilding amazing videos with AI\n\n")
|
||||
f.write("3\n00:00:06,000 --> 00:00:10,000\nLet's see what we can create\n\n")
|
||||
|
||||
# Create fixtures
|
||||
CLIP_A = os.path.join(OUT, "compose_clip_a.mp4")
|
||||
CLIP_B = os.path.join(OUT, "compose_clip_b.mp4")
|
||||
AUDIO_MIX = os.path.join(OUT, "compose_audio.wav")
|
||||
SUBTITLE = os.path.join(OUT, "compose_subs.srt")
|
||||
|
||||
# Use 10s clips so -c copy has keyframe headroom (5s clips lose video stream)
|
||||
ensure_video(CLIP_A, duration=10, color="darkblue")
|
||||
ensure_video(CLIP_B, duration=10, color="darkgreen")
|
||||
ensure_audio(AUDIO_MIX, duration=10)
|
||||
ensure_subtitle(SUBTITLE)
|
||||
|
||||
# --- Tool setup ---
|
||||
|
||||
tool = VideoCompose()
|
||||
print(f"Tool status: {tool.get_status()}")
|
||||
|
||||
# --- Test 1: Compose with cuts + audio ---
|
||||
print("\n--- Test 1: Compose from edit_decisions ---")
|
||||
r1 = tool.execute({
|
||||
"operation": "compose",
|
||||
"edit_decisions": {
|
||||
"cuts": [
|
||||
{"source": CLIP_A, "in_seconds": 0, "out_seconds": 5},
|
||||
{"source": CLIP_B, "in_seconds": 0, "out_seconds": 5},
|
||||
],
|
||||
},
|
||||
"audio_path": AUDIO_MIX,
|
||||
"output_path": os.path.join(OUT, "compose_basic.mp4"),
|
||||
})
|
||||
print(f"Success: {r1.success}, Duration: {r1.duration_seconds:.2f}s")
|
||||
if r1.error: print(f"Error: {r1.error}")
|
||||
if r1.artifacts: print(f"Artifacts: {r1.artifacts}")
|
||||
|
||||
# --- Test 2: Compose with subtitles ---
|
||||
print("\n--- Test 2: Compose with subtitles ---")
|
||||
r2 = tool.execute({
|
||||
"operation": "compose",
|
||||
"edit_decisions": {
|
||||
"cuts": [
|
||||
{"source": CLIP_A, "in_seconds": 0, "out_seconds": 5},
|
||||
{"source": CLIP_B, "in_seconds": 0, "out_seconds": 5},
|
||||
],
|
||||
},
|
||||
"audio_path": AUDIO_MIX,
|
||||
"subtitle_path": SUBTITLE,
|
||||
"subtitle_style": {
|
||||
"font": "Arial",
|
||||
"font_size": 24,
|
||||
"primary_color": "&HFFFFFF",
|
||||
"outline_color": "&H000000",
|
||||
"outline_width": 2,
|
||||
"margin_v": 40,
|
||||
},
|
||||
"output_path": os.path.join(OUT, "compose_subtitled.mp4"),
|
||||
})
|
||||
print(f"Success: {r2.success}, Duration: {r2.duration_seconds:.2f}s")
|
||||
if r2.error: print(f"Error: {r2.error}")
|
||||
if r2.artifacts: print(f"Artifacts: {r2.artifacts}")
|
||||
|
||||
# --- Test 3: Burn subtitles onto existing video ---
|
||||
print("\n--- Test 3: Burn subtitles standalone ---")
|
||||
r3 = tool.execute({
|
||||
"operation": "burn_subtitles",
|
||||
"input_path": CLIP_A,
|
||||
"subtitle_path": SUBTITLE,
|
||||
"subtitle_style": {
|
||||
"font": "Arial",
|
||||
"font_size": 20,
|
||||
"bold": True,
|
||||
},
|
||||
"output_path": os.path.join(OUT, "compose_burn_subs.mp4"),
|
||||
})
|
||||
print(f"Success: {r3.success}, Duration: {r3.duration_seconds:.2f}s")
|
||||
if r3.error: print(f"Error: {r3.error}")
|
||||
if r3.artifacts: print(f"Artifacts: {r3.artifacts}")
|
||||
|
||||
# --- Test 4: Encode with media profile ---
|
||||
print("\n--- Test 4: Re-encode with profile ---")
|
||||
r4 = tool.execute({
|
||||
"operation": "encode",
|
||||
"input_path": CLIP_A,
|
||||
"profile": "YOUTUBE_LANDSCAPE",
|
||||
"crf": 20,
|
||||
"preset": "fast",
|
||||
"output_path": os.path.join(OUT, "compose_encoded.mp4"),
|
||||
})
|
||||
print(f"Success: {r4.success}, Duration: {r4.duration_seconds:.2f}s")
|
||||
if r4.error: print(f"Error: {r4.error}")
|
||||
if r4.artifacts: print(f"Artifacts: {r4.artifacts}")
|
||||
|
||||
# --- Test 5: Overlay ---
|
||||
print("\n--- Test 5: Overlay image on video ---")
|
||||
OVERLAY_IMG = os.path.join(OUT, "compose_overlay.png")
|
||||
ensure_image(OVERLAY_IMG, width=200, height=200, color="red")
|
||||
|
||||
r5 = tool.execute({
|
||||
"operation": "overlay",
|
||||
"input_path": CLIP_A,
|
||||
"overlays": [
|
||||
{
|
||||
"asset_path": OVERLAY_IMG,
|
||||
"x": 50, "y": 50,
|
||||
"width": 150, "height": 150,
|
||||
"start_seconds": 1,
|
||||
"end_seconds": 4,
|
||||
"opacity": 0.8,
|
||||
},
|
||||
],
|
||||
"output_path": os.path.join(OUT, "compose_overlay.mp4"),
|
||||
})
|
||||
print(f"Success: {r5.success}, Duration: {r5.duration_seconds:.2f}s")
|
||||
if r5.error: print(f"Error: {r5.error}")
|
||||
if r5.artifacts: print(f"Artifacts: {r5.artifacts}")
|
||||
|
||||
# --- Probe all outputs ---
|
||||
print("\n--- Output inspection ---")
|
||||
outputs = [
|
||||
"compose_basic.mp4",
|
||||
"compose_subtitled.mp4",
|
||||
"compose_burn_subs.mp4",
|
||||
"compose_encoded.mp4",
|
||||
"compose_overlay.mp4",
|
||||
]
|
||||
for name in outputs:
|
||||
path = os.path.join(OUT, name)
|
||||
if os.path.exists(path):
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "quiet", "-print_format", "json",
|
||||
"-show_format", "-show_streams", path],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
info = json.loads(probe.stdout)
|
||||
fmt = info.get("format", {})
|
||||
video = {}
|
||||
audio = {}
|
||||
for s in info.get("streams", []):
|
||||
if s.get("codec_type") == "video" and not video:
|
||||
video = s
|
||||
elif s.get("codec_type") == "audio" and not audio:
|
||||
audio = s
|
||||
print(f"\n[{name}]"
|
||||
f" Duration: {fmt.get('duration', '?')}s"
|
||||
f" | Video: {video.get('width', '?')}x{video.get('height', '?')}"
|
||||
f" {video.get('codec_name', '?')}@{video.get('r_frame_rate', '?')}fps"
|
||||
f" | Audio: {audio.get('codec_name', '?')}"
|
||||
f" {audio.get('sample_rate', '?')}Hz"
|
||||
f" {audio.get('channels', '?')}ch"
|
||||
f" | Size: {os.path.getsize(path)} bytes")
|
||||
else:
|
||||
print(f"\n[{name}] FILE NOT FOUND")
|
||||
|
||||
print("\n=== VIDEO COMPOSE TEST COMPLETE ===")
|
||||
225
tests/qa/test_06_video_stitch.py
Normal file
225
tests/qa/test_06_video_stitch.py
Normal file
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
"""QA Test 06: Video stitch — sequential concat, crossfade, fade, spatial PIP.
|
||||
|
||||
Tests the VideoStitch tool with both matching and mismatched clips.
|
||||
Generates fixtures via ffmpeg — no API keys needed.
|
||||
"""
|
||||
|
||||
import sys, os, json, subprocess
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
|
||||
from lib.env_loader import load_env
|
||||
load_env()
|
||||
|
||||
from tools.video.video_stitch import VideoStitch
|
||||
|
||||
OUT = os.path.join(os.path.dirname(__file__), "output")
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
# --- Fixture generation ---
|
||||
|
||||
def ensure_video(path, duration=4, width=1280, height=720, fps=30, color="blue"):
|
||||
"""Generate a test video clip with ffmpeg.
|
||||
|
||||
Uses forced keyframes every 1s so copy-mode operations retain the video stream.
|
||||
"""
|
||||
if os.path.exists(path):
|
||||
print(f" [fixture] Using existing: {path}")
|
||||
return
|
||||
print(f" [fixture] Generating {duration}s {width}x{height}@{fps}fps {color}: {path}")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y",
|
||||
"-f", "lavfi", "-i", f"color=c={color}:s={width}x{height}:d={duration}:r={fps}",
|
||||
"-f", "lavfi", "-i", f"sine=frequency=440:duration={duration}",
|
||||
"-c:v", "libx264", "-crf", "23", "-pix_fmt", "yuv420p",
|
||||
"-g", str(fps), "-keyint_min", str(fps),
|
||||
"-c:a", "aac", "-ar", "44100", "-ac", "2",
|
||||
"-shortest", path],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
# Matching clips (same resolution/fps/codec)
|
||||
CLIP_1 = os.path.join(OUT, "stitch_clip1.mp4")
|
||||
CLIP_2 = os.path.join(OUT, "stitch_clip2.mp4")
|
||||
CLIP_3 = os.path.join(OUT, "stitch_clip3.mp4")
|
||||
ensure_video(CLIP_1, duration=4, color="darkblue")
|
||||
ensure_video(CLIP_2, duration=4, color="darkgreen")
|
||||
ensure_video(CLIP_3, duration=4, color="darkred")
|
||||
|
||||
# Mismatched clip (different resolution + fps)
|
||||
CLIP_MISMATCH = os.path.join(OUT, "stitch_clip_mismatch.mp4")
|
||||
ensure_video(CLIP_MISMATCH, duration=4, width=640, height=480, fps=24, color="purple")
|
||||
|
||||
# --- Tool setup ---
|
||||
|
||||
tool = VideoStitch()
|
||||
print(f"Tool status: {tool.get_status()}")
|
||||
|
||||
# --- Test 1: Validate matching clips ---
|
||||
print("\n--- Test 1: Validate matching clips ---")
|
||||
r1 = tool.execute({
|
||||
"operation": "validate",
|
||||
"clips": [CLIP_1, CLIP_2, CLIP_3],
|
||||
})
|
||||
print(f"Success: {r1.success}")
|
||||
if r1.data:
|
||||
print(f" Compatible: {r1.data.get('compatible')}")
|
||||
print(f" Total duration: {r1.data.get('total_duration')}s")
|
||||
print(f" Mismatches: {len(r1.data.get('mismatches', []))}")
|
||||
if r1.error: print(f"Error: {r1.error}")
|
||||
|
||||
# --- Test 2: Validate mismatched clips ---
|
||||
print("\n--- Test 2: Validate mismatched clips ---")
|
||||
r2 = tool.execute({
|
||||
"operation": "validate",
|
||||
"clips": [CLIP_1, CLIP_MISMATCH],
|
||||
})
|
||||
print(f"Success: {r2.success}")
|
||||
if r2.data:
|
||||
print(f" Compatible: {r2.data.get('compatible')}")
|
||||
mismatches = r2.data.get("mismatches", [])
|
||||
for m in mismatches:
|
||||
print(f" Clip[{m['clip_index']}]: {', '.join(m['differences'])}")
|
||||
|
||||
# --- Test 3: Simple cut stitch (matching clips) ---
|
||||
print("\n--- Test 3: Cut stitch (2 clips) ---")
|
||||
r3 = tool.execute({
|
||||
"operation": "stitch",
|
||||
"clips": [CLIP_1, CLIP_2],
|
||||
"transition": "cut",
|
||||
"output_path": os.path.join(OUT, "stitch_cut.mp4"),
|
||||
})
|
||||
print(f"Success: {r3.success}, Duration: {r3.duration_seconds:.2f}s")
|
||||
if r3.data: print(f" Output duration: {r3.data.get('duration')}s, Method: {r3.data.get('method')}")
|
||||
if r3.error: print(f"Error: {r3.error}")
|
||||
|
||||
# --- Test 4: Crossfade stitch ---
|
||||
print("\n--- Test 4: Crossfade stitch (2 clips) ---")
|
||||
r4 = tool.execute({
|
||||
"operation": "stitch",
|
||||
"clips": [CLIP_1, CLIP_2],
|
||||
"transition": "crossfade",
|
||||
"transition_duration": 1.0,
|
||||
"output_path": os.path.join(OUT, "stitch_crossfade.mp4"),
|
||||
})
|
||||
print(f"Success: {r4.success}, Duration: {r4.duration_seconds:.2f}s")
|
||||
if r4.data: print(f" Output duration: {r4.data.get('duration')}s, Method: {r4.data.get('method')}")
|
||||
if r4.error: print(f"Error: {r4.error}")
|
||||
|
||||
# --- Test 5: Fade-through-black (3 clips) ---
|
||||
print("\n--- Test 5: Fade through black (3 clips) ---")
|
||||
r5 = tool.execute({
|
||||
"operation": "stitch",
|
||||
"clips": [CLIP_1, CLIP_2, CLIP_3],
|
||||
"transition": "fade",
|
||||
"transition_duration": 0.5,
|
||||
"output_path": os.path.join(OUT, "stitch_fadeblack.mp4"),
|
||||
})
|
||||
print(f"Success: {r5.success}, Duration: {r5.duration_seconds:.2f}s")
|
||||
if r5.data: print(f" Output duration: {r5.data.get('duration')}s, Method: {r5.data.get('method')}")
|
||||
if r5.error: print(f"Error: {r5.error}")
|
||||
|
||||
# --- Test 6: Auto-normalize mismatched clips ---
|
||||
print("\n--- Test 6: Stitch mismatched clips (auto_normalize) ---")
|
||||
r6 = tool.execute({
|
||||
"operation": "stitch",
|
||||
"clips": [CLIP_1, CLIP_MISMATCH],
|
||||
"transition": "cut",
|
||||
"auto_normalize": True,
|
||||
"output_path": os.path.join(OUT, "stitch_normalized.mp4"),
|
||||
})
|
||||
print(f"Success: {r6.success}, Duration: {r6.duration_seconds:.2f}s")
|
||||
if r6.data: print(f" Output duration: {r6.data.get('duration')}s, Normalized: {r6.data.get('auto_normalized')}")
|
||||
if r6.error: print(f"Error: {r6.error}")
|
||||
|
||||
# --- Test 7: Preview stitch (low-res) ---
|
||||
print("\n--- Test 7: Preview stitch ---")
|
||||
r7 = tool.execute({
|
||||
"operation": "preview_stitch",
|
||||
"clips": [CLIP_1, CLIP_2, CLIP_3],
|
||||
"transition": "cut",
|
||||
"output_path": os.path.join(OUT, "stitch_preview.mp4"),
|
||||
})
|
||||
print(f"Success: {r7.success}, Duration: {r7.duration_seconds:.2f}s")
|
||||
if r7.data: print(f" Preview resolution: {r7.data.get('preview_resolution')}")
|
||||
if r7.error: print(f"Error: {r7.error}")
|
||||
|
||||
# --- Test 8: Spatial — side by side ---
|
||||
print("\n--- Test 8: Spatial side-by-side ---")
|
||||
r8 = tool.execute({
|
||||
"operation": "spatial",
|
||||
"clips": [CLIP_1, CLIP_2],
|
||||
"layout": "side_by_side",
|
||||
"output_path": os.path.join(OUT, "stitch_side_by_side.mp4"),
|
||||
})
|
||||
print(f"Success: {r8.success}, Duration: {r8.duration_seconds:.2f}s")
|
||||
if r8.data: print(f" Layout: {r8.data.get('layout')}, Duration: {r8.data.get('duration')}s")
|
||||
if r8.error: print(f"Error: {r8.error}")
|
||||
|
||||
# --- Test 9: Spatial — picture-in-picture ---
|
||||
print("\n--- Test 9: Spatial PIP (bottom-right) ---")
|
||||
r9 = tool.execute({
|
||||
"operation": "spatial",
|
||||
"clips": [CLIP_1, CLIP_2],
|
||||
"layout": "picture_in_picture",
|
||||
"pip_position": "bottom_right",
|
||||
"pip_scale": 0.3,
|
||||
"pip_margin": 20,
|
||||
"output_path": os.path.join(OUT, "stitch_pip.mp4"),
|
||||
})
|
||||
print(f"Success: {r9.success}, Duration: {r9.duration_seconds:.2f}s")
|
||||
if r9.data: print(f" Layout: {r9.data.get('layout')}, Duration: {r9.data.get('duration')}s")
|
||||
if r9.error: print(f"Error: {r9.error}")
|
||||
|
||||
# --- Test 10: Spatial — vertical stack ---
|
||||
print("\n--- Test 10: Spatial vertical stack ---")
|
||||
r10 = tool.execute({
|
||||
"operation": "spatial",
|
||||
"clips": [CLIP_1, CLIP_2],
|
||||
"layout": "vertical_stack",
|
||||
"output_path": os.path.join(OUT, "stitch_vstack.mp4"),
|
||||
})
|
||||
print(f"Success: {r10.success}, Duration: {r10.duration_seconds:.2f}s")
|
||||
if r10.data: print(f" Layout: {r10.data.get('layout')}, Duration: {r10.data.get('duration')}s")
|
||||
if r10.error: print(f"Error: {r10.error}")
|
||||
|
||||
# --- Probe all video outputs ---
|
||||
print("\n--- Output inspection ---")
|
||||
outputs = [
|
||||
"stitch_cut.mp4",
|
||||
"stitch_crossfade.mp4",
|
||||
"stitch_fadeblack.mp4",
|
||||
"stitch_normalized.mp4",
|
||||
"stitch_preview.mp4",
|
||||
"stitch_side_by_side.mp4",
|
||||
"stitch_pip.mp4",
|
||||
"stitch_vstack.mp4",
|
||||
]
|
||||
for name in outputs:
|
||||
path = os.path.join(OUT, name)
|
||||
if os.path.exists(path):
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "quiet", "-print_format", "json",
|
||||
"-show_format", "-show_streams", path],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
info = json.loads(probe.stdout)
|
||||
fmt = info.get("format", {})
|
||||
video = {}
|
||||
audio = {}
|
||||
for s in info.get("streams", []):
|
||||
if s.get("codec_type") == "video" and not video:
|
||||
video = s
|
||||
elif s.get("codec_type") == "audio" and not audio:
|
||||
audio = s
|
||||
print(f"\n[{name}]"
|
||||
f" Duration: {fmt.get('duration', '?')}s"
|
||||
f" | Video: {video.get('width', '?')}x{video.get('height', '?')}"
|
||||
f" {video.get('codec_name', '?')}"
|
||||
f" | Audio: {audio.get('codec_name', '?')}"
|
||||
f" | Size: {os.path.getsize(path)} bytes")
|
||||
else:
|
||||
print(f"\n[{name}] FILE NOT FOUND")
|
||||
|
||||
print("\n=== VIDEO STITCH TEST COMPLETE ===")
|
||||
279
tests/qa/test_07_playbook_intelligence.py
Normal file
279
tests/qa/test_07_playbook_intelligence.py
Normal file
@@ -0,0 +1,279 @@
|
||||
#!/usr/bin/env python3
|
||||
"""QA Test 07: Playbook design intelligence — no API calls.
|
||||
|
||||
Tests contrast validation, color harmony generation, color-blind safety,
|
||||
type scale computation, type hierarchy validation, font pairing suggestions,
|
||||
and full accessibility audit across all 3 playbooks.
|
||||
"""
|
||||
|
||||
import sys, os, json
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
|
||||
|
||||
from styles.playbook_loader import (
|
||||
load_playbook,
|
||||
validate_playbook,
|
||||
list_playbooks,
|
||||
validate_contrast,
|
||||
check_color_blind_safety,
|
||||
validate_palette,
|
||||
generate_harmony,
|
||||
compute_type_scale,
|
||||
validate_type_hierarchy,
|
||||
suggest_font_pairing,
|
||||
validate_accessibility,
|
||||
TYPE_SCALE_RATIOS,
|
||||
)
|
||||
|
||||
PASS = 0
|
||||
FAIL = 0
|
||||
|
||||
def check(name, condition, detail=""):
|
||||
global PASS, FAIL
|
||||
status = "PASS" if condition else "FAIL"
|
||||
if condition:
|
||||
PASS += 1
|
||||
else:
|
||||
FAIL += 1
|
||||
suffix = f" — {detail}" if detail else ""
|
||||
print(f" [{status}] {name}{suffix}")
|
||||
|
||||
# ===================================================================
|
||||
# Test 1: List and load all playbooks
|
||||
# ===================================================================
|
||||
print("--- Test 1: List and load playbooks ---")
|
||||
playbooks_available = list_playbooks()
|
||||
print(f" Found {len(playbooks_available)} playbooks: {playbooks_available}")
|
||||
check("At least 3 playbooks exist", len(playbooks_available) >= 3)
|
||||
|
||||
loaded = {}
|
||||
for name in ["clean-professional", "flat-motion-graphics", "minimalist-diagram"]:
|
||||
try:
|
||||
pb = load_playbook(name)
|
||||
loaded[name] = pb
|
||||
check(f"Load + validate {name}", True)
|
||||
except Exception as e:
|
||||
loaded[name] = None
|
||||
check(f"Load + validate {name}", False, str(e))
|
||||
|
||||
# ===================================================================
|
||||
# Test 2: Contrast validation (manual spot-checks)
|
||||
# ===================================================================
|
||||
print("\n--- Test 2: Contrast validation ---")
|
||||
|
||||
# Known pair: black on white should be 21:1
|
||||
r = validate_contrast("#000000", "#FFFFFF")
|
||||
check("Black on white ~21:1", abs(r["ratio"] - 21.0) < 0.1, f"ratio={r['ratio']}")
|
||||
check("Black on white passes AAA", r["normal_text"]["AAA"])
|
||||
|
||||
# Known pair: white on white should be 1:1
|
||||
r = validate_contrast("#FFFFFF", "#FFFFFF")
|
||||
check("White on white = 1:1", abs(r["ratio"] - 1.0) < 0.01, f"ratio={r['ratio']}")
|
||||
check("White on white fails AA", not r["normal_text"]["AA"])
|
||||
|
||||
# Mid-gray on white (~4.5:1 boundary)
|
||||
r = validate_contrast("#767676", "#FFFFFF")
|
||||
check("#767676 on white passes AA normal", r["normal_text"]["AA"], f"ratio={r['ratio']}")
|
||||
|
||||
# Just-below threshold
|
||||
r = validate_contrast("#777777", "#FFFFFF")
|
||||
check("#777777 on white borderline", r["ratio"] >= 4.4, f"ratio={r['ratio']}")
|
||||
|
||||
# Dark-on-dark: low contrast
|
||||
r = validate_contrast("#1A1A1A", "#2B2B2B")
|
||||
check("Dark-on-dark fails AA", not r["normal_text"]["AA"], f"ratio={r['ratio']}")
|
||||
|
||||
# Check each playbook's text-on-bg contrast
|
||||
for name, pb in loaded.items():
|
||||
if pb is None:
|
||||
continue
|
||||
palette = pb.get("visual_language", {}).get("color_palette", {})
|
||||
text = palette.get("text", "#000000")
|
||||
bg = palette.get("background", "#FFFFFF")
|
||||
r = validate_contrast(text, bg)
|
||||
check(f"{name}: text on bg passes AA", r["normal_text"]["AA"], f"ratio={r['ratio']}")
|
||||
|
||||
# ===================================================================
|
||||
# Test 3: Color harmony generation
|
||||
# ===================================================================
|
||||
print("\n--- Test 3: Color harmony generation ---")
|
||||
|
||||
for harmony_type in ["complementary", "analogous", "triadic", "split-complementary"]:
|
||||
colors = generate_harmony("#3B82F6", harmony_type)
|
||||
check(f"Harmony {harmony_type}", len(colors) >= 2, f"generated {len(colors)} colors: {colors}")
|
||||
# First color should match base
|
||||
check(f" Base preserved in {harmony_type}", colors[0].upper() == generate_harmony("#3B82F6", harmony_type)[0].upper())
|
||||
|
||||
# Edge case: pure red
|
||||
colors = generate_harmony("#FF0000", "triadic")
|
||||
check("Triadic from pure red", len(colors) == 3, f"{colors}")
|
||||
|
||||
# ===================================================================
|
||||
# Test 4: Color-blind safety
|
||||
# ===================================================================
|
||||
print("\n--- Test 4: Color-blind safety ---")
|
||||
|
||||
# Safe palette (blue + orange — distinguishable by all CVD types due to lightness diff)
|
||||
safe = check_color_blind_safety(["#2563EB", "#F59E0B"])
|
||||
print(f" Blue + orange: safe={safe['safe']}, issues={len(safe.get('issues', []))}")
|
||||
|
||||
# Problematic palette (red + green, similar lightness)
|
||||
risky = check_color_blind_safety(["#DC2626", "#16A34A"])
|
||||
print(f" Red + green: safe={risky['safe']}, issues={len(risky.get('issues', []))}")
|
||||
check("Red+green flagged as risky", not risky["safe"] or len(risky.get("issues", [])) > 0,
|
||||
"should flag deuteranopia/protanopia")
|
||||
|
||||
# Single color = no pairs to check
|
||||
single = check_color_blind_safety(["#FF0000"])
|
||||
check("Single color is safe", single["safe"])
|
||||
|
||||
# Grays should be safe (low saturation)
|
||||
grays = check_color_blind_safety(["#333333", "#999999", "#CCCCCC"])
|
||||
check("Grays are safe", grays["safe"])
|
||||
|
||||
# ===================================================================
|
||||
# Test 5: Full palette validation per playbook
|
||||
# ===================================================================
|
||||
print("\n--- Test 5: Palette validation (all playbooks) ---")
|
||||
|
||||
for name, pb in loaded.items():
|
||||
if pb is None:
|
||||
continue
|
||||
issues = validate_palette(pb)
|
||||
errors = [i for i in issues if i.get("severity") == "error"]
|
||||
warnings = [i for i in issues if i.get("severity") == "warning"]
|
||||
print(f" [{name}] {len(errors)} errors, {len(warnings)} warnings, {len(issues)} total issues")
|
||||
check(f"{name}: no contrast errors", len(errors) == 0,
|
||||
"; ".join(e["message"] for e in errors) if errors else "all clear")
|
||||
for issue in issues:
|
||||
sev = issue.get("severity", "?")
|
||||
print(f" [{sev}] {issue.get('message', '')}")
|
||||
|
||||
# ===================================================================
|
||||
# Test 6: Type scale computation
|
||||
# ===================================================================
|
||||
print("\n--- Test 6: Type scale computation ---")
|
||||
|
||||
for ratio_name, ratio_val in TYPE_SCALE_RATIOS.items():
|
||||
scale = compute_type_scale(24, ratio_name)
|
||||
sizes = scale["sizes"]
|
||||
check(f"Scale {ratio_name}: display > heading > subheading > body > caption",
|
||||
sizes["display"] > sizes["heading"] > sizes["subheading"] > sizes["body"] > sizes["caption"],
|
||||
f"{sizes}")
|
||||
check(f" Base preserved", sizes["body"] == 24)
|
||||
|
||||
# Custom numeric ratio
|
||||
scale = compute_type_scale(24, "1.5")
|
||||
check("Custom ratio 1.5", scale["ratio_value"] == 1.5, f"sizes={scale['sizes']}")
|
||||
|
||||
# ===================================================================
|
||||
# Test 7: Type hierarchy validation
|
||||
# ===================================================================
|
||||
print("\n--- Test 7: Type hierarchy validation ---")
|
||||
|
||||
for name, pb in loaded.items():
|
||||
if pb is None:
|
||||
continue
|
||||
issues = validate_type_hierarchy(pb)
|
||||
print(f" [{name}] {len(issues)} type hierarchy issues")
|
||||
for issue in issues:
|
||||
print(f" [{issue.get('severity')}] {issue.get('message')}")
|
||||
# No errors expected in shipped playbooks
|
||||
errors = [i for i in issues if i.get("severity") == "error"]
|
||||
check(f"{name}: no type hierarchy errors", len(errors) == 0)
|
||||
|
||||
# Deliberately bad playbook
|
||||
bad_typography = {
|
||||
"typography": {
|
||||
"headings": {"font": "Inter", "weight": 400},
|
||||
"body": {"font": "Inter", "weight": 400},
|
||||
"stat_card": {"font": "Inter", "size_multiplier": 0.8},
|
||||
}
|
||||
}
|
||||
issues = validate_type_hierarchy(bad_typography)
|
||||
check("Bad typography flagged", len(issues) > 0, f"{len(issues)} issues found")
|
||||
|
||||
# ===================================================================
|
||||
# Test 8: Font pairing suggestions
|
||||
# ===================================================================
|
||||
print("\n--- Test 8: Font pairing suggestions ---")
|
||||
|
||||
for font in ["Inter", "Space Grotesk", "IBM Plex Sans", "Lora", "JetBrains Mono"]:
|
||||
pairings = suggest_font_pairing(font)
|
||||
check(f"Pairings for {font}", len(pairings) >= 1, f"{len(pairings)} suggestions")
|
||||
for p in pairings:
|
||||
print(f" → {p['font']} ({p['category']}): {p['rationale']}")
|
||||
|
||||
# Unknown font fallback
|
||||
pairings = suggest_font_pairing("UnknownFont")
|
||||
check("Unknown font gets fallback", len(pairings) >= 1)
|
||||
|
||||
# ===================================================================
|
||||
# Test 9: Full accessibility audit (all playbooks)
|
||||
# ===================================================================
|
||||
print("\n--- Test 9: Accessibility audit (all playbooks) ---")
|
||||
|
||||
for name, pb in loaded.items():
|
||||
if pb is None:
|
||||
continue
|
||||
result = validate_accessibility(pb)
|
||||
status = "PASS" if result["pass"] else "FAIL"
|
||||
print(f"\n [{name}] Overall: {status}"
|
||||
f" | Errors: {result['error_count']}"
|
||||
f" | Warnings: {result['warning_count']}"
|
||||
f" | Total: {result['total_issues']}")
|
||||
check(f"{name}: a11y audit passes", result["pass"])
|
||||
for issue in result["issues"]:
|
||||
cat = issue.get("category", "?")
|
||||
sev = issue.get("severity", "?")
|
||||
print(f" [{cat}/{sev}] {issue.get('message', '')}")
|
||||
|
||||
# ===================================================================
|
||||
# Test 10: Deliberately low-contrast custom playbook
|
||||
# ===================================================================
|
||||
print("\n--- Test 10: Low-contrast custom playbook ---")
|
||||
|
||||
low_contrast_pb = {
|
||||
"identity": {"name": "low-contrast-test", "category": "test", "mood": "test", "pace": "moderate", "best_for": ["testing"]},
|
||||
"visual_language": {
|
||||
"color_palette": {
|
||||
"primary": ["#555555"],
|
||||
"accent": ["#666666"],
|
||||
"background": "#444444",
|
||||
"text": "#555555",
|
||||
"muted": "#4A4A4A",
|
||||
},
|
||||
"composition": "centered",
|
||||
"texture": "none",
|
||||
},
|
||||
"typography": {
|
||||
"headings": {"font": "Arial", "weight": 700, "size_multiplier": 1.5},
|
||||
"body": {"font": "Arial", "weight": 400, "size_multiplier": 1.0},
|
||||
"code": {"font": "Courier", "weight": 400, "size_multiplier": 0.9},
|
||||
"stat_card": {"font": "Arial", "weight": 700, "size_multiplier": 2.5},
|
||||
"scale_system": "major_third",
|
||||
"weight_matrix": {"title": 800, "heading": 700, "body": 400, "caption": 300},
|
||||
},
|
||||
"motion": {"transitions": "cut", "animation_style": "none", "pacing_rules": {}},
|
||||
"audio": {"voice_style": "neutral", "music_mood": "none"},
|
||||
"asset_generation": {"image_prompt_prefix": "test", "negative_prompt": ""},
|
||||
"overlays": {
|
||||
"stat_card": {"bg": "#444444", "text": "#555555", "border": "#444444", "radius": 8, "shadow": "none"},
|
||||
},
|
||||
"quality_rules": [],
|
||||
"chart_palette": ["#555555", "#666666", "#777777"],
|
||||
}
|
||||
|
||||
# This should be flagged with errors
|
||||
issues = validate_palette(low_contrast_pb)
|
||||
errors = [i for i in issues if i.get("severity") == "error"]
|
||||
check("Low-contrast playbook has errors", len(errors) > 0, f"{len(errors)} contrast errors")
|
||||
for e in errors:
|
||||
print(f" [error] {e.get('message')}")
|
||||
|
||||
# ===================================================================
|
||||
# Summary
|
||||
# ===================================================================
|
||||
print(f"\n{'='*60}")
|
||||
print(f"PLAYBOOK INTELLIGENCE TEST COMPLETE: {PASS} passed, {FAIL} failed")
|
||||
print(f"{'='*60}")
|
||||
548
tests/qa/test_08_end_to_end.py
Normal file
548
tests/qa/test_08_end_to_end.py
Normal file
@@ -0,0 +1,548 @@
|
||||
#!/usr/bin/env python3
|
||||
"""QA Test 08: End-to-end animated-explainer pipeline simulation.
|
||||
|
||||
Walks through all 7 stages (idea -> publish) with synthetic artifacts,
|
||||
validating checkpoints, artifact schemas, and cost tracking at each step.
|
||||
The compose stage runs real tools (audio_mixer + video_compose) to produce
|
||||
an actual output video. All other stages use synthetic data.
|
||||
|
||||
No API keys needed -- uses ffmpeg-generated fixtures throughout.
|
||||
"""
|
||||
|
||||
import sys, os, json, subprocess, shutil
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = str(Path(__file__).resolve().parent.parent.parent)
|
||||
sys.path.insert(0, PROJECT_ROOT)
|
||||
|
||||
from lib.env_loader import load_env
|
||||
load_env()
|
||||
|
||||
from lib.checkpoint import (
|
||||
write_checkpoint,
|
||||
read_checkpoint,
|
||||
get_completed_stages,
|
||||
get_next_stage,
|
||||
STAGES,
|
||||
CANONICAL_STAGE_ARTIFACTS,
|
||||
)
|
||||
from tools.cost_tracker import CostTracker, BudgetMode
|
||||
from schemas.artifacts import validate_artifact, list_schemas
|
||||
from styles.playbook_loader import load_playbook, validate_accessibility
|
||||
|
||||
OUT = os.path.join(os.path.dirname(__file__), "output")
|
||||
PIPELINE_DIR = Path(OUT) / "e2e_pipeline"
|
||||
PROJECT_ID = "qa_e2e_test"
|
||||
ASSETS_DIR = Path(OUT) / "e2e_assets"
|
||||
|
||||
# Clean previous run
|
||||
if PIPELINE_DIR.exists():
|
||||
shutil.rmtree(PIPELINE_DIR)
|
||||
if ASSETS_DIR.exists():
|
||||
shutil.rmtree(ASSETS_DIR)
|
||||
ASSETS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
PASS = 0
|
||||
FAIL = 0
|
||||
|
||||
def check(name, condition, detail=""):
|
||||
global PASS, FAIL
|
||||
if condition:
|
||||
PASS += 1
|
||||
print(f" [PASS] {name}" + (f" -- {detail}" if detail else ""))
|
||||
else:
|
||||
FAIL += 1
|
||||
print(f" [FAIL] {name}" + (f" -- {detail}" if detail else ""))
|
||||
|
||||
def ensure_audio(path, duration=5):
|
||||
if os.path.exists(path):
|
||||
return
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-f", "lavfi", "-i",
|
||||
f"sine=frequency=440:duration={duration}",
|
||||
"-ar", "44100", "-ac", "1", path],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
def ensure_video(path, duration=5, width=1280, height=720, color="blue"):
|
||||
if os.path.exists(path):
|
||||
return
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y",
|
||||
"-f", "lavfi", "-i", f"color=c={color}:s={width}x{height}:d={duration}:r=30",
|
||||
"-f", "lavfi", "-i", f"sine=frequency=440:duration={duration}",
|
||||
"-c:v", "libx264", "-crf", "23", "-pix_fmt", "yuv420p",
|
||||
"-g", "30", "-keyint_min", "30",
|
||||
"-c:a", "aac", "-shortest", path],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
# ===================================================================
|
||||
# Setup: Cost tracker + playbook
|
||||
# ===================================================================
|
||||
print("--- Setup ---")
|
||||
cost_log = PIPELINE_DIR / PROJECT_ID / "cost_log.json"
|
||||
tracker = CostTracker(
|
||||
budget_total_usd=5.0,
|
||||
mode=BudgetMode.OBSERVE,
|
||||
cost_log_path=cost_log,
|
||||
)
|
||||
print(f" Budget: ${tracker.budget_total_usd}")
|
||||
print(f" Available schemas: {list_schemas()}")
|
||||
|
||||
playbook = load_playbook("clean-professional")
|
||||
a11y = validate_accessibility(playbook)
|
||||
print(f" Playbook a11y: pass={a11y['pass']}, errors={a11y['error_count']}, warnings={a11y['warning_count']}")
|
||||
|
||||
# ===================================================================
|
||||
# Stage 1: idea -> brief
|
||||
# ===================================================================
|
||||
print("\n--- Stage 1: idea ---")
|
||||
brief = {
|
||||
"version": "1.0",
|
||||
"title": "AI Video Production in 60 Seconds",
|
||||
"hook": "What if you could create a professional video in 60 seconds with just a text prompt?",
|
||||
"key_points": [
|
||||
"Traditional video production takes days or weeks",
|
||||
"AI can automate scripting, visuals, narration, and editing",
|
||||
"OpenMontage orchestrates the full pipeline",
|
||||
],
|
||||
"tone": "confident, energetic",
|
||||
"style": "clean-professional",
|
||||
"target_platform": "youtube",
|
||||
"target_duration_seconds": 60,
|
||||
"target_audience": "content creators and developers",
|
||||
"cta": "Try OpenMontage today",
|
||||
"angle_options": [
|
||||
{"name": "democratization", "description": "AI makes video creation accessible to everyone"},
|
||||
{"name": "workflow", "description": "AI automates the tedious parts of video production"},
|
||||
{"name": "quality", "description": "AI-generated content is reaching professional quality"},
|
||||
],
|
||||
"selected_angle": "workflow",
|
||||
}
|
||||
|
||||
try:
|
||||
validate_artifact("brief", brief)
|
||||
check("Brief validates against schema", True)
|
||||
except Exception as e:
|
||||
check("Brief validates against schema", False, str(e))
|
||||
|
||||
cp_path = write_checkpoint(
|
||||
PIPELINE_DIR, PROJECT_ID, "idea", "completed",
|
||||
artifacts={"brief": brief},
|
||||
pipeline_type="animated-explainer",
|
||||
style_playbook="clean-professional",
|
||||
)
|
||||
check("Idea checkpoint written", cp_path.exists())
|
||||
# Next uncompleted stage in global STAGES order (research/proposal come before idea)
|
||||
check("Next stage after idea", get_next_stage(PIPELINE_DIR, PROJECT_ID) == "research")
|
||||
|
||||
# ===================================================================
|
||||
# Stage 2: script
|
||||
# ===================================================================
|
||||
print("\n--- Stage 2: script ---")
|
||||
|
||||
# Section timestamps (cumulative)
|
||||
SECTIONS = [
|
||||
("s1_hook", "Hook", 0, 8, "What if creating a professional video took less time than making your morning coffee?"),
|
||||
("s2_setup", "Setup", 8, 20, "Traditional video production involves scripting, filming, editing, and post-production. It takes days, sometimes weeks."),
|
||||
("s3_build", "Build", 20, 38, "Now imagine an AI that handles all of that. You type a topic, and it writes the script, generates visuals, creates narration, mixes audio, and delivers a finished video."),
|
||||
("s4_climax", "Climax", 38, 50, "This is not science fiction. OpenMontage is an open-source platform that orchestrates AI tools into a complete video pipeline."),
|
||||
("s5_landing", "Landing", 50, 60, "The future of video creation is open, automated, and available right now. Try it yourself."),
|
||||
]
|
||||
|
||||
script = {
|
||||
"version": "1.0",
|
||||
"title": "AI Video Production in 60 Seconds",
|
||||
"total_duration_seconds": 60,
|
||||
"sections": [
|
||||
{
|
||||
"id": sid,
|
||||
"label": label,
|
||||
"text": text,
|
||||
"start_seconds": start,
|
||||
"end_seconds": end,
|
||||
"speaker_directions": "Confident, engaging tone",
|
||||
"enhancement_cues": [
|
||||
{"type": "overlay", "description": f"Visual for {label} section", "timestamp_seconds": start + 2},
|
||||
],
|
||||
}
|
||||
for sid, label, start, end, text in SECTIONS
|
||||
],
|
||||
}
|
||||
|
||||
try:
|
||||
validate_artifact("script", script)
|
||||
check("Script validates against schema", True)
|
||||
except Exception as e:
|
||||
check("Script validates against schema", False, str(e))
|
||||
|
||||
write_checkpoint(
|
||||
PIPELINE_DIR, PROJECT_ID, "script", "completed",
|
||||
artifacts={"script": script},
|
||||
pipeline_type="animated-explainer",
|
||||
)
|
||||
check("Completed stages", get_completed_stages(PIPELINE_DIR, PROJECT_ID) == ["idea", "script"]) # idea and script appear in STAGES order
|
||||
|
||||
# ===================================================================
|
||||
# Stage 3: scene_plan
|
||||
# ===================================================================
|
||||
print("\n--- Stage 3: scene_plan ---")
|
||||
|
||||
SCENE_TYPES = ["text_card", "diagram", "animation", "generated", "text_card"]
|
||||
scene_plan = {
|
||||
"version": "1.0",
|
||||
"style_playbook": "clean-professional",
|
||||
"scenes": [
|
||||
{
|
||||
"id": f"sc{i+1}",
|
||||
"type": SCENE_TYPES[i],
|
||||
"description": f"Scene for {label} section",
|
||||
"start_seconds": start,
|
||||
"end_seconds": end,
|
||||
"script_section_id": sid,
|
||||
"required_assets": [
|
||||
{"type": "narration", "description": f"TTS narration for {label}", "source": "generate"},
|
||||
{"type": "image", "description": f"Visual for {label}", "source": "generate"},
|
||||
],
|
||||
}
|
||||
for i, (sid, label, start, end, _) in enumerate(SECTIONS)
|
||||
],
|
||||
}
|
||||
|
||||
try:
|
||||
validate_artifact("scene_plan", scene_plan)
|
||||
check("Scene plan validates against schema", True)
|
||||
except Exception as e:
|
||||
check("Scene plan validates against schema", False, str(e))
|
||||
|
||||
write_checkpoint(
|
||||
PIPELINE_DIR, PROJECT_ID, "scene_plan", "completed",
|
||||
artifacts={"scene_plan": scene_plan},
|
||||
pipeline_type="animated-explainer",
|
||||
)
|
||||
|
||||
# ===================================================================
|
||||
# Stage 4: assets (generate fixtures)
|
||||
# ===================================================================
|
||||
print("\n--- Stage 4: assets ---")
|
||||
|
||||
# Generate TTS fixtures (one per section)
|
||||
tts_files = {}
|
||||
for sid, label, start, end, _ in SECTIONS:
|
||||
path = str(ASSETS_DIR / f"tts_{sid}.mp3")
|
||||
ensure_audio(path, duration=end - start)
|
||||
tts_files[sid] = path
|
||||
|
||||
# Generate music fixture
|
||||
music_path = str(ASSETS_DIR / "music_bg.mp3")
|
||||
ensure_audio(music_path, duration=60)
|
||||
|
||||
# Build asset manifest (schema: id, type, path, source_tool, scene_id required)
|
||||
clean_assets = []
|
||||
for i, (sid, label, start, end, _) in enumerate(SECTIONS):
|
||||
scene_id = f"sc{i+1}"
|
||||
clean_assets.append({
|
||||
"id": f"a_tts_{sid}",
|
||||
"type": "narration",
|
||||
"path": tts_files[sid],
|
||||
"source_tool": "elevenlabs_tts",
|
||||
"scene_id": scene_id,
|
||||
})
|
||||
# Image fixture
|
||||
img_path = str(ASSETS_DIR / f"img_{scene_id}.png")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-f", "lavfi", "-i",
|
||||
f"color=c=darkblue:s=1280x720:d=1", "-frames:v", "1", img_path],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
clean_assets.append({
|
||||
"id": f"a_img_{scene_id}",
|
||||
"type": "image",
|
||||
"path": img_path,
|
||||
"source_tool": "image_selector",
|
||||
"scene_id": scene_id,
|
||||
})
|
||||
|
||||
clean_assets.append({
|
||||
"id": "a_music",
|
||||
"type": "music",
|
||||
"path": music_path,
|
||||
"source_tool": "music_gen",
|
||||
"scene_id": "sc1",
|
||||
})
|
||||
|
||||
asset_manifest = {
|
||||
"version": "1.0",
|
||||
"assets": clean_assets,
|
||||
"total_cost_usd": 0.0,
|
||||
}
|
||||
|
||||
try:
|
||||
validate_artifact("asset_manifest", asset_manifest)
|
||||
check("Asset manifest validates against schema", True)
|
||||
except Exception as e:
|
||||
check("Asset manifest validates against schema", False, str(e))
|
||||
|
||||
# Verify all files exist
|
||||
all_exist = all(os.path.exists(a["path"]) for a in asset_manifest["assets"])
|
||||
check("All asset files exist on disk", all_exist)
|
||||
|
||||
# Track costs
|
||||
eid = tracker.estimate("image_selector", "generate", 0.15)
|
||||
tracker.approve_tool("image_selector")
|
||||
tracker.reserve(eid)
|
||||
tracker.reconcile(eid, 0.0, success=True)
|
||||
print(f" Cost snapshot: {tracker.cost_snapshot()}")
|
||||
|
||||
write_checkpoint(
|
||||
PIPELINE_DIR, PROJECT_ID, "assets", "completed",
|
||||
artifacts={"asset_manifest": asset_manifest},
|
||||
pipeline_type="animated-explainer",
|
||||
cost_snapshot=tracker.cost_snapshot(),
|
||||
)
|
||||
|
||||
# ===================================================================
|
||||
# Stage 5: edit (edit_decisions)
|
||||
# ===================================================================
|
||||
print("\n--- Stage 5: edit ---")
|
||||
|
||||
# Create video clips for the compose step
|
||||
colors = ["darkblue", "darkgreen", "darkorange", "darkred", "purple"]
|
||||
scene_videos = {}
|
||||
for i, scene in enumerate(scene_plan["scenes"]):
|
||||
scid = scene["id"]
|
||||
dur = scene["end_seconds"] - scene["start_seconds"]
|
||||
vid_path = str(ASSETS_DIR / f"scene_{scid}.mp4")
|
||||
ensure_video(vid_path, duration=dur, color=colors[i % len(colors)])
|
||||
scene_videos[scid] = vid_path
|
||||
|
||||
edit_decisions = {
|
||||
"version": "1.0",
|
||||
"cuts": [
|
||||
{
|
||||
"id": f"cut_{scene['id']}",
|
||||
"source": scene_videos[scene["id"]],
|
||||
"in_seconds": 0,
|
||||
"out_seconds": scene["end_seconds"] - scene["start_seconds"],
|
||||
"speed": 1.0,
|
||||
}
|
||||
for scene in scene_plan["scenes"]
|
||||
],
|
||||
"music": {
|
||||
"asset_id": "a_music",
|
||||
"volume": 0.2,
|
||||
"ducking": True,
|
||||
"fade_in_seconds": 1.0,
|
||||
"fade_out_seconds": 2.0,
|
||||
},
|
||||
"subtitles": {
|
||||
"enabled": True,
|
||||
"style": "clean-professional",
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
validate_artifact("edit_decisions", edit_decisions)
|
||||
check("Edit decisions validates against schema", True)
|
||||
except Exception as e:
|
||||
check("Edit decisions validates against schema", False, str(e))
|
||||
|
||||
write_checkpoint(
|
||||
PIPELINE_DIR, PROJECT_ID, "edit", "completed",
|
||||
artifacts={"edit_decisions": edit_decisions},
|
||||
pipeline_type="animated-explainer",
|
||||
)
|
||||
|
||||
# ===================================================================
|
||||
# Stage 6: compose (REAL tool execution)
|
||||
# ===================================================================
|
||||
print("\n--- Stage 6: compose (real tools) ---")
|
||||
|
||||
from tools.audio.audio_mixer import AudioMixer
|
||||
from tools.video.video_compose import VideoCompose
|
||||
|
||||
# Step 1: Mix narration + music
|
||||
print(" Mixing audio...")
|
||||
mixer = AudioMixer()
|
||||
mix_output = str(ASSETS_DIR / "final_mix.wav")
|
||||
|
||||
# Combine all narration into one track first
|
||||
concat_narration = str(ASSETS_DIR / "narration_concat.wav")
|
||||
narration_list = str(ASSETS_DIR / "narration_list.txt")
|
||||
with open(narration_list, "w") as f:
|
||||
for sid, _, _, _, _ in SECTIONS:
|
||||
safe = tts_files[sid].replace("\\", "/")
|
||||
f.write(f"file '{safe}'\n")
|
||||
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", narration_list,
|
||||
"-c", "copy", concat_narration],
|
||||
capture_output=True, check=True,
|
||||
)
|
||||
|
||||
mix_result = mixer.execute({
|
||||
"operation": "duck",
|
||||
"tracks": [
|
||||
{"path": concat_narration, "role": "speech"},
|
||||
{"path": music_path, "role": "music"},
|
||||
],
|
||||
"ducking": {"enabled": True, "music_volume_during_speech": 0.15},
|
||||
"output_path": mix_output,
|
||||
})
|
||||
check("Audio mix succeeded", mix_result.success, mix_result.error or "")
|
||||
|
||||
# Step 2: Compose video
|
||||
print(" Composing video...")
|
||||
composer = VideoCompose()
|
||||
final_video = str(Path(OUT) / "e2e_final_output.mp4")
|
||||
|
||||
compose_result = composer.execute({
|
||||
"operation": "compose",
|
||||
"edit_decisions": {
|
||||
"cuts": [
|
||||
{"source": c["source"], "in_seconds": c["in_seconds"], "out_seconds": c["out_seconds"], "speed": c.get("speed", 1.0)}
|
||||
for c in edit_decisions["cuts"]
|
||||
],
|
||||
},
|
||||
"audio_path": mix_output,
|
||||
"codec": "libx264",
|
||||
"crf": 23,
|
||||
"preset": "fast",
|
||||
"output_path": final_video,
|
||||
})
|
||||
check("Video compose succeeded", compose_result.success, compose_result.error or "")
|
||||
check("Output video exists", os.path.exists(final_video))
|
||||
|
||||
# Probe the output
|
||||
duration = 0.0
|
||||
video_stream = {}
|
||||
audio_stream = {}
|
||||
if os.path.exists(final_video):
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "quiet", "-print_format", "json",
|
||||
"-show_format", "-show_streams", final_video],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
info = json.loads(probe.stdout)
|
||||
fmt = info.get("format", {})
|
||||
duration = float(fmt.get("duration", 0))
|
||||
for s in info.get("streams", []):
|
||||
if s.get("codec_type") == "video" and not video_stream:
|
||||
video_stream = s
|
||||
elif s.get("codec_type") == "audio" and not audio_stream:
|
||||
audio_stream = s
|
||||
|
||||
print(f" Output: {video_stream.get('width')}x{video_stream.get('height')}"
|
||||
f" {video_stream.get('codec_name')} | {duration:.1f}s"
|
||||
f" | Audio: {audio_stream.get('codec_name')}"
|
||||
f" | Size: {os.path.getsize(final_video)} bytes")
|
||||
|
||||
check("Video has audio track", bool(audio_stream))
|
||||
check("Video has video track", bool(video_stream))
|
||||
check("Duration > 30s", duration > 30, f"{duration:.1f}s")
|
||||
|
||||
render_report = {
|
||||
"version": "1.0",
|
||||
"outputs": [
|
||||
{
|
||||
"path": final_video,
|
||||
"format": "mp4",
|
||||
"codec": video_stream.get("codec_name", "h264"),
|
||||
"audio_codec": audio_stream.get("codec_name", "aac"),
|
||||
"resolution": f"{video_stream.get('width', 1280)}x{video_stream.get('height', 720)}",
|
||||
"fps": 30,
|
||||
"duration_seconds": round(duration, 2),
|
||||
"file_size_bytes": os.path.getsize(final_video) if os.path.exists(final_video) else 0,
|
||||
"platform_target": "youtube",
|
||||
}
|
||||
],
|
||||
"render_time_seconds": compose_result.duration_seconds,
|
||||
}
|
||||
|
||||
try:
|
||||
validate_artifact("render_report", render_report)
|
||||
check("Render report validates against schema", True)
|
||||
except Exception as e:
|
||||
check("Render report validates against schema", False, str(e))
|
||||
|
||||
write_checkpoint(
|
||||
PIPELINE_DIR, PROJECT_ID, "compose", "completed",
|
||||
artifacts={"render_report": render_report},
|
||||
pipeline_type="animated-explainer",
|
||||
cost_snapshot=tracker.cost_snapshot(),
|
||||
)
|
||||
|
||||
# ===================================================================
|
||||
# Stage 7: publish
|
||||
# ===================================================================
|
||||
print("\n--- Stage 7: publish ---")
|
||||
|
||||
publish_log = {
|
||||
"version": "1.0",
|
||||
"entries": [
|
||||
{
|
||||
"platform": "youtube",
|
||||
"status": "exported",
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"export_path": str(Path(OUT) / "e2e_export"),
|
||||
"metadata_used": {
|
||||
"title": "AI Video Production in 60 Seconds",
|
||||
"description": "See how AI orchestrates an entire video production pipeline.",
|
||||
"hashtags": ["#AI", "#VideoProduction", "#OpenMontage"],
|
||||
"chapters": [
|
||||
{"time": "0:00", "label": "Hook"},
|
||||
{"time": "0:08", "label": "The Problem"},
|
||||
{"time": "0:20", "label": "The Solution"},
|
||||
{"time": "0:38", "label": "OpenMontage"},
|
||||
{"time": "0:50", "label": "Try It"},
|
||||
],
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
try:
|
||||
validate_artifact("publish_log", publish_log)
|
||||
check("Publish log validates against schema", True)
|
||||
except Exception as e:
|
||||
check("Publish log validates against schema", False, str(e))
|
||||
|
||||
write_checkpoint(
|
||||
PIPELINE_DIR, PROJECT_ID, "publish", "completed",
|
||||
artifacts={"publish_log": publish_log},
|
||||
pipeline_type="animated-explainer",
|
||||
)
|
||||
|
||||
# ===================================================================
|
||||
# Final validation
|
||||
# ===================================================================
|
||||
print("\n--- Final validation ---")
|
||||
|
||||
completed = get_completed_stages(PIPELINE_DIR, PROJECT_ID)
|
||||
check("All 7 stages completed", len(completed) == 7, f"completed={completed}")
|
||||
check("Next stage is None (done)", get_next_stage(PIPELINE_DIR, PROJECT_ID) is None)
|
||||
check("Stages in correct order", completed == STAGES, f"{completed}")
|
||||
|
||||
# Verify all checkpoints are readable
|
||||
for stage in STAGES:
|
||||
cp = read_checkpoint(PIPELINE_DIR, PROJECT_ID, stage)
|
||||
check(f"Checkpoint {stage} readable", cp is not None)
|
||||
if cp:
|
||||
expected_artifact = CANONICAL_STAGE_ARTIFACTS[stage]
|
||||
check(f" Has canonical artifact '{expected_artifact}'", expected_artifact in cp.get("artifacts", {}))
|
||||
|
||||
# Cost summary
|
||||
print(f"\n Final cost: {tracker.cost_snapshot()}")
|
||||
|
||||
# ===================================================================
|
||||
# Summary
|
||||
# ===================================================================
|
||||
print(f"\n{'='*60}")
|
||||
print(f"END-TO-END TEST COMPLETE: {PASS} passed, {FAIL} failed")
|
||||
print(f"{'='*60}")
|
||||
|
||||
if os.path.exists(final_video):
|
||||
print(f"\nFinal video: {final_video}")
|
||||
print("INSPECT: Open in VLC/media player to verify A/V sync, transitions, and content.")
|
||||
Reference in New Issue
Block a user