Add an Azure AI Speech transcription tool. It is opt-in: when
AZURE_SPEECH_KEY is configured the agent may prefer it for cloud STT,
while the local faster-whisper `transcriber` stays the default offline
path. Shared pipeline manifests are intentionally left unchanged, so no
default provider selection is altered for existing users.
- tools/analysis/azure_stt.py: new `azure_stt` tool (capability=analysis,
provider=azure) calling the Fast Transcription REST API. The local file
is uploaded via multipart and transcribed synchronously with word-level
timestamps and optional diarization — no Blob storage or async polling.
Output schema mirrors `transcriber` exactly, so it is a drop-in for
`subtitle_gen` and other transcript consumers. Follows the existing
provider-tool conventions (env-var status check, `_transcribe` helper,
cost_usd/model on the result, fallback="transcriber").
- Auto-discovered by the registry; no registry or selector changes.
- tests/tools/test_azure_stt.py: contract, discovery, status, response
mapping, execute guardrails, and a mocked-network success path (no live
API calls).
- .agents/skills + .claude/skills: azure-speech-to-text Layer-3 skill.
- docs/PROVIDERS.md: Azure AI Speech setup, API notes, and pricing.
- .env.example, skills/INDEX.md, AGENT_GUIDE.md: document the optional
cloud STT path alongside the default whisper transcriber.
Address PR #240 review feedback from @calesthio:
1. dashscope_image: save EVERY returned image URL, not just the first.
The tool advertised multiple_outputs and accepted n>1 but only read
content[0], silently dropping paid outputs. Now collects all image
URLs across choices/content and downloads each to a distinct indexed
path (foo.png -> foo_1.png, foo_2.png, ...). images_generated now
reflects the actual count downloaded.
Per Qwen Cloud docs, a multi-output task is SUCCEEDED if at least one
image is generated; choices with finish_reason != "stop" are skipped
to avoid downloading partial/failed results.
2. Complete idempotency_key_fields so different requests no longer
collide and reuse stale artifacts:
- image: + negative_prompt, seed, prompt_extend, watermark
- tts: + instructions
- asr: + enable_words, language_hints
Adds 19 regression tests (114 total, all pass, no API keys needed):
- TestDashscopeImageMultiOutput: URL extraction across choices / within
one choice / failed-choice skipping, path resolution for
single/multi/no-extension, end-to-end multi-image download with a
mocked 3-URL DashScope response verifying all 3 files land on disk,
single-image legacy path behavior
- TestDashscopeIdempotencyKeys: field presence + key-differs-on-value
for every newly added field across all three tools
Separates creative grammar (renderer_family) from technical engine
(render_runtime) so HyperFrames can stand alongside Remotion as a
first-class runtime instead of masquerading as a Remotion sub-case.
Locks runtime choice at proposal stage and enforces it end-to-end: the
schemas require it, video_compose routes by it, the reviewer fails
closed on silent swaps, and a parametrized contract test walks every
pipeline manifest to ensure each planning-stage skill explains the
conversation to the user. Adds hyperframes_compose (scaffold/lint/
validate/render/doctor/add_block), a playbook -> CSS style bridge, and
vendored HyperFrames Layer 3 skills from commit d291358, pinned via
PROVENANCE.md for future re-sync. Final_review now records
render_runtime_used and runtime_swap_detected so compose lies are
catchable after the fact.
Video analyzer now uses Farneback dense optical flow to classify each
scene as motion_clip, animated_still, or static_image. This lets the
agent correctly identify whether a reference video uses AI-generated
video clips vs still images with pan/zoom — and plan the right pipeline.
Changes:
- video_analyzer.py: new Step 3b with _classify_scene_motion() and
_read_frame_at() helpers; updated _needs_motion() to use per-scene
motion data instead of pacing heuristic alone
- video-reference-analyst.md: added Motion line to summary template
and instructions to read motion_type field before proposing tools
Scene-director rewrite: agent now watches footage, understands content, and proposes
creative overlays (charts, stats, key terms, comparisons) before building anything.
Presents enhancement plan to user for approval before proceeding.
Compose-director fixes from Round 1 verification:
- eye_enhance: now explicitly required, not silently skippable
- Caption positioning: explicit MarginV=160 for 9:16, never center
- Final encode: mandatory with target file size table
- ASR corrections: new Step 2b to scan transcript and build corrections dict
- Overlay compositing: new Step 3b for burning approved graphics onto footage
Asset-director rewrite: generates Remotion overlay assets (callouts, stat cards,
charts, comparisons) from scene plan. Includes overlay type → Remotion cut mapping
table and dark theme requirements.
Bug fixes found during subagent verification:
- remotion_caption_burn.py: fix run_command API, add npx.cmd for Windows
- visual_qa.py: fix run_command API (3 places), Windows /dev/null → NUL
New tools: face_tracker, visual_qa, eye_enhance, auto_reframe, remotion_caption_burn, showcase_card, silence_cutter. Updated audio_mixer with segmented_music operation and subtitle_gen with ASR corrections. Registered TalkingHead composition in Root.tsx. Rewrote compose/edit/scene director skills for full enhancement chain, Remotion captions, multi-clip assembly, and visual QA. Gitignore cleanup: exclude test demo-props, downloaded music, and generated images.
Add anime_scene rendering engine (AnimeScene + ParticleOverlay components) with multi-image
crossfade, 9 camera motion types, 5 particle systems, and cinematic lighting overlays.
Fix critical Remotion durationInFrames footgun by passing sceneDurationSeconds from parent.
Add audio offset/loop support in Explainer for skipping quiet music intros.
New tools: audio_energy.py analyzes per-second loudness via ebur128 to find optimal music
offset and detect when looping is needed.
Update all 6 animation pipeline skills (proposal, scene, asset, compose, executive-producer,
remotion.md) with battle-tested image_animation workflow including tool availability scan,
FLUX multi-image generation, composition JSON format, pre-render validation, and post-render
self-review.
Add 3 demo compositions (Candyland, Mori no Seishin, Deep Ocean) and anime-ghibli style
playbook. Update README with 3 anime video showcases and animation prompts. Add Animation
Pipeline section to PROMPT_GALLERY.md.
Prove that adding one API key (OPENAI_API_KEY) to the zero-key foundation
produces dramatically better videos for ~$0.69 each. Two compositions built:
The Abyss (deep ocean visual essay) and VOID (neural interface product ad).
New tools:
- audio_probe: ffprobe wrapper with probe_duration() helper
- composition_validator: pre-render checks (asset existence, audio-video sync)
- pixabay_music: royalty-free music scraper (no API key needed)
- freesound_music: Freesound API search + download
Remotion upgrades:
- BackgroundImageLayer: AI images behind data scenes with ken-burns + dark overlay
- Gradient support: all 9 components changed from backgroundColor to background CSS
- CaptionOverlay: word spacing fix, WhisperX word-level subtitles
- HeroTitle: reduced overlay opacity so background images show through
Process codified in agent skills:
- compose-director: audio acquisition flow (present user with voice/music/subtitle
options), mandatory pre-render validation, post-render self-review (extract
frames + transcribe + inspect + present findings to user)
- scene-director: narration duration budgeting (word budget from video duration)
- remotion skill: pre-render validation section
- TTS tool: now returns audio_duration_seconds in result
README updated with VOID product ad video embed.
