execute() sends sampleCount=number_of_images and estimate_cost() bills
0.04 * n, but result handling decoded only predictions[0] and wrote it to
a single output_path. Images 2..n were dropped: never decoded, never
written, absent from artifacts. The user paid for n and received one.
The result also misreported the drop rather than failing loudly --
images_generated returned len(predictions) (what the API sent) while
artifacts held a single path, so an agent picking between variants read a
count that did not match the artifact list.
Add _output_paths() and loop over every prediction, mirroring the pattern
already used by openai_image and grok_image: suffix multi-image paths
_1/_2/... so none overwrite each other, keep the exact requested path when
n=1, return all paths in artifacts, and report images_generated as the
count actually written.
Closes#388
The CI Linux FFmpeg build carried the keyed frame forward without an alpha
plane, so overlay drew opaque green over the background (corner stayed green)
instead of compositing — the E2E test failed there even though it passed on
macOS/Windows.
Force `format=yuva420p` immediately after chromakey so the keyed transparency
always has an explicit alpha plane, and size the background to the frame up
front (color=...:size=WxH, passing the probed width/height into
_process_chromakey) instead of scaling a 1x1 source with scale2ref — dropping
scale2ref also removes the format negotiation that discarded the alpha on some
builds. Output is flattened to yuv420p after the overlay.
Add an Azure AI Speech transcription tool. It is opt-in: when
AZURE_SPEECH_KEY is configured the agent may prefer it for cloud STT,
while the local faster-whisper `transcriber` stays the default offline
path. Shared pipeline manifests are intentionally left unchanged, so no
default provider selection is altered for existing users.
- tools/analysis/azure_stt.py: new `azure_stt` tool (capability=analysis,
provider=azure) calling the Fast Transcription REST API. The local file
is uploaded via multipart and transcribed synchronously with word-level
timestamps and optional diarization — no Blob storage or async polling.
Output schema mirrors `transcriber` exactly, so it is a drop-in for
`subtitle_gen` and other transcript consumers. Follows the existing
provider-tool conventions (env-var status check, `_transcribe` helper,
cost_usd/model on the result, fallback="transcriber").
- Auto-discovered by the registry; no registry or selector changes.
- tests/tools/test_azure_stt.py: contract, discovery, status, response
mapping, execute guardrails, and a mocked-network success path (no live
API calls).
- .agents/skills + .claude/skills: azure-speech-to-text Layer-3 skill.
- docs/PROVIDERS.md: Azure AI Speech setup, API notes, and pricing.
- .env.example, skills/INDEX.md, AGENT_GUIDE.md: document the optional
cloud STT path alongside the default whisper transcriber.
_process_chromakey built the composite background from a 1x1 lavfi color source
and tried to size it with `[0:v]scale=iw:ih`. That scale is a no-op — iw/ih are
the 1x1 source's own dimensions, and there is no cross-reference to the frame.
FFmpeg's overlay then takes the size of its first input (the 1x1 background), so
every processed frame is clipped to a single pixel. The exception fallback never
runs because the primary command exits 0 (a valid 1x1 PNG), and
_reconstruct_video upscales those 1x1 frames — producing a solid-color video
with the keyed subject entirely gone. Total data loss for method="chromakey"
(and method="auto" when it selects chromakey).
Use scale2ref to resize the background to the actual frame dimensions before
overlaying, so the keyed subject is composited at full resolution.
Verified with ffmpeg: a 320x240 green frame with a red subject now produces a
320x240 output with the subject preserved and green replaced by the background,
instead of a 1x1 (then upscaled solid-color) frame.
Review findings from PR #333:
P1: _download_via_uri assumed output_video.uri is always files/<id>.
The API can return a full resource URI or a ready-made
.../files/<id>:download?alt=media download URL, which produced an
invalid poll path with a second :download appended. New
_file_id_from_uri() extracts the bare id from every documented shape;
regression tests cover the full-URL form plus a parametrized matrix of
URI shapes.
P2: docs/PROVIDERS.md still described the Google key as TTS + Imagen
only. The shared-key section now covers gemini_omni_video (model id,
~$0.10/sec pricing table, paid-tier-only, edit-turn billing note), and
the env snippet, provider-to-tool mapping, and capability coverage
tables include the new provider.
Add gemini_omni_video, a native Gemini API provider wrapping
gemini-omni-flash-preview via the Interactions API. Text-to-video,
image/reference-to-video with <FIRST_FRAME>/<IMAGE_REF_N> prompt tags,
and stateful edit_video turns via previous_interaction_id — the only
provider in the fleet that can refine a clip without regenerating it.
Reuses the existing GOOGLE_API_KEY / GEMINI_API_KEY, so one Google key
now unlocks images, TTS, and video.
- New Layer 3 skill .agents/skills/gemini-omni (prompting, edit-loop
rules, tag/timecode syntax, preview limits) sourced from official
Google docs; linked via agent_skills and the AGENT_GUIDE Layer 3 map
- ai-video-gen gains the Gemini API gateway row + editing pointer
- veo_video/sora_video fallback lists and video_selector agent_skills
reference the new provider; quality_score 0.85 with rationale
- Contract tests: registry discovery, selector routing, status from
env keys, uri + inline delivery, edit turns, typed image parts,
store=false editability, cost clamp
_ts_srt/_ts_vtt computed the seconds and millisecond fields independently:
`ms = int(round((seconds % 1) * 1000))`. When the fractional part is >= 0.9995
that rounds to 1000, emitting a malformed 4-digit `…,1000` value with no carry
into the seconds field (and, at 59.9999/3599.9999, no carry into minutes/hours).
For example 0.9999s became `00:00:00,1000` instead of `00:00:01,000`. ASR word
and segment end-times routinely land on such fractional boundaries, and the
resulting cue is rejected or mistimed by strict SRT/VTT parsers (ffmpeg
subtitles filter, VLC, browser WebVTT).
Decompose from a single rounded total-milliseconds value so the carry
propagates across all fields. Both formatters now share one `_hmsms` helper.
skills/creative/music-gen-usage.md mandates 'Always set
force_instrumental=true for video background', but music_gen.py never
sent the kwarg, so ElevenLabs could return vocal tracks that collide
with narration/dialogue.
- Add force_instrumental to input_schema (default True) so the mandate
holds by default; callers may opt out only with an explicit False.
- Include force_instrumental in the /v1/music payload.
- Add tests pinning: kwarg sent True by default, explicit opt-out
honored, and the schema default.
Refs: docs/REVIEW-image-to-video-voice.md §8 #8
Co-Authored-By: Claude <noreply@anthropic.com>
COGVIDEO_VARIANTS declares cogvideo-2b i2v=False (it is t2v-only), but
cogvideo_video advertised image_to_video + reference_image unconditionally
and the variant flag was never consulted. An image_to_video brief against
the 2B variant reached the diffusion pipeline and failed opaquely.
- Add is_operation_available(operation) that derives capability from the
variant table (the selector calls it without inputs, so it reports the
DEFAULT variant cogvideo-5b: t2v + i2v both True). This replaces an
implicit unconditional-True.
- Add an execute()-time guard that consults the CALLER's chosen variant
and fails fast with a clear error when it lacks the requested mode
(2B + image_to_video), instead of dropping into generate_local_video.
- Add _variant_for(inputs) helper shared by estimate_runtime / the guard.
Tests pin: the 2B premise (i2v=False), default-variant capability
reporting, fast-fail for 2B+i2v (generation never runs), and that 5B+i2v
still routes through to generate_local_video.
Refs: docs/REVIEW-image-to-video-voice.md §8 #4
Co-Authored-By: Claude <noreply@anthropic.com>
