diff --git a/.agents/skills/ai-video-gen/SKILL.md b/.agents/skills/ai-video-gen/SKILL.md index feb155ac..b801209b 100644 --- a/.agents/skills/ai-video-gen/SKILL.md +++ b/.agents/skills/ai-video-gen/SKILL.md @@ -1,7 +1,7 @@ --- name: ai-video-gen description: | - Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax). Supports two gateways: HeyGen API and fal.ai API. + Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax, Gemini Omni). Supports gateways: HeyGen API, fal.ai API, and the Gemini API (Gemini Omni Flash). allowed-tools: mcp__heygen__* metadata: openclaw: @@ -9,6 +9,8 @@ metadata: env_any: - HEYGEN_API_KEY - FAL_KEY + - GEMINI_API_KEY + - GOOGLE_API_KEY --- # Video Generation (Multi-Gateway) @@ -19,6 +21,9 @@ Generate AI videos from text prompts. Supports multiple providers via two API ga |---------|-------------|-----------|------| | **fal.ai** | `FAL_KEY` | **Seedance 2.0** (standard + fast), Kling v3/v2.1, MiniMax, VEO | `seedance_video`, `kling_video`, `minimax_video`, `veo_video` | | **HeyGen** | `HEYGEN_API_KEY` | VEO 3.1, Kling Pro, Sora v2, Runway Gen-4, Seedance Pro / Lite (1.x) | `heygen_video` | +| **Gemini API** | `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Gemini Omni Flash (generation + conversational editing) | `gemini_omni_video` | + +**Iterative editing — Gemini Omni.** When the brief calls for *refining an existing clip* (add/remove objects, restyle, change lighting or on-screen text) rather than regenerating, Gemini Omni Flash is the only provider in the fleet with stateful multi-turn editing. See Layer 3 `gemini-omni` for the authoritative prompting guide (reference-image tags, timecode syntax, edit-prompt rules) before writing any prompt for it. **Preferred premium default — Seedance 2.0.** When any premium gateway is configured (`FAL_KEY` → `seedance_video`, or HeyGen's Video Agent / Avatar Shots path), Seedance 2.0 is the preferred default for cinematic, trailer, and high-fidelity clip work. It is the only model in the fleet with **single-pass native synchronized audio, multi-shot generation, director-level camera control, and lip-sync from quoted dialogue**, and it ranks #1 on Artificial Analysis Elo as of early 2026. Switch off it only when the user has a specific reason (budget, provider preference, stylistic fit like VEO for photoreal landscape or Kling for specific anime look). See Layer 3 `seedance-2-0` for the authoritative prompting and parameter guide. diff --git a/.agents/skills/gemini-omni/SKILL.md b/.agents/skills/gemini-omni/SKILL.md new file mode 100644 index 00000000..1587c66b --- /dev/null +++ b/.agents/skills/gemini-omni/SKILL.md @@ -0,0 +1,113 @@ +--- +name: gemini-omni +description: | + Generate and conversationally edit short videos with Google Gemini Omni Flash (`gemini-omni-flash-preview`). Use when: (1) iterating on a clip with natural-language edits instead of regenerating ("make the phone invisible, keep everything else the same"), (2) generating 3-10s 720p clips with synthesized audio, rendered on-screen text, or timecoded beats, (3) binding reference images to roles with / prompt tags, (4) editing an existing uploaded video. Accessed via the `gemini_omni_video` tool using the project's GEMINI_API_KEY/GOOGLE_API_KEY — the same key as Imagen and Google TTS. +allowed-tools: Bash, Read, Write +metadata: + openclaw: + requires: + env_any: + - GEMINI_API_KEY + - GOOGLE_API_KEY +--- + +# Gemini Omni Flash (Google DeepMind) + +Gemini Omni is Google DeepMind's video generation **and editing** model family, announced at I/O 2026. The first model, **Gemini Omni Flash** (`gemini-omni-flash-preview`, developer access since June 30, 2026), generates 3-10 second clips at 720p/24fps with synthesized audio via the Gemini **Interactions API**. Its differentiator in the OpenMontage fleet is **stateful conversational editing**: each generation returns an `interaction_id`, and a follow-up call with `previous_interaction_id` edits that video in place — no other wrapped provider can refine a clip without regenerating it. + +OpenMontage wraps it as `gemini_omni_video` (native Gemini API, no gateway). It shares `GOOGLE_API_KEY`/`GEMINI_API_KEY` with `google_imagen` and `google_tts` — one key, three capabilities. Paid tier only: ~$0.10 per second of output video (billed as 5,792 output tokens/sec at $17.50/1M). + +## When to pick it (and when not) + +| Use it for | Prefer another provider for | +|---|---| +| Iterative refinement — generate, review, then edit the same clip in layers | One-shot cinematic hero clips (→ Seedance 2.0, see `seedance-2-0`) | +| Editing an existing/uploaded clip (restyle, add/remove objects, change text) | Clips longer than 10s or above 720p | +| On-screen rendered text and word-by-word text beats | Seed-reproducible generations (no seed support) | +| Reference-image-bound subjects/styles via prompt tags | First/last-frame interpolation (→ `veo_video`) | +| Timecode-scheduled multi-beat clips from one prompt | Non-English narration (English only fully supported) | + +Route through `video_selector` for generation operations. **Editing (`edit_video`) is a direct-tool operation** — call `gemini_omni_video` from the registry, because the multi-turn interaction state lives outside the selector's model. + +## Generation prompting + +Describe **scene + camera + lighting + motion + audio**. Official example: + +> Continuous, unbroken handheld shot of a fluffy tabby cat sitting on a sunny windowsill, looking out into a leafy garden. The cat's tail twitches slowly, and its ears rotate slightly toward ambient noises. Sunbeams illuminate dust motes in the air. + +- **Force a single shot** explicitly: "In a single continuous shot," / "No scene cuts." Otherwise the model may cut between scenes. +- **Negatives go in prose** — there is no `negative_prompt` parameter: "No dialogue," "No extra sound effects." +- **No sampler controls**: system instructions, temperature, top_p, and seeds are all unsupported. The prompt is the only lever. +- **Meta-prompt for quality**: "Consider micro-detail, expression and timing to create a very rich, detailed but entirely natural scene." + +### Timecode syntax + +Schedule beats with bracketed ranges or natural language — this maps directly onto OpenMontage scene-plan timings: + +``` +[0-3s] A person is walking [3-6s] They stop and turn around +``` + +> "After 3 seconds, a woman enters the scene." / "At 5s the chorus starts in the background audio." + +### Audio and on-screen text + +Audio is synthesized automatically; direct it in the prompt: "Include calm background music," "The audio is a low tinny radio broadcast in the background." Rendered text works and can be timed: + +> One word on the screen at a time: 'did, you, know, that, Omni, can, do, awesome, text?' Each word appears for 1s. + +## Reference images (`` / `` tags) + +Pass local images via `reference_image_paths` (they are sent in order), then bind them to roles **inside the prompt** with tags. `` indexes from 0 in the order supplied: + +``` +in the style of a woman is walking +``` + +``` +[0-3s] A studio fashion sequence. Starting with woman , she is +holding [3-6s] Then we see the man holding +``` + +- `` makes an image the opening frame: ` a woman is walking`. +- Use high-resolution images; describe the intended motion specifically rather than "make it move." +- Say what each image *is* (product / character / style / background reference) — the model decides usage from context. + +## Conversational editing (the differentiator) + +**Editing prompts are the opposite of generation prompts: short and surgical.** Overly descriptive edit prompts cause unintended changes. + +1. Generate the base clip (subject + scene + motion). The tool returns `interaction_id` in its result data. +2. Pass it back as `previous_interaction_id` with `operation="edit_video"` and describe **only the delta**. +3. Append **"Keep everything else the same."** to pin unmentioned elements. +4. Refine in layers — one turn for lighting, one for camera, one for action, one for audio. + +Official good/bad pairs: + +| Avoid | Instead | +|---|---| +| "In the video of the man sitting on the sofa, please add a small black cat..." | "Add a cat that jumps onto his lap, he begins to pet it. Keep everything else the same." | +| "Please remove the cell phone... and fill in the background so it looks like..." | "Make the phone invisible. Keep everything else the same." | + +Other working edit prompts: "Make this video anime" / "Put a fashionable hat on this person" / "Change the lighting to be more dramatic" / "Change the text on the sign to say 'Omni Flash'". + +**Gotcha — `store`:** editing via `previous_interaction_id` only works if the *prior* call kept the interaction server-side (`store` defaults to true in `gemini_omni_video`). Set `store=false` only for one-shot generations you will never edit. + +**Editing uploaded videos:** pass `input_video_path` instead of `previous_interaction_id`; the tool uploads it via the Files API. Unavailable in the EEA, Switzerland, and the UK (editing *generated* videos works everywhere). + +## Hard limitations (preview) + +- Output: 3-10s, 720p, 24fps, MP4 with audio; aspect ratio `16:9` or `9:16`. All output carries an invisible SynthID watermark. +- No seed, negative prompt, temperature, top_p, or system instructions. +- No video extension or first/last-frame interpolation; no voice editing. +- Audio reference inputs unsupported. Video references ≤3s are accepted by the schema but **not processed correctly** — don't rely on them. +- Multi-video prompting unsupported; may degrade output. +- English fully supported; other languages untested. +- Images of minors (EEA/CH/UK) and certain recognizable people are blocked for upload/editing. + +## Sources + +- Generation & editing guide: https://ai.google.dev/gemini-api/docs/omni +- Model card: https://ai.google.dev/gemini-api/docs/models/gemini-omni-flash +- Pricing: https://ai.google.dev/gemini-api/docs/pricing +- Announcement: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-omni/ diff --git a/.claude/skills/ai-video-gen/SKILL.md b/.claude/skills/ai-video-gen/SKILL.md index feb155ac..b801209b 100644 --- a/.claude/skills/ai-video-gen/SKILL.md +++ b/.claude/skills/ai-video-gen/SKILL.md @@ -1,7 +1,7 @@ --- name: ai-video-gen description: | - Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax). Supports two gateways: HeyGen API and fal.ai API. + Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax, Gemini Omni). Supports gateways: HeyGen API, fal.ai API, and the Gemini API (Gemini Omni Flash). allowed-tools: mcp__heygen__* metadata: openclaw: @@ -9,6 +9,8 @@ metadata: env_any: - HEYGEN_API_KEY - FAL_KEY + - GEMINI_API_KEY + - GOOGLE_API_KEY --- # Video Generation (Multi-Gateway) @@ -19,6 +21,9 @@ Generate AI videos from text prompts. Supports multiple providers via two API ga |---------|-------------|-----------|------| | **fal.ai** | `FAL_KEY` | **Seedance 2.0** (standard + fast), Kling v3/v2.1, MiniMax, VEO | `seedance_video`, `kling_video`, `minimax_video`, `veo_video` | | **HeyGen** | `HEYGEN_API_KEY` | VEO 3.1, Kling Pro, Sora v2, Runway Gen-4, Seedance Pro / Lite (1.x) | `heygen_video` | +| **Gemini API** | `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Gemini Omni Flash (generation + conversational editing) | `gemini_omni_video` | + +**Iterative editing — Gemini Omni.** When the brief calls for *refining an existing clip* (add/remove objects, restyle, change lighting or on-screen text) rather than regenerating, Gemini Omni Flash is the only provider in the fleet with stateful multi-turn editing. See Layer 3 `gemini-omni` for the authoritative prompting guide (reference-image tags, timecode syntax, edit-prompt rules) before writing any prompt for it. **Preferred premium default — Seedance 2.0.** When any premium gateway is configured (`FAL_KEY` → `seedance_video`, or HeyGen's Video Agent / Avatar Shots path), Seedance 2.0 is the preferred default for cinematic, trailer, and high-fidelity clip work. It is the only model in the fleet with **single-pass native synchronized audio, multi-shot generation, director-level camera control, and lip-sync from quoted dialogue**, and it ranks #1 on Artificial Analysis Elo as of early 2026. Switch off it only when the user has a specific reason (budget, provider preference, stylistic fit like VEO for photoreal landscape or Kling for specific anime look). See Layer 3 `seedance-2-0` for the authoritative prompting and parameter guide. diff --git a/.env.example b/.env.example index dad70908..ac6f0708 100644 --- a/.env.example +++ b/.env.example @@ -16,9 +16,11 @@ HIGGSFIELD_API_KEY= # Higgsfield Cloud key (higgsfield_video). Pair wit HIGGSFIELD_API_SECRET= # or use the combined HIGGSFIELD_KEY=":" form instead. # HIGGSFIELD_KEY= # Combined key:secret — set this INSTEAD of the _KEY/_SECRET pair if you prefer. -# --- Google (one key unlocks image gen + TTS) --- -GOOGLE_API_KEY= # Google Imagen images, Google Cloud TTS (700+ voices, 50+ languages) +# --- Google (one key unlocks image gen + TTS + video) --- +GOOGLE_API_KEY= # Google Imagen images, Google Cloud TTS (700+ voices, 50+ languages), + # Gemini Omni video (generation + conversational editing, paid tier) # Get one at https://aistudio.google.com/apikey +# GEMINI_API_KEY= # Alias for GOOGLE_API_KEY (takes precedence when both are set) # Alternative to the API key: service-account JSON auth. # TTS uses Cloud Text-to-Speech; Imagen routes to Vertex AI. GOOGLE_APPLICATION_CREDENTIALS= # path to a service-account JSON key file diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md index 1abfd8e5..feb67699 100644 --- a/AGENT_GUIDE.md +++ b/AGENT_GUIDE.md @@ -676,7 +676,7 @@ The `.agents/skills/` directory is large. When you're not coming in through a to | **Animation knowledge (generic)** | `gsap-core`, `gsap-timeline`, `gsap-plugins` (SplitText / MorphSVG / DrawSVG / MotionPath / Flip / CustomEase), `gsap-utils`, `gsap-react`, `gsap-performance`, `gsap-scrolltrigger`, `gsap-frameworks`, `framer-motion` (Disney 12 principles), `lottie-bodymovin` (Lottie export) | | **Character animation** | `character-rigging`, `svg-character-animation`, `pose-library-design`, `canvas-procedural-animation`, `character-animation-qa` | | **Image generation** | `bfl-api`, `flux-best-practices` | -| **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `ai-video-gen`, `ltx2` | +| **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` | | **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` | | **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `speech-to-text`, `agents` | | **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) | diff --git a/tests/tools/test_gemini_omni_video.py b/tests/tools/test_gemini_omni_video.py new file mode 100644 index 00000000..682fcad7 --- /dev/null +++ b/tests/tools/test_gemini_omni_video.py @@ -0,0 +1,244 @@ +"""Regression coverage for first-class Gemini Omni provider discovery and execution.""" + +from __future__ import annotations + +import base64 +import json +import sys +import types +from pathlib import Path + +import pytest + +from tools.base_tool import ToolStatus + + +class FakeResponse: + def __init__(self, json_data=None, content=b"", ok=True, status_code=200, headers=None, text=""): + self._json = json_data + self.content = content + self.ok = ok + self.status_code = status_code + self.headers = headers or {} + self.text = text or (json.dumps(json_data) if json_data is not None else "") + + def json(self): + return self._json + + def raise_for_status(self): + if not self.ok: + raise RuntimeError(f"HTTP {self.status_code}") + + +def _install_fake_requests(monkeypatch, post_responses, get_responses): + """Inject a fake requests module; returns the recorded calls.""" + calls = {"post": [], "get": []} + + fake = types.ModuleType("requests") + + def fake_post(url, headers=None, json=None, data=None, timeout=None, params=None): + calls["post"].append({"url": url, "headers": headers, "json": json, "data": data}) + return post_responses.pop(0) + + def fake_get(url, headers=None, timeout=None, params=None): + calls["get"].append({"url": url, "headers": headers, "params": params}) + return get_responses.pop(0) + + fake.post = fake_post + fake.get = fake_get + monkeypatch.setitem(sys.modules, "requests", fake) + return calls + + +@pytest.fixture() +def gemini_env(monkeypatch): + monkeypatch.delenv("GOOGLE_API_KEY", raising=False) + monkeypatch.setenv("GEMINI_API_KEY", "test-gemini-key") + + +def test_gemini_omni_is_discovered_as_video_provider(): + from tools.tool_registry import ToolRegistry + + registry = ToolRegistry() + registry.discover() + + tool = registry.get("gemini_omni_video") + assert tool is not None + assert tool.provider == "gemini_omni" + assert tool.capability == "video_generation" + # Ranking + selector-filter contract: without these the scorer buries the + # editing capability and the selector drops it from image_to_video routing. + assert tool.quality_score == 0.85 + assert tool.supports["image_to_video"] is True + assert tool.supports["reference_to_video"] is True + assert tool.supports["conversational_editing"] is True + assert "gemini-omni" in tool.agent_skills + + +def test_gemini_omni_is_routed_by_video_selector(): + from tools.video.video_selector import VideoSelector + + provider_names = [t.name for t in VideoSelector()._providers()] + assert "gemini_omni_video" in provider_names + + +def test_gemini_omni_status_tracks_google_api_keys(monkeypatch): + from tools.video.gemini_omni_video import GeminiOmniVideo + + monkeypatch.delenv("GEMINI_API_KEY", raising=False) + monkeypatch.delenv("GOOGLE_API_KEY", raising=False) + assert GeminiOmniVideo().get_status() == ToolStatus.UNAVAILABLE + + monkeypatch.setenv("GOOGLE_API_KEY", "test-google-key") + assert GeminiOmniVideo().get_status() == ToolStatus.AVAILABLE + + +def test_gemini_omni_cost_estimate_clamps_duration_hint(gemini_env): + from tools.video.gemini_omni_video import GeminiOmniVideo + + tool = GeminiOmniVideo() + assert tool.estimate_cost({"prompt": "x"}) == pytest.approx(0.80) + assert tool.estimate_cost({"prompt": "x", "duration": "5s"}) == pytest.approx(0.50) + assert tool.estimate_cost({"prompt": "x", "duration": "30"}) == pytest.approx(1.00) + + +def test_gemini_omni_text_to_video_via_uri_delivery(monkeypatch, tmp_path, gemini_env): + from tools.video.gemini_omni_video import GeminiOmniVideo + + calls = _install_fake_requests( + monkeypatch, + post_responses=[ + FakeResponse({"id": "int_123", "output_video": {"uri": "files/vid-123"}}), + ], + get_responses=[ + FakeResponse({"state": "ACTIVE"}), + FakeResponse(content=b"fake omni mp4"), + ], + ) + + output_path = tmp_path / "clip.mp4" + result = GeminiOmniVideo().execute( + { + "prompt": "A marble rolling on a track, single continuous shot.", + "aspect_ratio": "9:16", + "output_path": str(output_path), + } + ) + + assert result.success, result.error + assert output_path.read_bytes() == b"fake omni mp4" + assert result.data["interaction_id"] == "int_123" + assert result.data["editable"] is True + + payload = calls["post"][0]["json"] + assert payload["model"] == "gemini-omni-flash-preview" + assert payload["input"] == "A marble rolling on a track, single continuous shot." + assert payload["response_format"] == {"type": "video", "aspect_ratio": "9:16", "delivery": "uri"} + assert calls["post"][0]["headers"]["x-goog-api-key"] == "test-gemini-key" + assert calls["get"][1]["url"].endswith("files/vid-123:download") + assert calls["get"][1]["params"] == {"alt": "media"} + + +def test_gemini_omni_inline_data_response_is_handled(monkeypatch, tmp_path, gemini_env): + from tools.video.gemini_omni_video import GeminiOmniVideo + + inline = base64.b64encode(b"inline mp4").decode("ascii") + calls = _install_fake_requests( + monkeypatch, + post_responses=[FakeResponse({"id": "int_9", "output_video": {"data": inline}})], + get_responses=[], + ) + + output_path = tmp_path / "inline.mp4" + result = GeminiOmniVideo().execute({"prompt": "A sunset.", "output_path": str(output_path)}) + + assert result.success, result.error + assert output_path.read_bytes() == b"inline mp4" + assert calls["get"] == [] + + +def test_gemini_omni_edit_turn_sends_previous_interaction_id(monkeypatch, tmp_path, gemini_env): + from tools.video.gemini_omni_video import GeminiOmniVideo + + inline = base64.b64encode(b"edited mp4").decode("ascii") + calls = _install_fake_requests( + monkeypatch, + post_responses=[FakeResponse({"id": "int_2", "output_video": {"data": inline}})], + get_responses=[], + ) + + result = GeminiOmniVideo().execute( + { + "prompt": "Make the violin invisible. Keep everything else the same.", + "operation": "edit_video", + "previous_interaction_id": "int_1", + "output_path": str(tmp_path / "edit.mp4"), + } + ) + + assert result.success, result.error + assert calls["post"][0]["json"]["previous_interaction_id"] == "int_1" + + +def test_gemini_omni_edit_without_source_is_rejected(gemini_env): + from tools.video.gemini_omni_video import GeminiOmniVideo + + result = GeminiOmniVideo().execute({"prompt": "Make it anime", "operation": "edit_video"}) + assert not result.success + assert "previous_interaction_id" in result.error + + +def test_gemini_omni_image_to_video_sends_typed_parts(monkeypatch, tmp_path, gemini_env): + from tools.video.gemini_omni_video import GeminiOmniVideo + + ref = tmp_path / "cat.png" + ref.write_bytes(b"png bytes") + inline = base64.b64encode(b"cat mp4").decode("ascii") + calls = _install_fake_requests( + monkeypatch, + post_responses=[FakeResponse({"id": "int_3", "output_video": {"data": inline}})], + get_responses=[], + ) + + result = GeminiOmniVideo().execute( + { + "prompt": "A cat playfully batting at yarn.", + "operation": "image_to_video", + "reference_image_path": str(ref), + "output_path": str(tmp_path / "cat.mp4"), + } + ) + + assert result.success, result.error + parts = calls["post"][0]["json"]["input"] + assert parts[0]["type"] == "image" + assert parts[0]["mime_type"] == "image/png" + assert parts[0]["data"] == base64.b64encode(b"png bytes").decode("ascii") + assert parts[-1] == {"type": "text", "text": "A cat playfully batting at yarn."} + + +def test_gemini_omni_image_to_video_requires_reference(gemini_env): + from tools.video.gemini_omni_video import GeminiOmniVideo + + result = GeminiOmniVideo().execute({"prompt": "x", "operation": "image_to_video"}) + assert not result.success + assert "reference_image_path" in result.error + + +def test_gemini_omni_store_false_marks_result_not_editable(monkeypatch, tmp_path, gemini_env): + from tools.video.gemini_omni_video import GeminiOmniVideo + + inline = base64.b64encode(b"oneshot mp4").decode("ascii") + calls = _install_fake_requests( + monkeypatch, + post_responses=[FakeResponse({"id": "int_4", "output_video": {"data": inline}})], + get_responses=[], + ) + + result = GeminiOmniVideo().execute( + {"prompt": "A sunset.", "store": False, "output_path": str(tmp_path / "s.mp4")} + ) + + assert result.success, result.error + assert result.data["editable"] is False + assert calls["post"][0]["json"]["store"] is False diff --git a/tools/video/gemini_omni_video.py b/tools/video/gemini_omni_video.py new file mode 100644 index 00000000..d469c517 --- /dev/null +++ b/tools/video/gemini_omni_video.py @@ -0,0 +1,437 @@ +"""Google Gemini Omni Flash video generation and conversational editing. + +Calls the Gemini Interactions API (``POST /v1beta/interactions``) directly with +the project's Google API key — the same key that unlocks Imagen images and +Cloud TTS. Gemini Omni Flash generates 3-10 second 720p/24fps clips with +synthesized audio, and is the only provider in the fleet with stateful +conversational editing: pass ``previous_interaction_id`` and describe only the +delta ("Make the violin invisible. Keep everything else the same."). + +Reference images bind to roles via inline prompt tags (````, +````) and beats can be scheduled with timecode syntax +(``[0-3s] ... [3-6s] ...``). See the Layer 3 skill ``gemini-omni`` for the +authoritative prompting guide — read it before writing prompts. +""" + +from __future__ import annotations + +import base64 +import mimetypes +import os +import time +from pathlib import Path +from typing import Any + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ResourceProfile, + RetryPolicy, + ToolResult, + ToolRuntime, + ToolStability, + ToolStatus, + ToolTier, +) + +_BASE_URL = "https://generativelanguage.googleapis.com/v1beta" +_UPLOAD_URL = "https://generativelanguage.googleapis.com/upload/v1beta/files" +_DEFAULT_MODEL = "gemini-omni-flash-preview" +# Billed at 5,792 output tokens per second of 720p video, $17.50/1M tokens +# (ai.google.dev/gemini-api/docs/pricing) — effectively ~$0.10 per second. +_COST_PER_SECOND = 0.10 +_DEFAULT_DURATION_SECONDS = 8 +_POLL_INTERVAL_SECONDS = 5 +_MAX_POLL_SECONDS = 900 + + +class GeminiOmniVideo(BaseTool): + name = "gemini_omni_video" + version = "0.1.0" + tier = ToolTier.GENERATE + capability = "video_generation" + provider = "gemini_omni" + stability = ToolStability.EXPERIMENTAL + execution_mode = ExecutionMode.SYNC + determinism = Determinism.STOCHASTIC + runtime = ToolRuntime.API + + dependencies = [] + install_instructions = ( + "Set GEMINI_API_KEY or GOOGLE_API_KEY to a Google AI Studio API key.\n" + " Get one at https://aistudio.google.com/apikey\n" + " Gemini Omni Flash is paid-tier only (no free tier); ~$0.10 per second of video." + ) + agent_skills = ["gemini-omni", "ai-video-gen"] + + capabilities = ["text_to_video", "image_to_video", "reference_to_video", "edit_video"] + supports = { + "text_to_video": True, + "image_to_video": True, + "reference_to_video": True, + "edit_video": True, + "conversational_editing": True, + "native_audio": True, + "text_rendering": True, + "timecode_control": True, + # Preview limitations — no sampler controls of any kind. + "seed": False, + "negative_prompt": False, + "first_last_frame_to_video": False, + } + best_for = [ + "iterative natural-language video editing (edit a clip without regenerating it)", + "reference-image-driven clips via / prompt tags", + "fast 3-10s clips with synced audio, rendered text, and timecoded beats from one Google key", + ] + not_good_for = [ + "clips longer than 10 seconds or above 720p", + "seed-reproducible output or negative-prompt control", + "offline generation", + ] + fallback_tools = ["veo_video", "sora_video", "kling_video", "minimax_video"] + # Conversational editing + native audio are unique in the fleet, but preview + # output is capped at 720p/10s — below seedance (0.95) and grok/runway (0.9) + # on raw generation fidelity. Without a quality_score the scorer would only + # count supports/stability flags and bury the editing capability entirely. + # See lib/scoring.py. + quality_score = 0.85 + + input_schema = { + "type": "object", + "required": ["prompt"], + "properties": { + "prompt": { + "type": "string", + "description": ( + "Video description, or for edit_video the change to apply. " + "Supports / tags and [0-3s] timecodes — " + "see the gemini-omni skill." + ), + }, + "operation": { + "type": "string", + "enum": ["text_to_video", "image_to_video", "reference_to_video", "edit_video"], + "default": "text_to_video", + }, + "aspect_ratio": { + "type": "string", + "enum": ["16:9", "9:16"], + "default": "16:9", + }, + "duration": { + "type": "string", + "description": ( + "Duration hint in seconds (3-10). The model chooses the actual length; " + "this only shapes the prompt-independent cost estimate." + ), + }, + "reference_image_path": { + "type": "string", + "description": "Local reference image (jpg/png) for image_to_video.", + }, + "reference_image_paths": { + "type": "array", + "items": {"type": "string"}, + "description": "Local reference images, bound in the prompt as , , ...", + }, + "previous_interaction_id": { + "type": "string", + "description": ( + "Interaction id from a prior gemini_omni_video result — edits that video " + "in place (edit_video). Requires the prior call to have used store=true." + ), + }, + "input_video_path": { + "type": "string", + "description": ( + "Local video to edit (edit_video). Uploaded via the Files API. " + "Editing uploaded videos is unavailable in the EEA, Switzerland, and the UK." + ), + }, + "store": { + "type": "boolean", + "default": True, + "description": ( + "Keep the interaction server-side so the result can be edited in later turns " + "via previous_interaction_id. Set false only for one-shot generations." + ), + }, + "output_path": {"type": "string"}, + }, + } + + resource_profile = ResourceProfile( + cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True + ) + retry_policy = RetryPolicy(max_retries=1, retryable_errors=["rate_limit", "timeout"]) + idempotency_key_fields = ["prompt", "operation", "aspect_ratio", "previous_interaction_id"] + side_effects = [ + "writes video file to output_path", + "calls the Gemini Interactions API", + "stores the interaction server-side when store=true (enables later edits)", + ] + user_visible_verification = [ + "Watch generated clip for visual quality, motion, and prompt adherence", + "Listen for synthesized audio quality and any requested dialogue/music", + "After an edit turn, confirm unmentioned elements were preserved", + ] + + @staticmethod + def _get_api_key() -> str | None: + return os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY") + + def get_status(self) -> ToolStatus: + if self._get_api_key(): + return ToolStatus.AVAILABLE + return ToolStatus.UNAVAILABLE + + @staticmethod + def _duration_hint(inputs: dict[str, Any]) -> int: + raw = str(inputs.get("duration") or _DEFAULT_DURATION_SECONDS).strip().lower() + raw = raw[:-1] if raw.endswith("s") else raw + try: + seconds = int(float(raw)) + except ValueError: + seconds = _DEFAULT_DURATION_SECONDS + return max(3, min(10, seconds)) + + def estimate_cost(self, inputs: dict[str, Any]) -> float: + return _COST_PER_SECOND * self._duration_hint(inputs) + + def estimate_runtime(self, inputs: dict[str, Any]) -> float: + return 180.0 + + @staticmethod + def _image_part(path_str: str) -> dict[str, Any]: + path = Path(path_str) + if not path.exists(): + raise FileNotFoundError(f"Reference image not found: {path}") + mime_type, _ = mimetypes.guess_type(path.name) + if not mime_type or not mime_type.startswith("image/"): + mime_type = "image/png" + return { + "type": "image", + "data": base64.b64encode(path.read_bytes()).decode("ascii"), + "mime_type": mime_type, + } + + def _upload_video_file(self, requests_mod: Any, api_key: str, path_str: str) -> str: + """Upload a local video via the Files API (resumable) and return its URI.""" + path = Path(path_str) + if not path.exists(): + raise FileNotFoundError(f"Input video not found: {path}") + mime_type, _ = mimetypes.guess_type(path.name) + if not mime_type or not mime_type.startswith("video/"): + mime_type = "video/mp4" + video_bytes = path.read_bytes() + + start_resp = requests_mod.post( + _UPLOAD_URL, + headers={ + "x-goog-api-key": api_key, + "X-Goog-Upload-Protocol": "resumable", + "X-Goog-Upload-Command": "start", + "X-Goog-Upload-Header-Content-Length": str(len(video_bytes)), + "X-Goog-Upload-Header-Content-Type": mime_type, + "Content-Type": "application/json", + }, + json={"file": {"display_name": path.name}}, + timeout=30, + ) + start_resp.raise_for_status() + upload_url = start_resp.headers.get("X-Goog-Upload-URL") + if not upload_url: + raise RuntimeError("Files API did not return an upload URL") + + upload_resp = requests_mod.post( + upload_url, + headers={ + "X-Goog-Upload-Command": "upload, finalize", + "X-Goog-Upload-Offset": "0", + "Content-Length": str(len(video_bytes)), + }, + data=video_bytes, + timeout=300, + ) + upload_resp.raise_for_status() + file_info = upload_resp.json().get("file", {}) + + # Wait until the uploaded video is processed before referencing it. + deadline = time.time() + _MAX_POLL_SECONDS + while str(file_info.get("state", "")).upper() == "PROCESSING": + if time.time() > deadline: + raise TimeoutError("Uploaded video did not finish processing in time") + time.sleep(_POLL_INTERVAL_SECONDS) + status_resp = requests_mod.get( + f"{_BASE_URL}/{file_info.get('name')}", + headers={"x-goog-api-key": api_key}, + timeout=15, + ) + status_resp.raise_for_status() + file_info = status_resp.json() + if str(file_info.get("state", "")).upper() == "FAILED": + raise RuntimeError("Files API failed to process the uploaded video") + + uri = file_info.get("uri") + if not uri: + raise RuntimeError(f"Files API response missing uri: {file_info}") + return uri + + @staticmethod + def _extract_output_video(data: dict[str, Any]) -> dict[str, Any] | None: + """Find the output video payload ({'data': b64} or {'uri': files/...}).""" + for key in ("output_video", "outputVideo"): + video = data.get(key) + if isinstance(video, dict) and (video.get("data") or video.get("uri")): + return video + # REST responses may also carry the video inside steps[].content[]. + for step in data.get("steps") or []: + for item in step.get("content") or []: + if isinstance(item, dict) and (item.get("data") or item.get("uri")): + if "video" in str(item.get("type", "")).lower() or item.get("mime_type", "").startswith("video/"): + return item + if item.get("data") or str(item.get("uri", "")).startswith("files/"): + return item + return None + + def _download_via_uri(self, requests_mod: Any, api_key: str, uri: str) -> bytes: + """Poll a Files API entry until ACTIVE, then download its bytes.""" + file_id = uri.rstrip("/").split("/")[-1] + headers = {"x-goog-api-key": api_key} + deadline = time.time() + _MAX_POLL_SECONDS + while True: + status_resp = requests_mod.get( + f"{_BASE_URL}/files/{file_id}", headers=headers, timeout=15 + ) + status_resp.raise_for_status() + state = str(status_resp.json().get("state", "")).upper() + if state == "ACTIVE": + break + if state == "FAILED": + raise RuntimeError("Gemini Omni video generation failed during processing") + if time.time() > deadline: + raise TimeoutError("Timed out waiting for Gemini Omni video to become ACTIVE") + time.sleep(_POLL_INTERVAL_SECONDS) + + download_resp = requests_mod.get( + f"{_BASE_URL}/files/{file_id}:download", + params={"alt": "media"}, + headers=headers, + timeout=300, + ) + download_resp.raise_for_status() + return download_resp.content + + def execute(self, inputs: dict[str, Any]) -> ToolResult: + api_key = self._get_api_key() + if not api_key: + return ToolResult( + success=False, + error="GEMINI_API_KEY / GOOGLE_API_KEY not set. " + self.install_instructions, + ) + + import requests + + start = time.time() + operation = inputs.get("operation", "text_to_video") + prompt = str(inputs["prompt"]).strip() + aspect_ratio = inputs.get("aspect_ratio", "16:9") + previous_interaction_id = inputs.get("previous_interaction_id") + + if operation == "edit_video" and not previous_interaction_id and not inputs.get("input_video_path"): + return ToolResult( + success=False, + error="edit_video requires previous_interaction_id (edit a generated clip) or input_video_path (edit an uploaded clip)", + ) + + reference_paths = list(inputs.get("reference_image_paths") or []) + if inputs.get("reference_image_path"): + reference_paths.insert(0, inputs["reference_image_path"]) + if operation in {"image_to_video", "reference_to_video"} and not reference_paths: + return ToolResult( + success=False, + error=f"{operation} requires reference_image_path or reference_image_paths", + ) + + try: + parts: list[dict[str, Any]] = [self._image_part(p) for p in reference_paths] + if inputs.get("input_video_path"): + video_uri = self._upload_video_file(requests, api_key, inputs["input_video_path"]) + parts.append({"type": "document", "uri": video_uri}) + except Exception as e: + return ToolResult(success=False, error=f"Gemini Omni input preparation failed: {e}") + + payload: dict[str, Any] = { + "model": _DEFAULT_MODEL, + # Plain string for text-only turns (the documented minimal form), + # typed parts when images or an uploaded video ride along. + "input": prompt if not parts else parts + [{"type": "text", "text": prompt}], + # uri delivery avoids the ~4MB inline-payload ceiling; inline data in + # the response is still handled below if the API returns it anyway. + "response_format": { + "type": "video", + "aspect_ratio": aspect_ratio, + "delivery": "uri", + }, + } + if previous_interaction_id: + payload["previous_interaction_id"] = previous_interaction_id + if inputs.get("store") is False: + payload["store"] = False + + try: + resp = requests.post( + f"{_BASE_URL}/interactions", + headers={"x-goog-api-key": api_key, "Content-Type": "application/json"}, + json=payload, + timeout=600, + ) + if not resp.ok: + detail = resp.text[:1000] + return ToolResult( + success=False, + error=f"Gemini Omni interaction failed ({resp.status_code}): {detail}", + ) + data = resp.json() + + interaction_id = data.get("id") + video = self._extract_output_video(data) + if not video: + return ToolResult( + success=False, + error=f"Gemini Omni response did not include an output video: {str(data)[:1000]}", + ) + + if video.get("data"): + video_bytes = base64.b64decode(video["data"]) + else: + video_bytes = self._download_via_uri(requests, api_key, str(video["uri"])) + + output_path = Path(inputs.get("output_path", "gemini_omni_output.mp4")) + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_bytes(video_bytes) + except Exception as e: + return ToolResult(success=False, error=f"Gemini Omni video generation failed: {e}") + + editable = inputs.get("store") is not False + return ToolResult( + success=True, + data={ + "provider": self.provider, + "model": _DEFAULT_MODEL, + "prompt": prompt, + "operation": operation, + "output": str(output_path), + "aspect_ratio": aspect_ratio, + "has_audio": True, + # Feed this back as previous_interaction_id to edit this clip. + "interaction_id": interaction_id, + "editable": editable, + }, + artifacts=[str(output_path)], + cost_usd=self.estimate_cost(inputs), + duration_seconds=round(time.time() - start, 2), + model=_DEFAULT_MODEL, + ) diff --git a/tools/video/sora_video.py b/tools/video/sora_video.py index c2d4a5aa..6c9060c6 100644 --- a/tools/video/sora_video.py +++ b/tools/video/sora_video.py @@ -65,7 +65,7 @@ class SoraVideo(BaseTool): "4, 8, or 12 second social-video clips that OpenMontage can stitch and compose", ] not_good_for = ["offline generation", "long continuous scenes", "projects without Sora API access"] - fallback_tools = ["veo_video", "seedance_video", "kling_video", "minimax_video"] + fallback_tools = ["veo_video", "gemini_omni_video", "seedance_video", "kling_video", "minimax_video"] input_schema = { "type": "object", diff --git a/tools/video/veo_video.py b/tools/video/veo_video.py index 522128d5..33b177a6 100644 --- a/tools/video/veo_video.py +++ b/tools/video/veo_video.py @@ -62,7 +62,7 @@ class VeoVideo(BaseTool): "ambient sound and music generation built in", ] not_good_for = ["budget projects", "offline generation", "quick iteration"] - fallback_tools = ["kling_video", "minimax_video", "wan_video"] + fallback_tools = ["gemini_omni_video", "kling_video", "minimax_video", "wan_video"] input_schema = { "type": "object", diff --git a/tools/video/video_selector.py b/tools/video/video_selector.py index a651cd82..fd8fdfa1 100644 --- a/tools/video/video_selector.py +++ b/tools/video/video_selector.py @@ -20,7 +20,7 @@ class VideoSelector(BaseTool): provider = "selector" stability = ToolStability.BETA runtime = ToolRuntime.HYBRID - agent_skills = ["ai-video-gen", "create-video", "ltx2"] + agent_skills = ["ai-video-gen", "create-video", "ltx2", "gemini-omni"] # Operations that REQUIRE motion: an image-only tool (image_selector) is not # an acceptable last-resort fallback for these, so fallback_tools_for() drops it.