zlei6/OpenMontage
1
0
Fork 0
mirror of https://github.com/calesthio/OpenMontage.git synced 2026-08-05 15:20:40 +08:00
Code Issues Packages Projects Releases Wiki Activity

Harden cinematic pipeline and add reusable renderer

Browse Source
This commit is contained in:
calesthio
2026-03-29 15:49:49 -07:00
parent 5f71d63ff4
commit ede8aac4d2
11 changed files with 658 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
AGENT_GUIDE.md
View File

@@ -45,6 +45,66 @@ Core loop:
4. Present the user with concepts, tool plan, production plan, and cost.
5. Execute stage by stage with checkpoints.
## Decision Communication Contract
For any meaningful production decision, the agent must communicate the decision before acting. The user should never have to infer which provider, model, or render path was chosen after the fact.
### Announce Before Execution
Before any paid or consequential generation call, state:
- the exact tool name,
- the provider,
- the model or provider variant,
- the reason it was chosen,
- whether it is a sample or a batch run.
### Ask Before Major Changes
The agent must ask the user before changing any major production choice, including:
- switching provider,
- switching model family or provider variant,
- switching from video-led to still-led treatment,
- switching composition engine when that changes the output character,
- dropping narration, music, or other approved creative elements,
- changing from sample mode to batch mode.
Minor prompt refinements inside an already approved provider/model path do not require separate approval unless they materially change the creative direction.
### Escalate Blockers Explicitly
When a blocker occurs, the agent must surface it immediately using this structure:
1. What was attempted
2. What failed
3. Whether the issue is auth, provider access, tool bug, or prompt/design quality
4. What options exist next
5. Which option the agent recommends, with reasoning
Do not continue with a substitute path until the user approves.
### Recommendation Style
When asking the user to choose, do not just list options. The agent should:
- provide the shortlist,
- explain the tradeoffs briefly,
- recommend one option,
- wait for approval before proceeding.
### No Unilateral Substitutions
If the approved path is blocked, the agent may investigate and prepare alternatives, but may not execute those alternatives without user approval.
This applies especially to:
- provider swaps,
- model swaps,
- fallback tools,
- prompt-only substitutes for reference-driven generation,
- still-image animatics in place of true motion.
## Orchestrator
The agent itself orchestrates the production state machine:
@@ -232,6 +292,24 @@ print('Remotion note:', info.get('remotion_note'))
| **FFmpeg** | Video-only cuts, concat, trim, subtitle burn | `ffmpeg` binary (always available) |
| **Remotion** | Still images -> animated video, text cards, stat cards, charts, callouts, comparisons, transitions with spring physics | Node.js (`npx`) + `remotion-composer/` project |
### Critical Rule: Motion-Required Requests
For any request where the deliverable inherently depends on motion rather than static coverage, treat motion as a hard requirement. Examples:
- sci-fi trailers,
- cinematic teasers built from generated clips,
- hype edits,
- avatar or agent videos,
- any brief whose promise depends on moving shots rather than still frames.
For these requests:
- `Remotion` availability must be confirmed up front if the planned visual treatment depends on Remotion rendering.
- Still-image fallback is forbidden. Do not quietly convert the job into a Ken Burns teaser, animatic, or slide-based video.
- FFmpeg-only fallback is forbidden when it changes the approved deliverable from motion-led video to still-led video.
- Bubble critical issues immediately. If Remotion is unavailable, fails to render, or provider clip generation fails in a way that blocks the approved treatment, stop and tell the user before proceeding.
- Do not spend more tokens or time on downgraded output unless the user explicitly approves the downgrade as an animatic or proof-of-concept.
**When Remotion is available**, the agent should design production plans around it:
- Explainer videos with `flat-motion-graphics` playbook -> Remotion animated scenes, not Ken Burns
- Data-driven videos -> Remotion stat cards and charts, not static image screenshots
@@ -239,6 +317,8 @@ print('Remotion note:', info.get('remotion_note'))
**When Remotion is NOT available**, `video_compose` falls back to FFmpeg Ken Burns motion on still images. This still works but produces less engaging visuals. Mention this tradeoff in the proposal.
That fallback is only acceptable when it does not violate the approved delivery shape. If the user asked for a motion-led trailer or comparable clip-driven piece, the project is blocked until Remotion is working or the user explicitly approves a lower-fidelity alternative.
The routing is automatic — the `render` operation in `video_compose` calls `_needs_remotion()` and routes accordingly. But the **agent must know Remotion exists at proposal time** so it can design the visual approach to take advantage of it (animated text cards, component scenes, spring transitions) rather than designing around static images.
## Capability Discovery
@@ -487,3 +567,4 @@ Example: Before calling `kling_video`, read its `agent_skills` → `ai-video-gen
- Do not hide degraded paths. Record substitutions and blocked options explicitly.
- Do not present a single unavailable tool in isolation. Always show the full capability picture: "X of Y providers configured for this capability."
- Do not skip the Provider Menu at preflight. The user must see what they have AND what they could unlock.
- Do not change provider, model, or render path without telling the user first and getting approval when the change is material.

3
pipeline_defs/cinematic.yaml
View File

@@ -44,6 +44,7 @@ stages:
human_approval_default: true
review_focus:
- Source mode and emotional arc are clear
- Motion-required delivery is explicitly identified when applicable
- Delivery shape is realistic for available footage and assets
- Cinematic treatment is justified rather than decorative
success_criteria:
@@ -121,6 +122,7 @@ stages:
human_approval_default: false
review_focus:
- Source selects and support assets are clearly separated
- Motion-required beats use actual video clips rather than still-image substitutes
- Music and ambience plan matches the beat map
- Optional generated inserts stay limited and justified
success_criteria:
@@ -175,6 +177,7 @@ stages:
human_approval_default: false
review_focus:
- Output mood matches intended pacing and grade
- Motion-required delivery was preserved without silent fallback
- Audio dynamics are controlled and intelligible
- Letterbox or frame treatment improves rather than harms the output
success_criteria:

292
remotion-composer/src/CinematicRenderer.tsx Normal file
View File

@@ -0,0 +1,292 @@
import React from "react";
import { loadFont } from "@remotion/google-fonts/SpaceGrotesk";
import {
AbsoluteFill,
CalculateMetadataFunction,
OffthreadVideo,
Sequence,
interpolate,
spring,
useCurrentFrame,
useVideoConfig,
} from "remotion";
import { CinematicRendererProps, CinematicTone, CinematicVideoScene } from "./cinematic/types";
const FPS = 30;
const { fontFamily } = loadFont("normal", {
weights: ["400", "500", "700"],
subsets: ["latin"],
});
const toneGradient = (tone: CinematicTone) => {
switch (tone) {
case "steel":
return "linear-gradient(180deg, rgba(6,12,18,0.18) 0%, rgba(2,4,8,0.48) 100%)";
case "void":
return "linear-gradient(180deg, rgba(2,4,8,0.14) 0%, rgba(0,0,0,0.56) 100%)";
case "neutral":
return "linear-gradient(180deg, rgba(10,10,12,0.16) 0%, rgba(0,0,0,0.42) 100%)";
case "cold":
default:
return "linear-gradient(180deg, rgba(8,16,24,0.18) 0%, rgba(2,4,8,0.42) 100%)";
}
};
const SceneVideo: React.FC<{ scene: CinematicVideoScene }> = ({ scene }) => {
const frame = useCurrentFrame();
const { durationInFrames, fps } = useVideoConfig();
const opacity = interpolate(
frame,
[0, 10, durationInFrames - 10, durationInFrames],
[0, 1, 1, 0],
{
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
},
);
const scale = interpolate(frame, [0, durationInFrames], [1.015, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
const trimBefore =
scene.trimBeforeSeconds !== undefined
? Math.round(scene.trimBeforeSeconds * fps)
: undefined;
const trimAfter =
scene.trimAfterSeconds !== undefined
? Math.round(scene.trimAfterSeconds * fps)
: undefined;
return (
<AbsoluteFill style={{ backgroundColor: "#020407", opacity }}>
<OffthreadVideo
muted
src={scene.src}
trimBefore={trimBefore}
trimAfter={trimAfter}
style={{
width: "100%",
height: "100%",
objectFit: "cover",
transform: `scale(${scale})`,
filter:
scene.filter ?? "contrast(1.06) saturate(0.88) brightness(0.92)",
}}
/>
<AbsoluteFill
style={{
background: toneGradient(scene.tone ?? "cold"),
mixBlendMode: "multiply",
}}
/>
<AbsoluteFill
style={{
background:
"radial-gradient(circle at center, transparent 52%, rgba(0,0,0,0.52) 100%)",
}}
/>
<AbsoluteFill
style={{
background:
"linear-gradient(180deg, rgba(255,255,255,0.02) 0%, transparent 8%, transparent 92%, rgba(255,255,255,0.02) 100%)",
opacity: 0.6,
}}
/>
</AbsoluteFill>
);
};
const SignalTexture: React.FC<{
accent: string;
intensity: number;
lineCount: number;
}> = ({ accent, intensity, lineCount }) => {
const frame = useCurrentFrame();
return (
<AbsoluteFill style={{ pointerEvents: "none" }}>
{new Array(lineCount).fill(true).map((_, index) => {
const pulse = Math.max(0, Math.sin(frame * 0.06 + index * 0.85));
const opacity = (0.025 + pulse * 0.07) * intensity;
const width = 18 + ((index * 37) % 56);
const top = 140 + index * 42;
const left = index % 2 === 0 ? 0 : 1920 - width;
return (
<div
key={index}
style={{
position: "absolute",
top,
left,
width,
height: 1,
background: accent,
boxShadow: `0 0 16px ${accent}`,
opacity,
}}
/>
);
})}
<div
style={{
position: "absolute",
inset: 0,
background:
"repeating-linear-gradient(180deg, rgba(255,255,255,0.028) 0px, rgba(255,255,255,0.028) 1px, transparent 2px, transparent 6px)",
opacity: 0.12 * intensity,
}}
/>
</AbsoluteFill>
);
};
const TitleCard: React.FC<{
text: string;
accent: string;
intensity: number;
titleFontSize: number;
titleWidth: number;
signalLineCount: number;
}> = ({
text,
accent,
intensity,
titleFontSize,
titleWidth,
signalLineCount,
}) => {
const frame = useCurrentFrame();
const { fps, durationInFrames } = useVideoConfig();
const reveal = spring({
fps,
frame,
config: { damping: 18, stiffness: 90 },
});
const exit = interpolate(
frame,
[durationInFrames - 12, durationInFrames],
[1, 0],
{
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
},
);
const y = interpolate(reveal, [0, 1], [18, 0]);
const letterSpacing = interpolate(reveal, [0, 1], [0.3, 0.18]);
const flareOpacity =
0.18 + Math.max(0, Math.sin(frame * 0.08)) * 0.14 * intensity;
return (
<AbsoluteFill
style={{
background:
"radial-gradient(circle at 50% 42%, rgba(16,28,40,0.9) 0%, rgba(3,5,8,1) 58%, rgba(0,0,0,1) 100%)",
justifyContent: "center",
alignItems: "center",
}}
>
<SignalTexture
accent={accent}
intensity={intensity}
lineCount={signalLineCount}
/>
<div
style={{
position: "absolute",
width: 880,
height: 2,
background: accent,
boxShadow: `0 0 28px ${accent}`,
opacity: flareOpacity,
transform: "translateY(-126px)",
}}
/>
<div
style={{
position: "absolute",
width: 880,
height: 2,
background: accent,
boxShadow: `0 0 28px ${accent}`,
opacity: flareOpacity * 0.7,
transform: "translateY(126px)",
}}
/>
<div
style={{
opacity: reveal * exit,
transform: `translateY(${y}px)`,
fontFamily,
fontWeight: 700,
fontSize: titleFontSize,
lineHeight: 1.06,
letterSpacing: `${letterSpacing}em`,
textAlign: "center",
color: "#f3f6fa",
textTransform: "uppercase",
width: titleWidth,
textShadow: "0 0 22px rgba(255,255,255,0.08)",
}}
>
{text}
</div>
</AbsoluteFill>
);
};
export const calculateCinematicMetadata: CalculateMetadataFunction<CinematicRendererProps> =
async ({ props }) => {
const totalSeconds =
props.scenes.length === 0
? 30
: Math.max(
...props.scenes.map((scene) => scene.startSeconds + scene.durationSeconds),
);
return {
durationInFrames: Math.max(1, Math.ceil(totalSeconds * FPS)),
fps: FPS,
width: 1920,
height: 1080,
};
};
export const CinematicRenderer: React.FC<CinematicRendererProps> = ({
scenes,
titleFontSize = 78,
titleWidth = 1320,
signalLineCount = 18,
}) => {
return (
<AbsoluteFill style={{ backgroundColor: "#000000" }}>
{scenes.map((scene) => (
<Sequence
key={scene.id}
from={Math.round(scene.startSeconds * FPS)}
durationInFrames={Math.round(scene.durationSeconds * FPS)}
>
{scene.kind === "video" ? (
<SceneVideo scene={scene} />
) : (
<TitleCard
text={scene.text}
accent={scene.accent ?? "#86d8ff"}
intensity={scene.intensity ?? 1}
titleFontSize={titleFontSize}
titleWidth={titleWidth}
signalLineCount={signalLineCount}
/>
)}
</Sequence>
))}
</AbsoluteFill>
);
};

1
remotion-composer/src/Explainer.tsx
View File

@@ -93,6 +93,7 @@ interface AudioConfig {
}
export interface ExplainerProps {
[key: string]: unknown;
cuts: Cut[];
overlays?: Overlay[];
captions?: WordCaption[];

19
remotion-composer/src/Root.tsx
View File

@@ -1,5 +1,9 @@
import { Composition, CalculateMetadataFunction } from "remotion";
import { Explainer, ExplainerProps } from "./Explainer";
import {
CinematicRenderer,
calculateCinematicMetadata,
} from "./CinematicRenderer";
const calculateMetadata: CalculateMetadataFunction<ExplainerProps> = async ({
props,
@@ -31,6 +35,21 @@ export const Root: React.FC = () => {
}}
calculateMetadata={calculateMetadata}
/>
<Composition
id="CinematicRenderer"
component={CinematicRenderer}
durationInFrames={30 * 30}
fps={30}
width={1920}
height={1080}
defaultProps={{
scenes: [],
titleFontSize: 78,
titleWidth: 1320,
signalLineCount: 18,
}}
calculateMetadata={calculateCinematicMetadata}
/>
</>
);
};

33
remotion-composer/src/cinematic/types.ts Normal file
View File

@@ -0,0 +1,33 @@
export type CinematicTone = "cold" | "steel" | "void" | "neutral";
export interface CinematicBaseScene {
id: string;
startSeconds: number;
durationSeconds: number;
}
export interface CinematicVideoScene extends CinematicBaseScene {
kind: "video";
src: string;
tone?: CinematicTone;
trimBeforeSeconds?: number;
trimAfterSeconds?: number;
filter?: string;
}
export interface CinematicTitleScene extends CinematicBaseScene {
kind: "title";
text: string;
accent?: string;
intensity?: number;
}
export type CinematicScene = CinematicVideoScene | CinematicTitleScene;
export interface CinematicRendererProps {
[key: string]: unknown;
scenes: CinematicScene[];
titleFontSize?: number;
titleWidth?: number;
signalLineCount?: number;
}

23
skills/pipelines/cinematic/asset-director.md
View File

@@ -26,6 +26,12 @@ Start with:
These are the primary materials. Everything else is support.
If `brief.metadata.motion_required = true`, actual moving footage or generated video clips are mandatory. In that case:
- stills may be used only as reference material or backing elements inside a larger motion composition,
- stills may not replace the planned motion shots,
- a still-image teaser is not an acceptable fallback unless the user explicitly approves an animatic.
### 1b. Sample Preview (Prevents Wasted Spend)
Before batch-generating support assets, produce one sample of each expensive generated type and show the user:
@@ -33,8 +39,20 @@ Before batch-generating support assets, produce one sample of each expensive gen
1. **Generated insert sample** (if using `image_selector` or `video_selector`): Generate one representative visual. Confirm it complements the source footage before batching.
2. **Music sample** (if using `music_gen`): Generate a short clip. Confirm mood and energy match the beat plan.
If `motion_required = true`, the representative visual must be a video clip sample, not a still image sample.
If rejected, adjust parameters and retry (max 3 iterations). Do not batch until approved.
Before the sample is generated, tell the user exactly which generation path will be used:
- tool,
- provider,
- model or variant,
- generation mode,
- why it was selected.
If that path fails, stop and ask before trying a different provider, model, or generation mode.
### 2. Generate Support Assets Only Where Needed
Optional generated assets should fill clear gaps:
@@ -44,6 +62,8 @@ Optional generated assets should fill clear gaps:
- texture or atmosphere cards,
- simple textural motion backgrounds.
For motion-required jobs, use `video_selector` first for generated shots. `image_selector` may support look development, concept frames, or embedded design layers, but it does not satisfy the motion requirement by itself.
### 3. Prepare A Real Audio Plan
Store:
@@ -70,9 +90,12 @@ Recommended metadata keys:
- generated inserts are limited and purposeful,
- audio plan matches the beat map,
- every referenced file exists.
- if motion is required, the asset set contains actual video clips for the motion-led beats.
## Common Pitfalls
- Generating extra shots before proving the source edit works.
- Treating music as a single loop instead of a beat-aware element.
- Forgetting rights or provenance notes for supplied assets.
- Quietly downgrading from video clips to still images because one provider or renderer failed.
- Quietly switching providers or models after the user approved a generation path.

10
skills/pipelines/cinematic/compose-director.md
View File

@@ -15,6 +15,15 @@ Render the cinematic piece with careful attention to grade, audio dynamics, and
## Process
### 0. Check Hard Requirements Before Rendering
If the approved brief or scene plan makes motion a hard requirement, verify that the render path still preserves that promise.
- If Remotion is required and unavailable or failing, stop and bubble the issue to the user immediately.
- Do not switch to an FFmpeg-only still-image fallback for a motion-led trailer, teaser, or agent video.
- Do not convert the piece into an animatic unless the user explicitly approves that downgrade.
- If the render engine changes materially, tell the user before rendering and explain why.
### 1. Use Frame Treatment Deliberately
Only use letterbox, 24fps intent, or heavy grading if they help the piece. Do not apply them because the pipeline name says cinematic.
@@ -51,3 +60,4 @@ Recommended metadata keys:
- Flattening the audio so the piece loses dynamics.
- Applying letterbox to footage that needs every pixel.
- Letting grading or sharpening damage faces or text.
- Silently swapping a blocked Remotion render for a lower-fidelity still-image export.

27
skills/pipelines/cinematic/executive-producer.md
View File

@@ -50,6 +50,28 @@ Same as standard EP: Initialize → Execute stages serially (idea → script →
Each stage: PREPARE → SPAWN DIRECTOR → REVIEW → GATE DECISION (pass / revise / send-back).
### User-Facing Decision Flow
For this pipeline, the EP must make the decision trail visible to the user.
Before any expensive or consequential generation step, present:
- selected tool,
- provider,
- model or variant,
- why it was chosen,
- whether the run is a sample or a batch.
If the approved path becomes blocked, the EP must stop and present:
- the attempted path,
- the concrete failure,
- the likely class of issue (auth, provider access, tool bug, or creative mismatch),
- the available next options,
- the recommended next option.
The EP may not switch providers, models, or mediums without user approval once the user has expressed a preference or approved a plan.
## EP-Specific Cross-Stage Checks
### After IDEA stage:
@@ -57,6 +79,7 @@ Each stage: PREPARE → SPAWN DIRECTOR → REVIEW → GATE DECISION (pass / revi
CHECK: Emotional arc definition
- Is the emotional arc explicit (build → reveal → landing)?
- Is source mode clear (supplied footage vs generated inserts)?
- Does the brief explicitly say whether motion is required?
- Is the target mood defined and achievable?
```
@@ -88,6 +111,7 @@ CHECK: Visual consistency
CHECK: Music/ambience alignment
- Does the music beat map align with the script beat map?
- Are generated inserts limited and justified?
- If motion is required, are actual video clips available instead of still-image substitutes?
- Budget gate: 90% threshold warning
CHECK: Source selects quality
@@ -114,6 +138,7 @@ CHECK: Output validation
- Color grade applied and consistent
- Audio dynamics controlled — dialogue intelligible, music balanced
- Letterbox or frame treatment improves (not harms) the output
- If motion was required, does the output still satisfy that promise instead of degrading into a still-led animatic?
```
## Quality Gates Summary
@@ -145,3 +170,5 @@ CHECK: Output validation
- **Overuse of generated inserts**: Source footage should be primary. Generated content fills gaps, not replaces.
- **Ignoring audio dynamics**: Cinematic videos live and die by their audio. Music/dialogue balance is critical.
- **Rushing the reveal**: The climax moment needs breathing room. Don't let pacing compress it.
- **Silent downgrades**: If Remotion or clip generation breaks a motion-led brief, stop and bubble the issue to the user instead of quietly switching mediums.
- **Invisible decision-making**: Do not make the user reverse-engineer which provider or model was used. State it before execution and when anything changes.

19
skills/pipelines/cinematic/idea-director.md
View File

@@ -24,6 +24,16 @@ Capture the source mode:
- `generated_support`
- `mixed_montage`
Also classify whether the requested delivery is motion-required. Store this as a boolean in `brief.metadata.motion_required`.
Set `motion_required = true` when the promise of the video depends on moving shots or animated compositions rather than static frames. This includes:
- sci-fi trailers,
- cinematic teasers,
- action or hype edits,
- agent/avatar outputs,
- any concept whose quality depends on generated video clips.
Do not assume stock, generated b-roll, or music exists unless the user has provided it or the environment can actually make it.
### 2. Define The Emotional Arc
@@ -53,6 +63,7 @@ Store longer planning detail in `brief.metadata`.
Recommended metadata keys:
- `source_mode`
- `motion_required`
- `delivery_shape`
- `emotional_arc`
- `anchor_assets`
@@ -65,6 +76,12 @@ Recommended metadata keys:
If the user has weak source media and no generation path, say so. A cinematic result still needs enough visual or audio material to carry mood.
If `motion_required = true`, be explicit about the motion path:
- confirm the planned clip-generation providers,
- confirm whether Remotion is required for the intended composition,
- if either is unavailable or unstable, mark the treatment as blocked rather than silently redesigning it around still images.
### 5. Music Plan (Mandatory)
Cinematic videos live and die by their audio. **Surface the music situation before the user approves the brief.**
@@ -95,6 +112,7 @@ Record the decision in `brief.metadata.music_strategy` with the chosen source an
### 6. Quality Gate
- the source truth is explicit,
- the brief says whether motion is a hard requirement,
- the emotional arc is specific,
- the output shape fits the available assets,
- the music plan is resolved (source chosen or explicitly deferred),
@@ -104,4 +122,5 @@ Record the decision in `brief.metadata.music_strategy` with the chosen source an
- Calling something cinematic when it is really just a normal edit with black bars.
- Assuming generated inserts are available without checking tools.
- Quietly turning a motion-led brief into a still-led teaser.
- Planning a trailer shape with no reveal or payoff.

173
tools/video/veo_video.py
View File

@@ -1,11 +1,15 @@
"""Google Veo 3 video generation via fal.ai API.
"""Google Veo 3.1 video generation via fal.ai API.
State-of-the-art video generation with native audio/dialogue synthesis.
Supports text-to-video, image-to-video, reference-to-video, and first/last-frame
interpolation so agents can preserve visual consistency instead of relying only on
raw text prompts.
"""
from __future__ import annotations
import os
import mimetypes
import base64
import time
from pathlib import Path
from typing import Any
@@ -37,15 +41,17 @@ class VeoVideo(BaseTool):
dependencies = []
install_instructions = (
"Set FAL_KEY to your fal.ai API key.\n"
"Set FAL_KEY or FAL_AI_API_KEY to your fal.ai API key.\n"
" Get one at https://fal.ai/dashboard/keys"
)
agent_skills = ["ai-video-gen"]
capabilities = ["text_to_video", "image_to_video"]
capabilities = ["text_to_video", "image_to_video", "reference_to_video", "first_last_frame_to_video"]
supports = {
"text_to_video": True,
"image_to_video": True,
"reference_to_video": True,
"first_last_frame_to_video": True,
"native_audio": True,
"dialogue_generation": True,
"ambient_sound": True,
@@ -65,18 +71,18 @@ class VeoVideo(BaseTool):
"prompt": {"type": "string"},
"operation": {
"type": "string",
"enum": ["text_to_video", "image_to_video"],
"enum": ["text_to_video", "image_to_video", "reference_to_video", "first_last_frame_to_video"],
"default": "text_to_video",
},
"model_variant": {
"type": "string",
"enum": ["veo3", "veo3/fast", "veo3.1", "veo3.1/fast"],
"default": "veo3",
"default": "veo3.1",
},
"duration": {
"type": "string",
"enum": ["5", "8"],
"default": "8",
"enum": ["4s", "6s", "8s"],
"default": "8s",
"description": "Duration in seconds",
},
"aspect_ratio": {
@@ -89,7 +95,35 @@ class VeoVideo(BaseTool):
"default": True,
"description": "Whether to generate synchronized audio",
},
"resolution": {
"type": "string",
"enum": ["720p", "1080p", "4k"],
"default": "1080p",
},
"negative_prompt": {"type": "string"},
"seed": {"type": "integer"},
"auto_fix": {"type": "boolean", "default": True},
"safety_tolerance": {
"type": "string",
"enum": ["1", "2", "3", "4", "5", "6"],
"default": "4",
},
"image_url": {"type": "string", "description": "Reference image URL for image_to_video"},
"image_path": {"type": "string", "description": "Local reference image path for image_to_video"},
"reference_image_urls": {
"type": "array",
"items": {"type": "string"},
"description": "Reference image URLs for reference_to_video",
},
"reference_image_paths": {
"type": "array",
"items": {"type": "string"},
"description": "Local reference image paths for reference_to_video",
},
"first_frame_url": {"type": "string"},
"first_frame_path": {"type": "string"},
"last_frame_url": {"type": "string"},
"last_frame_path": {"type": "string"},
"output_path": {"type": "string"},
},
}
@@ -114,49 +148,136 @@ class VeoVideo(BaseTool):
return ToolStatus.UNAVAILABLE
def estimate_cost(self, inputs: dict[str, Any]) -> float:
variant = inputs.get("model_variant", "veo3")
duration = int(inputs.get("duration", "8"))
variant = inputs.get("model_variant", "veo3.1")
duration_text = str(inputs.get("duration", "8s")).replace("s", "")
duration = int(duration_text)
resolution = inputs.get("resolution", "1080p")
generate_audio = bool(inputs.get("generate_audio", True))
if "fast" in variant:
per_second = 0.12
base_per_second = 0.10
audio_per_second = 0.20
else:
per_second = 0.30
return per_second * duration
if resolution == "4k":
base_per_second = 0.40
audio_per_second = 0.60
else:
base_per_second = 0.20
audio_per_second = 0.40
return (audio_per_second if generate_audio else base_per_second) * duration
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
variant = inputs.get("model_variant", "veo3")
variant = inputs.get("model_variant", "veo3.1")
if "fast" in variant:
return 45.0
return 120.0
@staticmethod
def _file_to_data_uri(path_str: str) -> str:
path = Path(path_str)
if not path.exists():
raise FileNotFoundError(f"Input file not found: {path}")
mime_type, _ = mimetypes.guess_type(path.name)
if not mime_type:
mime_type = "application/octet-stream"
encoded = base64.b64encode(path.read_bytes()).decode("ascii")
return f"data:{mime_type};base64,{encoded}"
def _normalize_file_input(self, url_value: str | None, path_value: str | None) -> str | None:
if url_value:
return url_value
if path_value:
return self._file_to_data_uri(path_value)
return None
def execute(self, inputs: dict[str, Any]) -> ToolResult:
api_key = self._get_api_key()
if not api_key:
return ToolResult(
success=False,
error="FAL_KEY not set. " + self.install_instructions,
error="FAL_KEY / FAL_AI_API_KEY not set. " + self.install_instructions,
)
import requests
start = time.time()
operation = inputs.get("operation", "text_to_video")
variant = inputs.get("model_variant", "veo3")
variant = inputs.get("model_variant", "veo3.1")
duration = inputs.get("duration", "8s")
# Current fal Veo 3.1 image-guided endpoints only accept 8-second clips.
if variant == "veo3.1" and operation in {"reference_to_video", "first_last_frame_to_video"} and duration != "8s":
return ToolResult(
success=False,
error=(
f"{operation} with {variant} currently requires duration='8s' on fal.ai; "
f"received duration='{duration}'"
),
)
# Build fal.ai model path
if operation == "image_to_video":
model_path = f"{variant}/image-to-video"
else:
model_path = variant # text-to-video is the default endpoint
operation_map = {
"text_to_video": variant,
"image_to_video": f"{variant}/image-to-video",
"reference_to_video": f"{variant}/reference-to-video",
"first_last_frame_to_video": f"{variant}/first-last-frame-to-video",
}
model_path = operation_map[operation]
payload: dict[str, Any] = {"prompt": inputs["prompt"]}
if inputs.get("duration"):
payload["duration"] = inputs["duration"]
if inputs.get("aspect_ratio"):
payload["aspect_ratio"] = inputs["aspect_ratio"]
if inputs.get("resolution"):
payload["resolution"] = inputs["resolution"]
if inputs.get("generate_audio") is not None:
payload["generate_audio"] = inputs["generate_audio"]
if operation == "image_to_video" and inputs.get("image_url"):
payload["image_url"] = inputs["image_url"]
if inputs.get("negative_prompt"):
payload["negative_prompt"] = inputs["negative_prompt"]
if inputs.get("seed") is not None:
payload["seed"] = inputs["seed"]
if inputs.get("auto_fix") is not None:
payload["auto_fix"] = inputs["auto_fix"]
if inputs.get("safety_tolerance"):
payload["safety_tolerance"] = inputs["safety_tolerance"]
if operation == "image_to_video":
image_value = self._normalize_file_input(inputs.get("image_url"), inputs.get("image_path"))
if not image_value:
return ToolResult(
success=False,
error="image_to_video requires image_url or image_path",
)
payload["image_url"] = image_value
if operation == "reference_to_video":
image_urls = list(inputs.get("reference_image_urls") or [])
image_paths = list(inputs.get("reference_image_paths") or [])
normalized = list(image_urls)
normalized.extend(self._file_to_data_uri(path) for path in image_paths)
if not normalized:
return ToolResult(
success=False,
error="reference_to_video requires reference_image_urls or reference_image_paths",
)
payload["image_urls"] = normalized
if operation == "first_last_frame_to_video":
first_frame = self._normalize_file_input(
inputs.get("first_frame_url"), inputs.get("first_frame_path")
)
last_frame = self._normalize_file_input(
inputs.get("last_frame_url"), inputs.get("last_frame_path")
)
if not first_frame or not last_frame:
return ToolResult(
success=False,
error="first_last_frame_to_video requires first_frame_url/path and last_frame_url/path",
)
payload["first_frame_url"] = first_frame
payload["last_frame_url"] = last_frame
headers = {
"Authorization": f"Key {api_key}",
@@ -192,7 +313,12 @@ class VeoVideo(BaseTool):
# Fetch result
result_resp = requests.get(response_url, headers=headers, timeout=30)
result_resp.raise_for_status()
if not result_resp.ok:
detail = result_resp.text[:1000]
return ToolResult(
success=False,
error=f"Veo video generation result fetch failed ({result_resp.status_code}): {detail}",
)
data = result_resp.json()
video_url = data["video"]["url"]
@@ -214,6 +340,7 @@ class VeoVideo(BaseTool):
"prompt": inputs["prompt"],
"output": str(output_path),
"has_audio": inputs.get("generate_audio", True),
"operation": operation,
},
artifacts=[str(output_path)],
cost_usd=self.estimate_cost(inputs),