zlei6/OpenMontage
1
0
Fork 0
mirror of https://github.com/calesthio/OpenMontage.git synced 2026-08-15 05:04:42 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #323 from xucailiang/codex/kling-official-phase-1

Browse Source 
Add official Kling API providers
This commit is contained in:
Calesthio
2026-07-12 10:45:02 -07:00
committed by GitHub
parent f633b5f428 6b9ba782d8
commit df3801a6b9
35 changed files with 6784 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
.agents/skills/ai-video-gen/SKILL.md
View File

@@ -1,7 +1,7 @@
---
name: ai-video-gen
description: |
Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax, Gemini Omni). Supports gateways: HeyGen API, fal.ai API, and the Gemini API (Gemini Omni Flash).
Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax, Gemini Omni). Supports gateways: HeyGen API, fal.ai API, Kling official direct API, and the Gemini API (Gemini Omni Flash).
allowed-tools: mcp__heygen__*
metadata:
openclaw:
@@ -9,18 +9,20 @@ metadata:
env_any:
- HEYGEN_API_KEY
- FAL_KEY
- KLING_API_KEY
- GEMINI_API_KEY
- GOOGLE_API_KEY
---
# Video Generation (Multi-Gateway)
Generate AI videos from text prompts. Supports multiple providers via two API gateways:
Generate AI videos from text prompts. Supports multiple providers via four API paths:
| Gateway | Env Variable | Providers | Tool |
|---------|-------------|-----------|------|
| **fal.ai** | `FAL_KEY` | **Seedance 2.0** (standard + fast), Kling v3/v2.1, MiniMax, VEO | `seedance_video`, `kling_video`, `minimax_video`, `veo_video` |
| **HeyGen** | `HEYGEN_API_KEY` | VEO 3.1, Kling Pro, Sora v2, Runway Gen-4, Seedance Pro / Lite (1.x) | `heygen_video` |
| **Kling Official** | `KLING_API_KEY` | Kling official Classic, Turbo, and basic Omni video | `kling_official_video` |
| **Gemini API** | `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Gemini Omni Flash (generation + conversational editing) | `gemini_omni_video` |
**Iterative editing — Gemini Omni.** When the brief calls for *refining an existing clip* (add/remove objects, restyle, change lighting or on-screen text) rather than regenerating, Gemini Omni Flash is the only provider in the fleet with stateful multi-turn editing. See Layer 3 `gemini-omni` for the authoritative prompting guide (reference-image tags, timecode syntax, edit-prompt rules) before writing any prompt for it.
@@ -35,8 +37,12 @@ Use whichever configured gateway best matches the user's available providers and
- **HeyGen:** Set `HEYGEN_API_KEY` to access the multi-model gateway.
- **fal.ai:** Set `FAL_KEY` to access Kling, MiniMax, and Veo through fal.ai.
- **Kling Official:** Set `KLING_API_KEY` to access Kling's official direct API via `provider="kling_official"`.
- **Gemini API:** Set `GEMINI_API_KEY` or `GOOGLE_API_KEY` to access Gemini Omni video generation and conversational editing.
Do not describe either gateway as the default or top choice without checking the registry and current task fit first.
Do not describe any gateway as the default or top choice without checking the registry and current task fit first.
fal.ai Kling (`kling_video`, `provider="kling"`) and Kling Official (`kling_official_video`, `provider="kling_official"`) are different paths. Do not reuse fal.ai queue URLs, `FAL_KEY`, or image upload behavior when the official provider is selected.
```bash
curl -X POST "https://api.heygen.com/v1/workflows/executions" \

200
.agents/skills/kling-official/SKILL.md Normal file
View File

@@ -0,0 +1,200 @@
---
name: kling-official
description: Official Kling direct API guidance for OpenMontage providers. Use before calling `kling_official_video`, `kling_official_image`, `kling_tts`, `kling_avatar`, or `kling_lip_sync`.
metadata:
openclaw:
requires:
env_any:
- KLING_API_KEY
---
# Kling Official Direct API
Use this skill for OpenMontage tools with `provider="kling_official"`. This is not the fal.ai Kling gateway. Official Kling uses `KLING_API_KEY`, optional `KLING_API_BASE_URL`, and `Authorization: Bearer <KLING_API_KEY>`.
## Provider Split
- `kling_video` uses fal.ai, `FAL_KEY`, fal.ai queue URLs, and `provider="kling"`.
- `kling_official_video` uses Kling official API, `KLING_API_KEY`, official task protocols, and `provider="kling_official"`.
- `kling_official_image` uses the same official auth and task protocol for image generation.
- `kling_tts` uses the official audio TTS endpoint and stays in the existing `tts` capability.
- `kling_avatar` and `kling_lip_sync` use official avatar/lip-sync endpoints and stay in the existing `avatar` capability. They do not replace local `talking_head` or `lip_sync`.
Never silently switch between these paths. If the selected provider is unavailable, surface the blocker and ask before substituting.
## Auth And Endpoint
Default base URL:
```text
https://api-singapore.klingai.com
```
Users may override it with `KLING_API_BASE_URL`, for example for a regional endpoint. All requests send JSON and:
```text
Authorization: Bearer <KLING_API_KEY>
```
## Task Protocols
Classic APIs:
- Create ID path: `data.task_id`
- Statuses: `submitted`, `processing`, `succeed`, `failed`
- Result paths: `data.task_result.videos[]`, `data.task_result.images[]`, `data.task_result.audios[]`
Turbo APIs:
- Create ID path: `data.id`
- Poll path: `GET /tasks?task_ids=<id>`
- Statuses: `submitted`, `processing`, `succeeded`, `failed`
- Result path: `data[0].outputs[]`
Keep the parsers separate. Do not write a fuzzy parser that guesses between `task_id` and `id` or between `succeed` and `succeeded`.
## Omni References
Video Omni and Image Omni stay inside the existing provider tools through `api_family="omni"`.
Do not create selector-level Omni operations.
Video Omni accepts official reference structures:
- `image_list[]` with `image_url` and optional `type` such as `first_frame` or `end_frame`.
- `video_list[]` with `video_url`, official `refer_type` values such as `feature` or `base`, and optional `keep_original_sound`.
- `element_list[]` with official `element_id` values.
- Structured `multi_prompt[]`; do not split natural language into shots automatically.
Local image references may be normalized through `tools/_kling/media.py`. Local video paths must not be silently uploaded through fal.ai; ask for or require a reachable URL.
Image Omni accepts `image_list[]` with official `image` values. Prompt placeholders such as `<<<image_1>>>` must map stably to the provided image order. If the prompt already contains placeholders, validate that the referenced images exist and do not insert duplicates.
## Capability Boundaries
TTS, avatar, and lip sync are provider additions to existing OpenMontage capabilities. Audio effects and video effects are official Kling endpoints, but they are not registered as default OpenMontage tools until a pipeline has a stable capability slot for them.
- Do not add `sound_effects` or `video_effects` capabilities from inside a provider implementation.
- Do not let video effects enter the ordinary `video_generation` selector path.
- Do not disguise short sound effects as long background music unless a pipeline explicitly consumes that shape and the tool's `best_for` / `not_good_for` says so.
## Video Parameters
Use `operation` for OpenMontage semantics:
- `text_to_video`
- `image_to_video`
- `reference_to_video`
Use `api_family` for official protocol choice:
- `classic`
- `turbo`
- `omni`
Important constraints:
- Official video provider input schema must not expose top-level `image_url`; use `reference_image_url` or `reference_image_path`.
- Classic image-to-video accepts `reference_image_url` or a local path converted to raw base64 in official field `image`.
- Turbo image-to-video requires a URL first frame. Do not upload local files through fal.ai as a fallback.
- Send `aspect_ratio` only where the current schema supports it: Classic text-to-video, Turbo text-to-video, and Video Omni.
- Default paid path should avoid `4k`, native sound, or batch behavior unless explicitly selected.
## Image Parameters
Use `api_family="generation"` for `/v1/images/generations` and `api_family="omni"` for `/v1/images/omni-image`.
Generation/edit path:
- `prompt` is required and should stay under the official 2500 character limit.
- `image_url` passes through as official `image`.
- `image_path` is converted to raw base64 and sent as official `image`.
- `image_reference` can be `subject` or `face`.
Omni path:
- Put references in `image_list[]` using official `image` values.
- Use prompt placeholders such as `<<<image_1>>>` only when the prompt needs to bind a specific reference image.
## TTS Parameters
`kling_tts` uses:
- `text`
- `voice_id`
- `voice_language`, currently `zh` or `en`
- `voice_speed`
Require an explicit `voice_id` unless an official account-specific default has been verified. Do not hard-code a made-up voice. Download every returned audio item, set `data.output_path` to the first local file, and include `voice_id`, `voice_language`, `voice_speed`, `task_id`, and non-zero `cost_usd`.
## Avatar Parameters
`kling_avatar` uses `/v1/videos/avatar/image2video` and accepts:
- avatar image via URL or local path converted to raw base64
- `audio_id` or `sound_file`
- optional `prompt`
- `mode`, such as `std` or `pro`
Keep it separate from local `talking_head`. Pipelines that want Kling avatar output must list and choose it explicitly.
## Lip Sync Parameters
`kling_lip_sync` has two steps:
1. `POST /v1/videos/identify-face` with `video_id` or `video_url`; read faces from `data.face_data[]`
2. `POST /v1/videos/advanced-lip-sync` with `session_id` and one `face_choose[]` item containing `face_id`, `audio_id` or `sound_file`, and the sound start/end/insert times
Local video paths must not be silently uploaded through fal.ai or any other provider. If multiple faces are returned and the user did not pass `face_id` or `face_choose`, stop and return the face list for confirmation unless `auto_select_face=True` was explicitly set. If auto-selecting, record the selection reason and selected face in the result/artifact.
## Audio Effects And Video Effects
Official Kling audio effects (`/v1/audio/text-to-audio`, `/v1/audio/video-to-audio`) and video effects (`/v1/videos/effects`) are intentionally not default OpenMontage selector tools. Record the non-mapping reason in docs/tests instead of registering tools that current pipelines might misuse.
## Elements Helper
Elements are an internal Kling Official helper, not a new OpenMontage tool capability.
Use `tools/_kling/elements.py` to normalize `element_list[].element_id`, optionally query read-only element endpoints, and record element metadata when queried. Do not create or delete elements from the default provider path.
## Account Usage Helper
Account Usage is diagnostic only. Use `tools/_kling/account.py` for low-frequency `/account/costs` checks, with local cache and throttle protection. Do not call it before every generation and do not put it in selectors or production pipeline stages.
For `1101` or `1102`, surface that the account or resource pack is exhausted and include an account-usage diagnostic hint.
## Callback Notes
Providers may accept `callback_url`, but polling remains the default execution mode.
- Classic and Omni paths pass `callback_url` at the top level.
- Turbo paths pass it as `options.callback_url`.
- Successful results should record `callback_requested=true`, `polling_used=true`, the `callback_url`, and `task_id`.
- Validate callback URLs before sending; only absolute `http` or `https` URLs should pass.
## Error Handling
Surface official `code`, `message`, and `request_id` whenever available.
Do not retry:
- Auth failures: `1000`-`1004`
- Balance/resource-pack exhaustion: `1101`, `1102`
- Permission/model access: `1103`
- Parameter errors: `1200`, `1201`
- Safety policy: `1301`
Limited retry is acceptable for:
- `1302` request too fast
- `1303` concurrency/resource-pack slot limit
- `5000`, `5001`, `5002` server/maintenance/backlog errors
For `1303`, explain that the account hit a concurrency or resource-pack slot limit.
## Cost Governance
Official Kling is a paid remote API. Provider tools must return non-zero conservative estimates from `estimate_cost()` and include `cost_usd` on successful paid results. Treat estimates as low-confidence until account usage reconciliation is implemented.
High-cost Omni inputs such as multiple references, element IDs, `result_type="series"`, `mode="4k"`, and `sound="on"` must increase or flag the cost estimate.
## Prompt Notes
For video, start from the universal OpenMontage video prompt skeleton: subject, subject motion, scene, spatial framing, and camera. Kling tends to respond well to clear temporal action order, camera movement verbs, and concise negative prompts. For reference workflows, state what should stay consistent from the reference and what should change.

5
.env.example
View File

@@ -16,6 +16,11 @@ HIGGSFIELD_API_KEY= # Higgsfield Cloud key (higgsfield_video). Pair wit
HIGGSFIELD_API_SECRET= # or use the combined HIGGSFIELD_KEY="<key>:<secret>" form instead.
# HIGGSFIELD_KEY= # Combined key:secret — set this INSTEAD of the _KEY/_SECRET pair if you prefer.
# --- Kling official direct API ---
KLING_API_KEY= # Official Kling API key; enables video, image, TTS, avatar, lip sync
KLING_API_BASE_URL= # Optional endpoint override; leave blank for default https://api-singapore.klingai.com
# Mainland China accounts can use https://api-beijing.klingai.com
# --- Google (one key unlocks image gen + TTS + video) ---
GOOGLE_API_KEY= # Google Imagen images, Google Cloud TTS (700+ voices, 50+ languages),
# Gemini Omni video (generation + conversational editing, paid tier)

17
README.md
View File

@@ -224,6 +224,10 @@ This repo is built for agentic operation. If you're an OpenClaw-style agent, her
# Image + video gateway:
FAL_KEY=your-key # FLUX images + Google Veo, Kling, MiniMax video + Recraft images
# Kling official direct API:
KLING_API_KEY=your-key # Official Kling video, image, TTS, avatar, lip sync
KLING_API_BASE_URL= # Optional; default Singapore API endpoint
# Free stock media:
PEXELS_API_KEY=your-key # Free stock footage and images
PIXABAY_API_KEY=your-key # Free stock footage and images
@@ -469,11 +473,12 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
> **Full setup guide with pricing and free tiers:** [`docs/PROVIDERS.md`](docs/PROVIDERS.md)
<details>
<summary><strong>Video Generation — 14 providers</strong></summary>
<summary><strong>Video Generation — 15 providers</strong></summary>
| Provider | Type | Notes |
|----------|------|-------|
| **Kling** | Cloud API | High quality, fast |
| **Kling (fal.ai)** | Cloud API | High quality, fast via fal.ai gateway |
| **Kling Official** | Cloud API | Official direct API with separate `kling_official` provider |
| **Runway Gen-4** | Cloud API | Cinematic quality, Gen-3 Alpha Turbo / Gen-4 Turbo / Gen-4 Aleph |
| **Google Veo 3** | Cloud API | Long-form, cinematic. Via fal.ai or HeyGen. |
| **Grok Imagine Video** | Cloud API | Strong reference-image video and xAI-native short-form generation |
@@ -491,7 +496,7 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
</details>
<details>
<summary><strong>Image Generation — 10 tools/providers</strong></summary>
<summary><strong>Image Generation — 11 tools/providers</strong></summary>
| Provider | Type | Notes |
|----------|------|-------|
@@ -500,6 +505,7 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
| **Grok Imagine Image** | Cloud API | Strong image edits, style transfer, and multi-image compositing |
| **GPT Image 2** | Cloud API | OpenAI's image model |
| **Recraft** | Cloud API | Design-focused generation |
| **Kling Official** | Cloud API | Official direct API for Kling image generation and reference workflows |
| **Local Diffusion** | Local GPU | Stable Diffusion, free |
| **Pexels** | Stock | Free stock images |
| **Pixabay** | Stock | Free stock images |
@@ -509,12 +515,13 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
</details>
<details>
<summary><strong>Text-to-Speech — 4 providers</strong></summary>
<summary><strong>Text-to-Speech — 5 providers</strong></summary>
| Provider | Type | Notes |
|----------|------|-------|
| **ElevenLabs** | Cloud API | Premium voice quality |
| **Google TTS** | Cloud API | 700+ voices, 50+ languages — best for localization |
| **Kling Official TTS** | Cloud API | Official Kling narration when a `voice_id` is known |
| **OpenAI TTS** | Cloud API | Fast, affordable |
| **Piper** | Local | Completely free, offline |
@@ -567,6 +574,8 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
|------|-------------|
| **Talking Head** | SadTalker / MuseTalk avatar animation |
| **Lip Sync** | Wav2Lip audio-driven lip synchronization |
| **Kling Avatar** | Official Kling cloud avatar presenter generation |
| **Kling Lip Sync** | Official Kling cloud lip-sync with explicit face selection |
**Composition & Rendering:**

14
docs/ARCHITECTURE.md
View File

@@ -386,6 +386,8 @@ All config is validated via Pydantic models in `lib/config_model.py`.
| `OPENAI_API_KEY` | openai_tts, openai_image | TTS fallback, GPT Image 2 |
| `XAI_API_KEY` | grok_image, grok_video | Grok image editing/generation, Grok video generation |
| `FAL_KEY` | flux_image, kling_video, veo_video, minimax_video, recraft_image | fal.ai hosted models (FLUX, Veo, Kling, MiniMax, Recraft) |
| `KLING_API_KEY` | kling_official_video, kling_official_image, kling_tts, kling_avatar, kling_lip_sync | Official Kling direct API for video, image, TTS, avatar, and lip sync |
| `KLING_API_BASE_URL` | kling_official_video, kling_official_image, kling_tts, kling_avatar, kling_lip_sync | Optional official Kling API endpoint override |
| `HEYGEN_API_KEY` | heygen_video | Multi-provider video generation |
| `PEXELS_API_KEY` | pexels_image, pexels_video | Stock media |
| `PIXABAY_API_KEY` | pixabay_image, pixabay_video | Stock media |
@@ -396,6 +398,18 @@ All config is validated via Pydantic models in `lib/config_model.py`.
| `VIDEO_GEN_LOCAL_ENABLED` | local video tools | Enable local GPU generation |
| `VIDEO_GEN_LOCAL_MODEL` | wan, hunyuan, ltx, cogvideo | Select local model |
Kling Official support stays inside the existing provider and capability model.
`kling_official_video` and `kling_official_image` handle Classic, Turbo, and Omni
request shapes, while Elements and Account Usage live under `tools/_kling/` as
internal helpers for element ID references and low-frequency account diagnostics;
they are not separate pipeline stages, selectors, or generated-asset capabilities.
Kling Official also adds provider tools only where OpenMontage already has a
matching capability slot: `kling_tts` for `tts`, plus `kling_avatar` and
`kling_lip_sync` for `avatar`. Official Kling audio effects and video effects are
not registered as tools because current pipelines do not define stable
`sound_effects` or `video_effects` capability routing.
---
## Visual Style System

66
docs/PROVIDERS.md
View File

@@ -17,11 +17,12 @@ Everything you need to know about every provider in OpenMontage — setup instru
| 5 | **~$0.03/image** | fal.ai | FLUX images + Kling/Veo/MiniMax video + Recraft — broad single-key image + video coverage |
| 6 | **~$0.05/image** | OpenAI | GPT Image 2 images + OpenAI TTS |
| 7 | **~$0.04/image** | Google Imagen | Imagen 4 images (shares the Google API key) |
| 8 | **$12/month** | Runway | Gen-4 video — highest quality AI video |
| 9 | **pay-as-you-go** | HeyGen | Avatar videos, multi-model video gateway |
| 10 | **pay-as-you-go** | Suno | Full song generation with vocals and lyrics |
| 11 | **$0 + GPU** | Local video gen | WAN 2.1, Hunyuan, CogVideo, LTX — free, offline |
| 12 | **$0 + GPU** | Local Diffusion | Stable Diffusion images — free, offline |
| 8 | **pay-as-you-go** | Kling Official | Official direct Kling video, image, TTS, avatar, and lip-sync API, separate from fal.ai Kling |
| 9 | **$12/month** | Runway | Gen-4 video — highest quality AI video |
| 10 | **pay-as-you-go** | HeyGen | Avatar videos, multi-model video gateway |
| 11 | **pay-as-you-go** | Suno | Full song generation with vocals and lyrics |
| 12 | **$0 + GPU** | Local video gen | WAN 2.1, Hunyuan, CogVideo, LTX — free, offline |
| 13 | **$0 + GPU** | Local Diffusion | Stable Diffusion images — free, offline |
### Environment Variable Summary
@@ -46,6 +47,10 @@ DASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with
# MULTI-MODEL GATEWAY (one key, 6+ tools)
FAL_KEY= # FLUX, Recraft, Kling, Veo, MiniMax video
# KLING OFFICIAL DIRECT API
KLING_API_KEY= # Official Kling video, image, TTS, avatar, lip sync
KLING_API_BASE_URL= # Optional; default https://api-singapore.klingai.com
# VIDEO
HEYGEN_API_KEY= # HeyGen avatar video gateway
RUNWAY_API_KEY= # Runway Gen-4 video (direct)
@@ -171,6 +176,48 @@ No subscription — pure pay-as-you-go, no minimum spend.
---
### Kling Official — Direct API
> **Official Kling path.** This is separate from `kling_video` via fal.ai: it uses Kling's official `Authorization: Bearer <KLING_API_KEY>` API, provider name `kling_official`, and direct Classic/Turbo/Omni task protocols.
**Tools unlocked:** `kling_official_video`, `kling_official_image`, `kling_tts`, `kling_avatar`, `kling_lip_sync`
**Env vars:** `KLING_API_KEY`, optional `KLING_API_BASE_URL`
#### Setup
1. Create or open a Kling AI Open Platform account.
2. Generate an official API key in the Kling API console.
3. Add to `.env`:
```bash
KLING_API_KEY=your-key-here
# Optional, defaults to Singapore:
KLING_API_BASE_URL=https://api-singapore.klingai.com
```
#### What It Is Best For
- Direct official Kling API provenance rather than fal.ai gateway routing
- Text-to-video, image-to-video, and deep Video Omni reference workflows via `kling_official_video`
- Text-to-image, image edit/reference, and Image Omni multi-reference or series workflows via `kling_official_image`
- Text-to-speech via `kling_tts` when you already know the official Kling `voice_id`
- Cloud avatar presenter clips via `kling_avatar`, without replacing local `talking_head`
- Cloud lip-sync via `kling_lip_sync`, with explicit face selection for multi-person videos
- Accounts that need to use official Kling model permissions, resource packs, or regional endpoints
#### Notes
- `provider="kling_official"` is intentionally different from fal.ai's `provider="kling"`.
- Official Kling is a paid remote API. OpenMontage uses conservative cost estimates and includes high-cost factors such as Omni references, series output, 4k mode, and native sound.
- Local image paths are sent as raw base64 for supported Classic/image-generation fields. Turbo image-to-video requires a URL and will not silently upload through fal.ai.
- Video Omni and Image Omni can pass official `element_id` references through `element_list`; Elements remain an internal Kling Official helper, not a standalone OpenMontage capability.
- Account Usage is available as a low-frequency diagnostic helper under `tools/_kling/account.py`; it is not a selector or pipeline tool.
- `callback_url` is passed through and recorded when supplied, but OpenMontage still polls tasks by default.
- `kling_tts` requires an explicit `voice_id`; OpenMontage does not guess a default official voice.
- `kling_avatar` and `kling_lip_sync` register under the existing `avatar` capability and coexist with local SadTalker/Wav2Lip tools. Current avatar pipelines must opt into them explicitly; registry discovery alone does not replace local tools.
- Official Kling audio effects and video effects are documented but intentionally not registered as OpenMontage tools yet, because current pipelines do not have a stable sound-effects or video-effects capability slot for them.
---
### ElevenLabs — Voice, Music, Sound Effects
> **Premium voice quality.** Best TTS for narration-heavy videos. Also generates music and sound effects.
@@ -783,6 +830,7 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
| **Google** | `GOOGLE_API_KEY` (or `GEMINI_API_KEY`) | `google_tts`, `google_imagen`, `google_music`, `gemini_omni_video`, `veo_video` | Free tier (TTS) + paid |
| **ElevenLabs** | `ELEVENLABS_API_KEY` | `elevenlabs_tts`, `music_gen` | Free tier + paid |
| **fal.ai** | `FAL_KEY` | `flux_image`, `recraft_image`, `kling_video`, `veo_video`, `minimax_video` | Pay-as-you-go |
| **Kling Official** | `KLING_API_KEY` | `kling_official_video`, `kling_official_image`, `kling_tts`, `kling_avatar`, `kling_lip_sync` | Pay-as-you-go |
| **OpenAI** | `OPENAI_API_KEY` | `openai_tts`, `openai_image` | Paid only |
| **xAI** | `XAI_API_KEY` | `grok_image`, `grok_video` | Paid only |
| **Runway** | `RUNWAY_API_KEY` | `runway_video` | Free trial + paid |
@@ -801,14 +849,14 @@ How many providers cover each capability:
| Capability | Cloud Providers | Local Providers | Free Options |
|-----------|----------------|-----------------|--------------|
| **Image Generation** | FLUX, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
| **Video Generation** | Grok, Kling, Runway, Veo, Gemini Omni, Higgsfield, MiniMax, HeyGen | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
| **Text-to-Speech** | ElevenLabs, Google TTS, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier |
| **Image Generation** | FLUX, Kling Official, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
| **Video Generation** | Grok, Kling Official, Kling via fal.ai, Runway, Veo, Gemini Omni, Higgsfield, MiniMax, HeyGen | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
| **Text-to-Speech** | ElevenLabs, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier |
| **Music Generation** | ElevenLabs, Suno, Google Lyria | — | ElevenLabs free tier |
| **Post-Production** | — | FFmpeg (compose, stitch, trim, mix, enhance, grade) | All free |
| **Analysis** | — | WhisperX, Scene Detect, Frame Sampler, CLIP/BLIP-2 | All free |
| **Enhancement** | — | Upscale, BG Remove, Face Enhance, Face Restore | All free |
| **Avatar** | — | SadTalker, Wav2Lip | All free |
| **Avatar** | Kling Official | SadTalker, Wav2Lip | Local tools are free |
---

805
scripts/kling_official_animated_explainer_e2e.py Normal file
View File

@@ -0,0 +1,805 @@
#!/usr/bin/env python3
"""Kling Official animated-explainer E2E smoke script.
This script validates the official Kling provider path through OpenMontage
selectors and the animated-explainer asset/compose surface.
Default mode is a no-cost dry run. Use --live-tts for one paid TTS sample,
--live-full for TTS + image + image-to-video + local FFmpeg compose, or
--live-all to add avatar and lip-sync provider smokes.
"""
from __future__ import annotations
import argparse
import json
import math
import os
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Mapping, Sequence
ROOT = Path(__file__).resolve().parent.parent
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from lib.env_loader import load_env # noqa: E402
load_env(ROOT)
from lib.pipeline_loader import load_pipeline # noqa: E402
from tools.tool_registry import registry # noqa: E402
from tools.video.video_compose import VideoCompose # noqa: E402
DEFAULT_PROJECT = "kling-animated-explainer-e2e"
DEFAULT_VOICE_ID = "oversea_male1"
DEFAULT_TTS_TEXT = (
"Throughout my time in college, several memorable events left a significant impact on my life."
)
REQUIRED_LIVE_ENV = ("KLING_API_KEY",)
RELEVANT_ENV = ("KLING_API_KEY", "KLING_API_BASE_URL", "FAL_KEY", "OPENAI_API_KEY")
def _json_safe(value: Any) -> Any:
if isinstance(value, Path):
return str(value)
if isinstance(value, dict):
return {str(k): _json_safe(v) for k, v in value.items()}
if isinstance(value, list):
return [_json_safe(v) for v in value]
return value
def _write_json(path: Path, data: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(_json_safe(data), indent=2, ensure_ascii=False), encoding="utf-8")
def _probe_media(path: Path) -> dict[str, Any]:
cmd = [
"ffprobe",
"-v",
"error",
"-show_entries",
"stream=codec_type,codec_name,width,height,sample_rate,channels,duration",
"-show_entries",
"format=format_name,duration,size,bit_rate",
"-of",
"json",
str(path),
]
try:
completed = subprocess.run(cmd, capture_output=True, text=True, check=True)
return json.loads(completed.stdout or "{}")
except Exception as exc:
return {"error": str(exc)}
def _env_status(environ: Mapping[str, str | None] | None = None) -> dict[str, dict[str, Any]]:
source = environ if environ is not None else os.environ
status: dict[str, dict[str, Any]] = {}
for key in RELEVANT_ENV:
value = source.get(key)
present = bool(value)
if not present:
display = "<missing>"
elif key == "KLING_API_BASE_URL":
display = str(value)
else:
display = f"<set:{len(str(value))} chars>"
status[key] = {"present": present, "length": len(str(value or "")), "display": display}
return status
def _missing_required_env() -> list[str]:
return [key for key in REQUIRED_LIVE_ENV if not os.environ.get(key)]
def _discover() -> None:
registry.clear()
registry.discover("tools")
def _tool_statuses() -> dict[str, str]:
names = [
"tts_selector",
"image_selector",
"video_selector",
"video_compose",
"kling_tts",
"kling_official_image",
"kling_official_video",
"kling_avatar",
"kling_lip_sync",
]
statuses: dict[str, str] = {}
for name in names:
tool = registry.get(name)
statuses[name] = tool.get_status().value if tool else "missing"
return statuses
def _capability_summary() -> dict[str, Any]:
summary = registry.provider_menu_summary()
wanted = {"tts", "image_generation", "video_generation", "avatar", "video_post"}
return {
"composition_runtimes": summary.get("composition_runtimes", {}),
"capabilities": [
item for item in summary.get("capabilities", []) if item.get("capability") in wanted
],
"runtime_warnings": summary.get("runtime_warnings", []),
}
def _kling_entry(rank_result: dict[str, Any]) -> dict[str, Any] | None:
for item in rank_result.get("rankings", []):
if item.get("provider") == "kling_official":
return item
return None
def _rank_selectors(voice_id: str, voice_language: str, voice_speed: float) -> dict[str, Any]:
tts = registry.get("tts_selector")
image = registry.get("image_selector")
video = registry.get("video_selector")
assert tts and image and video
tts_rank = tts.execute(
{
"operation": "rank",
"allowed_providers": ["kling_official"],
"text": "Kling official TTS selector smoke test.",
"voice_id": voice_id,
"voice_language": voice_language,
"voice_speed": voice_speed,
}
).data
image_rank = image.execute(
{
"operation": "rank",
"allowed_providers": ["kling_official"],
"prompt": "Clean minimal explainer visual about AI video production.",
"api_family": "generation",
"model_name": "kling-v3",
}
).data
video_rank = video.execute(
{
"operation": "rank",
"target_operation": "image_to_video",
"allowed_providers": ["kling_official"],
"prompt": "Slow camera push over a clean explainer visual.",
"api_family": "classic",
"model_name": "kling-v3",
"duration": "3",
"mode": "std",
"sound": "off",
}
).data
return {
"note": (
"Rank mode is advisory. Live modes use preferred_provider and "
"allowed_providers to force kling_official selection."
),
"kling_official_entries": {
"tts": _kling_entry(tts_rank),
"image": _kling_entry(image_rank),
"video": _kling_entry(video_rank),
},
"tts_rank_all": tts_rank,
"image_rank_all": image_rank,
"video_rank_all": video_rank,
}
def _dry_run(voice_id: str, voice_language: str, voice_speed: float, text: str) -> dict[str, Any]:
dry: dict[str, Any] = {}
cases = {
"kling_tts": {
"text": text,
"voice_id": voice_id,
"voice_language": voice_language,
"voice_speed": voice_speed,
},
"kling_official_image": {
"prompt": "Clean minimal explainer visual about AI video production.",
"api_family": "generation",
"model_name": "kling-v3",
"resolution": "1k",
"aspect_ratio": "16:9",
"n": 1,
},
"kling_official_video": {
"prompt": "Slow camera push over a clean explainer visual.",
"operation": "image_to_video",
"api_family": "classic",
"model_name": "kling-v3",
"duration": "3",
"mode": "std",
"sound": "off",
},
}
for name, payload in cases.items():
tool = registry.get(name)
dry[name] = tool.dry_run(payload) if tool else {"status": "missing"}
return dry
def _require_success(name: str, result: Any) -> None:
if not result.success:
raise RuntimeError(f"{name} failed: {result.error}")
def _aligned_video_duration(requested_duration: str, narration_seconds: float | None) -> str:
requested = int(requested_duration)
if narration_seconds:
requested = max(requested, int(math.ceil(narration_seconds)))
return str(min(max(requested, 3), 15))
def _announce_paid_call(tool: str, provider: str, model: str, reason: str, run_type: str) -> None:
print(f"[paid:{run_type}] tool={tool} provider={provider} model={model}")
print(f"[paid:{run_type}] reason={reason}")
def _run_live_tts(
project_dir: Path,
*,
voice_id: str,
voice_language: str,
voice_speed: float,
text: str,
timeout_seconds: int,
poll_interval: float,
include_account_usage: bool,
) -> dict[str, Any]:
tts = registry.get("tts_selector")
assert tts
audio_dir = project_dir / "assets" / "audio"
audio_dir.mkdir(parents=True, exist_ok=True)
_announce_paid_call(
"tts_selector -> kling_tts",
"kling_official",
"kling-official-tts",
"Validate official Kling TTS through OpenMontage selector routing.",
"sample",
)
result = tts.execute(
{
"preferred_provider": "kling_official",
"allowed_providers": ["kling_official"],
"text": text,
"voice_id": voice_id,
"voice_language": voice_language,
"voice_speed": voice_speed,
"sample_mode": True,
"include_account_usage": include_account_usage,
"timeout_seconds": timeout_seconds,
"poll_interval": poll_interval,
"output_path": str(audio_dir / "narration.mp3"),
}
)
_require_success("tts_selector", result)
output_path = Path(result.data["output_path"])
return {
"result": result.data,
"artifacts": {"narration": str(output_path)},
"ffprobe": _probe_media(output_path),
"estimated_cost_usd": float(result.cost_usd or 0),
}
def _run_live_full(
project_dir: Path,
*,
voice_id: str,
voice_language: str,
voice_speed: float,
text: str,
timeout_seconds: int,
poll_interval: float,
include_account_usage: bool,
video_duration: str,
) -> dict[str, Any]:
image = registry.get("image_selector")
video = registry.get("video_selector")
assert image and video
assets = project_dir / "assets"
image_dir = assets / "images"
video_dir = assets / "video"
renders_dir = project_dir / "renders"
for path in (image_dir, video_dir, renders_dir):
path.mkdir(parents=True, exist_ok=True)
tts_data = _run_live_tts(
project_dir,
voice_id=voice_id,
voice_language=voice_language,
voice_speed=voice_speed,
text=text,
timeout_seconds=timeout_seconds,
poll_interval=poll_interval,
include_account_usage=include_account_usage,
)
narration_path = Path(tts_data["artifacts"]["narration"])
target_video_duration = _aligned_video_duration(
video_duration,
tts_data.get("result", {}).get("audio_duration_seconds"),
)
_announce_paid_call(
"image_selector -> kling_official_image",
"kling_official",
"kling-v3",
"Generate one reference frame for the animated-explainer E2E smoke.",
"sample",
)
image_result = image.execute(
{
"preferred_provider": "kling_official",
"allowed_providers": ["kling_official"],
"prompt": (
"A clean 16:9 animated-explainer hero frame: a luminous production "
"pipeline diagram on a dark desk, small cards labeled script, voice, "
"image, video, render, realistic yet crisp, no text artifacts."
),
"negative_prompt": "blurry, unreadable text, distorted interface, watermark",
"api_family": "generation",
"model_name": "kling-v3",
"resolution": "1k",
"aspect_ratio": "16:9",
"n": 1,
"output_path": str(image_dir / "hero_frame.png"),
}
)
_require_success("image_selector", image_result)
image_path = Path(image_result.data["output_path"])
_announce_paid_call(
"video_selector -> kling_official_video",
"kling_official",
"kling-v3 classic image_to_video",
"Animate the generated reference frame for a minimal provider E2E smoke.",
"sample",
)
video_result = video.execute(
{
"preferred_provider": "kling_official",
"allowed_providers": ["kling_official"],
"prompt": (
"A slow cinematic push-in over the explainer pipeline diagram. "
"Cards glow softly in sequence, subtle parallax, stable camera, smooth motion."
),
"operation": "image_to_video",
"api_family": "classic",
"model_name": "kling-v3",
"reference_image_path": str(image_path),
"duration": target_video_duration,
"mode": "std",
"sound": "off",
"output_path": str(video_dir / "kling_i2v_clip.mp4"),
"timeout_seconds": max(timeout_seconds, 900),
"poll_interval": poll_interval,
}
)
_require_success("video_selector", video_result)
clip_path = Path(video_result.data["output_path"])
print("[local] tool=video_compose runtime=ffmpeg reason=minimal one-clip provider smoke")
edit_decisions = {
"version": "1.0",
"render_runtime": "ffmpeg",
"renderer_family": "video_concat_smoke",
"cuts": [
{
"id": "cut-001",
"source": str(clip_path),
"in_seconds": 0,
"out_seconds": float(target_video_duration),
"speed": 1.0,
}
],
"subtitles": {"enabled": False},
"metadata": {
"pipeline": "animated-explainer",
"compose_target": {"width": 1280, "height": 720, "fit": "pad"},
"provider_smoke": True,
"approved_runtime_reason": "Minimal provider integration smoke uses ffmpeg compose for one clip.",
},
}
compose_result = VideoCompose().execute(
{
"operation": "compose",
"edit_decisions": edit_decisions,
"audio_path": str(narration_path),
"output_path": str(renders_dir / "final_kling_e2e_smoke.mp4"),
"profile": "youtube_landscape",
"crf": 23,
"preset": "medium",
}
)
_require_success("video_compose", compose_result)
final_path = Path(compose_result.data["output"])
return {
"tts": tts_data,
"image": image_result.data,
"video": video_result.data,
"compose": compose_result.data,
"requested_video_duration": video_duration,
"aligned_video_duration": target_video_duration,
"artifacts": {
"narration": str(narration_path),
"image": str(image_path),
"clip": str(clip_path),
"final": str(final_path),
},
"ffprobe": {
"narration": _probe_media(narration_path),
"image": _probe_media(image_path),
"clip": _probe_media(clip_path),
"final": _probe_media(final_path),
},
"estimated_cost_usd": sum(
float(getattr(result, "cost_usd", 0) or 0)
for result in (image_result, video_result)
)
+ float(tts_data.get("estimated_cost_usd") or 0),
}
def _first_remote_url(result_data: dict[str, Any]) -> str:
direct = result_data.get("remote_url")
if direct:
return str(direct)
for item in result_data.get("remote_outputs") or []:
if not isinstance(item, dict):
continue
url = item.get("url") or item.get("video_url") or item.get("resource_url")
if url:
return str(url)
raise RuntimeError("Kling result did not include a remote video URL for lip-sync input.")
def _run_live_avatar_suite(
project_dir: Path,
*,
timeout_seconds: int,
poll_interval: float,
) -> dict[str, Any]:
image = registry.get("image_selector")
avatar = registry.get("kling_avatar")
lip_sync = registry.get("kling_lip_sync")
assert image and avatar and lip_sync
assets_dir = project_dir / "assets"
image_dir = assets_dir / "images"
video_dir = assets_dir / "video"
artifacts_dir = project_dir / "artifacts"
narration_path = assets_dir / "audio" / "narration.mp3"
if not narration_path.is_file():
raise RuntimeError(
f"Avatar smoke requires an existing narration file: {narration_path}. "
"Run --live-full or --live-all first."
)
_announce_paid_call(
"image_selector -> kling_official_image",
"kling_official",
"kling-v3 generation",
"Create one synthetic single-face portrait for the avatar provider smoke.",
"sample",
)
portrait_result = image.execute(
{
"preferred_provider": "kling_official",
"allowed_providers": ["kling_official"],
"prompt": (
"Photorealistic studio portrait of one fictional adult presenter, front-facing, "
"head and shoulders centered, neutral expression, mouth closed, even soft lighting, "
"plain background, no text, no watermark."
),
"negative_prompt": "multiple people, profile view, open mouth, obscured face, text, watermark",
"api_family": "generation",
"model_name": "kling-v3",
"resolution": "1k",
"aspect_ratio": "1:1",
"n": 1,
"output_path": str(image_dir / "avatar_portrait.png"),
}
)
_require_success("avatar portrait", portrait_result)
portrait_path = Path(portrait_result.data["output_path"])
_announce_paid_call(
"kling_avatar",
"kling_official",
"kling-official-avatar std",
"Validate photo-and-audio to avatar video through the official provider.",
"sample",
)
avatar_result = avatar.execute(
{
"image_path": str(portrait_path),
"audio_path": str(narration_path),
"prompt": "Natural presenter delivery with subtle head movement and stable identity.",
"mode": "std",
"timeout_seconds": max(timeout_seconds, 900),
"poll_interval": poll_interval,
"output_path": str(video_dir / "kling_avatar_smoke.mp4"),
}
)
_require_success("kling_avatar", avatar_result)
avatar_path = Path(avatar_result.data["output_path"])
partial_report_path = artifacts_dir / "kling_avatar_live_partial.json"
_write_json(
partial_report_path,
{
"portrait": portrait_result.data,
"avatar": avatar_result.data,
"artifacts": {
"narration": str(narration_path),
"portrait": str(portrait_path),
"avatar": str(avatar_path),
},
"ffprobe": {
"portrait": _probe_media(portrait_path),
"avatar": _probe_media(avatar_path),
},
},
)
_announce_paid_call(
"kling_lip_sync",
"kling_official",
"kling-official-lip-sync full_lip_sync",
"Validate identify-face, explicit auto-selection, and advanced lip-sync as one smoke.",
"sample",
)
lip_sync_result = lip_sync.execute(
{
"operation": "full_lip_sync",
"video_url": _first_remote_url(avatar_result.data),
"audio_path": str(narration_path),
"auto_select_face": True,
"faces_artifact_path": str(artifacts_dir / "kling_lip_sync_faces.json"),
"timeout_seconds": max(timeout_seconds, 900),
"poll_interval": poll_interval,
"output_path": str(video_dir / "kling_lip_sync_smoke.mp4"),
}
)
_require_success("kling_lip_sync", lip_sync_result)
lip_sync_path = Path(lip_sync_result.data["output_path"])
estimated_cost = sum(
float(getattr(result, "cost_usd", 0) or 0)
for result in (portrait_result, avatar_result, lip_sync_result)
)
return {
"portrait": portrait_result.data,
"avatar": avatar_result.data,
"lip_sync": lip_sync_result.data,
"artifacts": {
"narration": str(narration_path),
"portrait": str(portrait_path),
"avatar": str(avatar_path),
"lip_sync": str(lip_sync_path),
"faces": str(artifacts_dir / "kling_lip_sync_faces.json"),
"avatar_partial_report": str(partial_report_path),
},
"ffprobe": {
"portrait": _probe_media(portrait_path),
"avatar": _probe_media(avatar_path),
"lip_sync": _probe_media(lip_sync_path),
},
"estimated_cost_usd": estimated_cost,
}
def _run_live_all(
project_dir: Path,
*,
voice_id: str,
voice_language: str,
voice_speed: float,
text: str,
timeout_seconds: int,
poll_interval: float,
include_account_usage: bool,
video_duration: str,
) -> dict[str, Any]:
core = _run_live_full(
project_dir,
voice_id=voice_id,
voice_language=voice_language,
voice_speed=voice_speed,
text=text,
timeout_seconds=timeout_seconds,
poll_interval=poll_interval,
include_account_usage=include_account_usage,
video_duration=video_duration,
)
avatar_suite = _run_live_avatar_suite(
project_dir,
timeout_seconds=timeout_seconds,
poll_interval=poll_interval,
)
return {
"core": core,
"avatar_suite": avatar_suite,
"artifacts": {
**core["artifacts"],
**avatar_suite["artifacts"],
},
"ffprobe": {
"core": core["ffprobe"],
"avatar_suite": avatar_suite["ffprobe"],
},
"estimated_cost_usd": float(core.get("estimated_cost_usd") or 0)
+ float(avatar_suite.get("estimated_cost_usd") or 0),
}
def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
live = parser.add_mutually_exclusive_group()
live.add_argument("--live-tts", action="store_true", help="Run one paid Kling TTS sample.")
live.add_argument(
"--live",
"--live-full",
dest="live_full",
action="store_true",
help="Run paid Kling TTS, image, video, and local compose.",
)
live.add_argument(
"--live-avatar",
action="store_true",
help="Use existing narration to run paid Kling portrait, avatar, and lip-sync samples.",
)
live.add_argument(
"--live-all",
action="store_true",
help="Run the full smoke plus paid Kling avatar and lip-sync samples.",
)
parser.add_argument("--voice-id", default=DEFAULT_VOICE_ID)
parser.add_argument("--voice-language", choices=["en", "zh"], default="en")
parser.add_argument("--voice-speed", type=float, default=1.0)
parser.add_argument("--text", default=DEFAULT_TTS_TEXT)
parser.add_argument("--video-duration", choices=[str(v) for v in range(3, 16)], default="3")
parser.add_argument("--timeout-seconds", type=int, default=300)
parser.add_argument("--poll-interval", type=float, default=3.0)
parser.add_argument("--include-account-usage", action="store_true")
parser.add_argument("--project", default=DEFAULT_PROJECT)
return parser.parse_args(argv)
def _execution_mode(args: argparse.Namespace) -> str:
if getattr(args, "live_tts", False):
return "live_tts"
if getattr(args, "live_full", False):
return "live_full"
if getattr(args, "live_avatar", False):
return "live_avatar"
if getattr(args, "live_all", False):
return "live_all"
return "dry_run"
def main(argv: Sequence[str] | None = None) -> int:
args = _parse_args(argv)
mode = _execution_mode(args)
project_dir = ROOT / "projects" / args.project
report_path = project_dir / "artifacts" / "kling_official_animated_explainer_e2e_report.json"
manifest = load_pipeline("animated-explainer")
_discover()
report: dict[str, Any] = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"pipeline": manifest["name"],
"purpose": "Kling official selector-level E2E smoke for animated-explainer assets + compose.",
"mode": mode,
"voice_id": args.voice_id,
"voice_language": args.voice_language,
"voice_speed": args.voice_speed,
"project_dir": str(project_dir),
"env_status": _env_status(),
"tool_statuses": _tool_statuses(),
"capability_summary": _capability_summary(),
"selector_rankings": _rank_selectors(args.voice_id, args.voice_language, args.voice_speed),
"dry_run": _dry_run(args.voice_id, args.voice_language, args.voice_speed, args.text),
}
missing_env = _missing_required_env()
if mode != "dry_run" and missing_env:
report["blocked"] = {
"reason": "missing required live environment variables",
"missing_env": missing_env,
}
_write_json(report_path, report)
print(f"blocked: missing required live env vars: {', '.join(missing_env)}")
print(f"report: {report_path}")
return 2
try:
if mode == "live_tts":
report["live_tts_result"] = _run_live_tts(
project_dir,
voice_id=args.voice_id,
voice_language=args.voice_language,
voice_speed=args.voice_speed,
text=args.text,
timeout_seconds=args.timeout_seconds,
poll_interval=args.poll_interval,
include_account_usage=args.include_account_usage,
)
elif mode == "live_full":
report["live_full_result"] = _run_live_full(
project_dir,
voice_id=args.voice_id,
voice_language=args.voice_language,
voice_speed=args.voice_speed,
text=args.text,
timeout_seconds=args.timeout_seconds,
poll_interval=args.poll_interval,
include_account_usage=args.include_account_usage,
video_duration=args.video_duration,
)
elif mode == "live_avatar":
report["live_avatar_result"] = _run_live_avatar_suite(
project_dir,
timeout_seconds=args.timeout_seconds,
poll_interval=args.poll_interval,
)
elif mode == "live_all":
report["live_all_result"] = _run_live_all(
project_dir,
voice_id=args.voice_id,
voice_language=args.voice_language,
voice_speed=args.voice_speed,
text=args.text,
timeout_seconds=args.timeout_seconds,
poll_interval=args.poll_interval,
include_account_usage=args.include_account_usage,
video_duration=args.video_duration,
)
else:
report["next_steps"] = [
"Run with --live-tts to make one paid Kling TTS sample call.",
"Run with --live-full to make paid Kling TTS/image/video calls and compose final_kling_e2e_smoke.mp4.",
"Run with --live-avatar to reuse narration for paid Kling avatar and lip-sync provider smokes.",
"Run with --live-all to add paid Kling avatar and lip-sync provider smokes.",
]
except Exception as exc:
report["failed"] = {"error": str(exc)}
_write_json(report_path, report)
print(f"failed: {exc}")
print(f"report: {report_path}")
return 1
_write_json(report_path, report)
print(f"report: {report_path}")
if mode == "live_tts":
print(f"narration: {report['live_tts_result']['artifacts']['narration']}")
elif mode == "live_full":
print(f"final: {report['live_full_result']['artifacts']['final']}")
elif mode == "live_avatar":
print(f"avatar: {report['live_avatar_result']['artifacts']['avatar']}")
print(f"lip_sync: {report['live_avatar_result']['artifacts']['lip_sync']}")
elif mode == "live_all":
print(f"avatar: {report['live_all_result']['artifacts']['avatar']}")
print(f"lip_sync: {report['live_all_result']['artifacts']['lip_sync']}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

1
skills/INDEX.md
View File

@@ -314,5 +314,6 @@ Claude Code accesses them via symlinks in `.claude/skills/`.
| **Animation** | `framer-motion`, `lottie-bodymovin` | `pproenca/dot-skills`, `dylantarre/animation-principles` |
| **Design** | `tailwind-design-system`, `web-design-guidelines`, `vercel-react-best-practices`, `vercel-composition-patterns` | `wshobson/agents`, `vercel-labs/agent-skills` |
| **AI Video (HeyGen)** | `heygen`, `avatar-video`, `create-video`, `faceswap`, `ai-video-gen`, `video-download`, `video-edit`, `video-translate`, `video-understand`, `visual-style` | `heygen-com/skills` |
| **AI Video/Image/TTS/Avatar (Kling Official)** | `kling-official` - official direct API auth, Classic/Turbo/Omni task protocols, multi-reference Omni syntax, internal Elements/Account Usage helpers, callback notes, TTS voice parameters, avatar/lip-sync face selection, error handling, and cost governance for `kling_official_video` / `kling_official_image` / `kling_tts` / `kling_avatar` / `kling_lip_sync` | Local OpenMontage skill |
| **AI Video (Premium)** | `seedance-2-0` — preferred premium default (cinematic, trailer, multi-shot, lip-sync, synced audio); accessed via `seedance_video` (fal.ai) or `heygen_video` Avatar Shots | Local OpenMontage skill |
| **Infrastructure** | `acestep`, `ltx2`, `playwright-recording` | `digitalsamba/claude-code-video-toolkit` |

1
skills/creative/video-gen-prompting.md
View File

@@ -22,6 +22,7 @@ For model-specific tips, see the linked guides below.
| **HunyuanVideo 1.5** | [Tencent Prompt Handbook](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5/blob/main/assets/HunyuanVideo_1_5_Prompt_Handbook_EN.md) | Formula: Subject + Motion + Scene + [Shot] + [Camera] + [Lighting] + [Style] + [Atmosphere]. |
| **Runway Gen-4** | [Runway Prompting Guide](https://help.runwayml.com/hc/en-us/articles/39789879462419-Gen-4-Video-Prompting-Guide) | "Focus on motion, not appearance." One scene per clip. Simplicity wins. |
| **Kling 2.6** | [Kling Prompt Guide](https://fal.ai/learn/devs/kling-2-6-pro-prompt-guide) | 4-part structure. Supports `++emphasis++` syntax for key elements. |
| **Kling Official** | Layer 3 `.agents/skills/kling-official/` | Direct official API. Use `provider="kling_official"` to distinguish it from fal.ai Kling. `api_family` selects Classic, Turbo, or Omni; Turbo image-to-video needs a URL reference image. |
| **Wan 2.1 / CogVideoX** | Use this generic guide | No official prompt guide. Standard cinematographic vocabulary works well. |
## Order Matters

15
tests/contracts/conftest.py Normal file
View File

@@ -0,0 +1,15 @@
"""Shared fixtures for contract tests."""
from __future__ import annotations
import pytest
from tools.tool_registry import ToolRegistry
@pytest.fixture()
def isolated_tool_registry(monkeypatch) -> ToolRegistry:
"""Provide a registry singleton replacement scoped to one test."""
test_registry = ToolRegistry()
monkeypatch.setattr("tools.tool_registry.registry", test_registry)
return test_registry

472
tests/contracts/test_kling_official_avatar.py Normal file
View File

@@ -0,0 +1,472 @@
"""Contract tests for Kling official avatar and lip-sync providers."""
from __future__ import annotations
import base64
import json
import sys
from pathlib import Path
import pytest
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
from tools.avatar.kling_avatar import KlingAvatar
from tools.avatar.kling_lip_sync import KlingLipSync
from tools.avatar.lip_sync import LipSync
from tools.avatar.talking_head import TalkingHead
def test_registry_discovers_kling_avatar(monkeypatch, isolated_tool_registry):
monkeypatch.delenv("KLING_API_KEY", raising=False)
isolated_tool_registry.discover("tools")
tool = isolated_tool_registry.get("kling_avatar")
assert tool is not None
assert tool.capability == "avatar"
assert tool.provider == "kling_official"
def test_avatar_schema_and_local_tool_are_distinct():
tool = KlingAvatar()
assert "anyOf" in tool.input_schema
assert "allOf" in tool.input_schema
assert "kling-official" in tool.agent_skills
assert "avatar-video" in tool.agent_skills
assert tool.runtime.value == "api"
assert TalkingHead().provider == "sadtalker"
assert TalkingHead().runtime.value == "local_gpu"
def test_avatar_payload_uses_image_and_audio_paths(tmp_path):
image_path = tmp_path / "avatar.png"
audio_path = tmp_path / "voice.mp3"
image_path.write_bytes(b"image")
audio_path.write_bytes(b"audio")
request = KlingAvatar()._build_request(
{
"image_path": str(image_path),
"audio_path": str(audio_path),
"prompt": "warm presenter, subtle head motion",
"mode": "pro",
"callback_url": "https://example.com/kling/callback",
}
)
assert request["path"] == "/v1/videos/avatar/image2video"
assert request["payload"]["image"] == base64.b64encode(b"image").decode("ascii")
assert request["payload"]["sound_file"] == base64.b64encode(b"audio").decode("ascii")
assert request["payload"]["mode"] == "pro"
assert request["payload"]["callback_url"] == "https://example.com/kling/callback"
assert request["audio_source"]["type"] == "sound_file"
def test_avatar_requires_image_and_audio():
tool = KlingAvatar()
try:
tool._build_request({"audio_id": "audio-a"})
except ValueError as exc:
assert "image_url or image_path" in str(exc)
else:
raise AssertionError("Kling avatar must require an image")
try:
tool._build_request({"image_url": "https://example.com/avatar.png"})
except ValueError as exc:
assert "requires audio_id" in str(exc)
else:
raise AssertionError("Kling avatar must require audio input")
def test_execute_downloads_avatar_video(monkeypatch, tmp_path):
class FakeClient:
def create_classic_task(self, path, payload):
self.path = path
self.payload = payload
return "avatar-task-1"
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
assert result_key == "videos"
return [{"url": "https://example.com/avatar.mp4"}]
def download(self, url, output_path):
output_path.write_bytes(b"video")
return output_path
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.avatar.kling_avatar.KlingClient", lambda: FakeClient())
monkeypatch.setattr("tools.avatar.kling_avatar.probe_output", lambda path: {"duration_seconds": 5.0})
result = KlingAvatar().execute(
{
"image_url": "https://example.com/avatar.png",
"audio_id": "audio-a",
"output_path": str(tmp_path / "avatar.mp4"),
}
)
assert result.success
assert result.data["provider"] == "kling_official"
assert result.data["task_id"] == "avatar-task-1"
assert result.data["duration_seconds"] == 5.0
assert Path(result.artifacts[0]).read_bytes() == b"video"
assert result.cost_usd > 0
def test_avatar_cost_estimate_is_not_zero():
tool = KlingAvatar()
base = tool.estimate_cost({"image_url": "x", "audio_id": "a"})
pro = tool.estimate_cost({"image_url": "x", "audio_id": "a", "mode": "pro"})
assert base > 0
assert pro > base
assert tool.dry_run({"image_url": "x", "audio_id": "a"})["cost_estimate_confidence"] == "low"
def test_registry_discovers_kling_lip_sync(monkeypatch, isolated_tool_registry):
monkeypatch.delenv("KLING_API_KEY", raising=False)
isolated_tool_registry.discover("tools")
tool = isolated_tool_registry.get("kling_lip_sync")
assert tool is not None
assert tool.capability == "avatar"
assert tool.provider == "kling_official"
def test_lip_sync_schema_and_local_tool_are_distinct():
tool = KlingLipSync()
props = tool.input_schema["properties"]
assert "kling-official" in tool.agent_skills
assert "avatar-video" in tool.agent_skills
assert "sound_start_time" in props
assert "sound_end_time" in props
assert "sound_insert_time" in props
assert tool.runtime.value == "api"
assert LipSync().provider == "wav2lip"
assert LipSync().runtime.value == "local_gpu"
def test_identify_face_payload_and_local_video_rejection():
tool = KlingLipSync()
request = tool._build_identify_request({"video_url": "https://example.com/source.mp4"})
assert request["path"] == "/v1/videos/identify-face"
assert request["payload"] == {"video_url": "https://example.com/source.mp4"}
try:
tool._build_identify_request({"video_path": "/tmp/local.mp4"})
except ValueError as exc:
assert "local video paths cannot be silently uploaded" in str(exc)
else:
raise AssertionError("Local video paths must not be silently uploaded")
def test_advanced_lip_sync_payload_uses_face_and_audio_path(tmp_path):
audio_path = tmp_path / "voice.mp3"
audio_path.write_bytes(b"audio")
request = KlingLipSync()._build_advanced_request(
{
"session_id": "session-a",
"face_id": "face-a",
"audio_path": str(audio_path),
"sound_start_time": 0,
"sound_end_time": 2500,
"sound_insert_time": 500,
"callback_url": "https://example.com/kling/callback",
}
)
assert request["path"] == "/v1/videos/advanced-lip-sync"
assert request["payload"]["session_id"] == "session-a"
assert request["payload"]["face_choose"] == [
{
"face_id": "face-a",
"sound_file": base64.b64encode(b"audio").decode("ascii"),
"sound_start_time": 0,
"sound_end_time": 2500,
"sound_insert_time": 500,
}
]
assert "sound_file" not in request["payload"]
assert request["payload"]["callback_url"] == "https://example.com/kling/callback"
def test_advanced_lip_sync_accepts_official_nested_face_choose():
face_choose = {
"face_id": "face-a",
"audio_id": "audio-a",
"sound_start_time": 0,
"sound_end_time": 4000,
"sound_insert_time": 500,
}
request = KlingLipSync()._build_advanced_request(
{"session_id": "session-a", "face_choose": [face_choose]}
)
assert request["payload"]["face_choose"] == [face_choose]
assert request["audio_source"] == {"type": "audio_id", "value": "audio-a"}
def test_full_lip_sync_preserves_nested_face_timing_defaults():
inputs = {
"session_id": "session-a",
"face_choose": [
{
"face_id": "face-a",
"audio_id": "audio-a",
"sound_start_time": 250,
"sound_end_time": 4250,
"sound_insert_time": 750,
}
],
}
tool = KlingLipSync()
tool._apply_face_timing_defaults(
inputs, {"face_id": "face-a", "start_time": 0, "end_time": 0}
)
request = tool._build_advanced_request(inputs)
assert request["payload"]["face_choose"][0]["sound_start_time"] == 250
assert request["payload"]["face_choose"][0]["sound_end_time"] == 4250
assert request["payload"]["face_choose"][0]["sound_insert_time"] == 750
@pytest.mark.parametrize(
("top_level", "nested", "message"),
[
({"audio_id": "audio-top"}, {"audio_id": "audio-nested"}, "audio input"),
({"sound_end_time": 5000}, {"sound_end_time": 4000}, "sound_end_time"),
],
)
def test_advanced_lip_sync_rejects_conflicting_top_level_and_nested_values(
top_level, nested, message
):
inputs = {
"session_id": "session-a",
"face_choose": [
{
"face_id": "face-a",
"audio_id": "audio-a",
"sound_start_time": 0,
"sound_end_time": 4000,
"sound_insert_time": 0,
**nested,
}
],
**top_level,
}
with pytest.raises(ValueError, match=message):
KlingLipSync()._build_advanced_request(inputs)
def test_identify_face_execute_writes_faces_artifact(monkeypatch, tmp_path):
class FakeClient:
def post(self, path, payload):
assert path == "/v1/videos/identify-face"
return {
"code": 0,
"data": {
"session_id": "session-a",
"face_data": [
{
"face_id": "face-a",
"face_image": "https://example.com/face.png",
"start_time": 0,
"end_time": 5200,
}
],
},
}
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.avatar.kling_lip_sync.KlingClient", lambda: FakeClient())
artifact_path = tmp_path / "faces.json"
result = KlingLipSync().execute(
{
"operation": "identify_face",
"video_url": "https://example.com/source.mp4",
"faces_artifact_path": str(artifact_path),
}
)
assert result.success
assert result.data["session_id"] == "session-a"
data = json.loads(artifact_path.read_text())
assert data["provider"] == "kling_official"
assert data["face_count"] == 1
def test_full_lip_sync_requires_confirmation_for_multiple_faces(monkeypatch, tmp_path):
class FakeClient:
def post(self, path, payload):
return {
"code": 0,
"data": {
"session_id": "session-a",
"face_data": [
{"face_id": "face-small", "bbox": [0, 0, 50, 50], "start_time": 0, "end_time": 4000},
{"face_id": "face-large", "bbox": [0, 0, 200, 200], "start_time": 500, "end_time": 5500},
],
},
}
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.avatar.kling_lip_sync.KlingClient", lambda: FakeClient())
result = KlingLipSync().execute(
{
"operation": "full_lip_sync",
"video_url": "https://example.com/source.mp4",
"audio_id": "audio-a",
"faces_artifact_path": str(tmp_path / "faces.json"),
}
)
assert not result.success
assert result.data["requires_face_selection"] is True
assert len(result.data["faces"]) == 2
assert "Multiple faces detected" in result.error
assert Path(result.artifacts[0]).is_file()
def test_full_lip_sync_auto_selects_largest_face_and_downloads(monkeypatch, tmp_path):
class FakeClient:
def post(self, path, payload):
return {
"code": 0,
"data": {
"session_id": "session-a",
"face_data": [
{"face_id": "face-small", "bbox": [0, 0, 50, 50], "start_time": 0, "end_time": 4000},
{"face_id": "face-large", "bbox": [0, 0, 200, 200], "start_time": 500, "end_time": 5500},
],
},
}
def create_classic_task(self, path, payload):
self.path = path
self.payload = payload
assert payload["face_choose"] == [
{
"face_id": "face-large",
"audio_id": "audio-a",
"sound_start_time": 0,
"sound_end_time": 5000,
"sound_insert_time": 500,
}
]
return "lip-task-1"
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
assert result_key == "videos"
return [{"url": "https://example.com/lip.mp4"}]
def download(self, url, output_path):
output_path.write_bytes(b"video")
return output_path
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.avatar.kling_lip_sync.KlingClient", lambda: FakeClient())
monkeypatch.setattr("tools.avatar.kling_lip_sync.probe_output", lambda path: {"duration_seconds": 4.0})
result = KlingLipSync().execute(
{
"operation": "full_lip_sync",
"video_url": "https://example.com/source.mp4",
"audio_id": "audio-a",
"auto_select_face": True,
"output_path": str(tmp_path / "lip.mp4"),
}
)
assert result.success
assert result.data["task_id"] == "lip-task-1"
assert result.data["face_selection"]["selection_method"] == "auto_selected"
assert result.data["face_choose"] == [
{
"face_id": "face-large",
"audio_id": "audio-a",
"sound_start_time": 0,
"sound_end_time": 5000,
"sound_insert_time": 500,
}
]
assert Path(result.artifacts[0]).read_bytes() == b"video"
faces_artifact = next(Path(path) for path in result.artifacts if Path(path).name == "kling_lip_sync_faces.json")
assert json.loads(faces_artifact.read_text())["selection"]["selection_method"] == "auto_selected"
assert result.cost_usd > 0
def test_auto_select_face_area_avoids_position_inflation():
tool = KlingLipSync()
assert tool._face_area({"bbox": [10, 20, 110, 220]}) == 100 * 200
assert tool._face_area({"bbox": [10, 20, 100, 200]}) == 90 * 180
assert tool._face_area({"box": {"width": 80, "height": 90}}) == 80 * 90
def test_advanced_lip_sync_execute_downloads_video(monkeypatch, tmp_path):
audio_path = tmp_path / "voice.mp3"
audio_path.write_bytes(b"audio")
class FakeClient:
def create_classic_task(self, path, payload):
assert path == "/v1/videos/advanced-lip-sync"
assert payload["face_choose"] == [
{
"face_id": "face-a",
"sound_file": base64.b64encode(b"audio").decode("ascii"),
"sound_start_time": 0,
"sound_end_time": 4000,
"sound_insert_time": 0,
}
]
return "lip-task-1"
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
return [{"video_url": "https://example.com/lip.mp4"}]
def download(self, url, output_path):
output_path.write_bytes(b"video")
return output_path
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.avatar.kling_lip_sync.KlingClient", lambda: FakeClient())
monkeypatch.setattr("tools.avatar.kling_lip_sync.probe_output", lambda path: {"duration_seconds": 4.0})
result = KlingLipSync().execute(
{
"operation": "advanced_lip_sync",
"session_id": "session-a",
"face_id": "face-a",
"audio_path": str(audio_path),
"sound_start_time": 0,
"sound_end_time": 4000,
"sound_insert_time": 0,
"output_path": str(tmp_path / "lip.mp4"),
}
)
assert result.success
assert result.data["provider"] == "kling_official"
assert result.data["task_id"] == "lip-task-1"
assert result.data["duration_seconds"] == 4.0
assert result.data["face_choose"] == [
{
"face_id": "face-a",
"sound_start_time": 0,
"sound_end_time": 4000,
"sound_insert_time": 0,
"sound_file_provided": True,
}
]
assert result.cost_usd > 0
def test_lip_sync_cost_estimate_is_not_zero():
tool = KlingLipSync()
assert tool.estimate_cost({"operation": "identify_face"}) > 0
assert tool.estimate_cost({"operation": "advanced_lip_sync"}) > tool.estimate_cost({"operation": "identify_face"})
assert tool.dry_run({"operation": "advanced_lip_sync"})["cost_estimate_confidence"] == "low"

517
tests/contracts/test_kling_official_core.py Normal file
View File

@@ -0,0 +1,517 @@
"""Contract tests for Kling official shared client, helpers, and docs."""
from __future__ import annotations
import json
import os
import sys
from pathlib import Path
import pytest
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
from tools._kling.client import KlingClient
from tools._kling.errors import KlingAPIError, is_retryable_kling_error
from tools._kling.account import get_account_costs, reset_account_usage_cache
from tools._kling.elements import (
get_custom_element,
list_custom_elements,
list_preset_elements,
normalize_element_list,
write_elements_artifact,
)
from tools._kling.schemas import DEFAULT_API_BASE_URL
from tools.audio.kling_tts import KlingTTS
from tools.avatar.kling_avatar import KlingAvatar
from tools.avatar.kling_lip_sync import KlingLipSync
from tools.graphics.kling_official_image import KlingOfficialImage
from tools.video.kling_official_video import KlingOfficialVideo
class FakeResponse:
def __init__(self, data=None, status_code=200, content=b"data", text=""):
self._data = data if data is not None else {"code": 0}
self.status_code = status_code
self.content = content
self.text = text
def json(self):
if isinstance(self._data, Exception):
raise self._data
return self._data
class FakeSession:
def __init__(self, responses):
self.responses = list(responses)
self.calls = []
def post(self, url, **kwargs):
self.calls.append(("post", url, kwargs))
return self.responses.pop(0)
def get(self, url, **kwargs):
self.calls.append(("get", url, kwargs))
return self.responses.pop(0)
class HelperFakeClient:
def __init__(self, api_key="fake-key", base_url="https://api.example.test"):
self.api_key = api_key
self.base_url = base_url
self.calls = []
def get(self, path, params=None):
self.calls.append((path, params or {}))
if path.startswith("/v1/general/advanced-custom-elements/"):
return {"code": 0, "data": {"element_id": 123}}
if path == "/v1/general/advanced-custom-elements":
return {"code": 0, "data": [{"element_id": 456}]}
if path == "/v1/general/advanced-presets-elements":
return {"code": 0, "data": [{"element_id": 1}]}
return {"code": 0, "data": {"resource_pack_subscribe_infos": [{"name": "pack-a"}]}}
class HelperFakeResponse:
status_code = 200
def json(self):
return {"code": 0, "data": {"resource_pack_subscribe_infos": [{"name": "pack-a"}]}}
class HelperFakeSession:
def __init__(self):
self.calls = []
def get(self, url, **kwargs):
self.calls.append(("get", url, kwargs))
return HelperFakeResponse()
def read(path: str) -> str:
return (PROJECT_ROOT / path).read_text(encoding="utf-8")
def test_missing_api_key_header_error(monkeypatch):
monkeypatch.delenv("KLING_API_KEY", raising=False)
client = KlingClient(session=FakeSession([]))
with pytest.raises(KlingAPIError) as exc:
_ = client.headers
assert "KLING_API_KEY" in str(exc.value)
def test_headers_use_bearer_api_key(monkeypatch):
monkeypatch.setenv("KLING_API_KEY", "test-key")
session = FakeSession([FakeResponse({"code": 0, "data": {"ok": True}})])
client = KlingClient(session=session)
client.post("/v1/test", {"prompt": "x"})
headers = session.calls[0][2]["headers"]
assert headers["Authorization"] == "Bearer test-key"
assert headers["Content-Type"] == "application/json"
def test_default_and_env_base_url(monkeypatch):
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.delenv("KLING_API_BASE_URL", raising=False)
assert KlingClient().base_url == DEFAULT_API_BASE_URL
monkeypatch.setenv("KLING_API_BASE_URL", "https://api-beijing.klingai.com")
assert KlingClient().base_url == "https://api-beijing.klingai.com"
def test_business_error_preserves_code_message_request_id(monkeypatch):
monkeypatch.setenv("KLING_API_KEY", "test-key")
session = FakeSession([FakeResponse({"code": 1200, "message": "bad parameter", "request_id": "req-1"})])
client = KlingClient(session=session, max_retries=0)
with pytest.raises(KlingAPIError) as exc:
client.post("/v1/videos/text2video", {})
assert exc.value.code == 1200
assert exc.value.message == "bad parameter"
assert exc.value.request_id == "req-1"
def test_1303_retryable_message_mentions_concurrency(monkeypatch):
monkeypatch.setenv("KLING_API_KEY", "test-key")
session = FakeSession([FakeResponse({"code": 1303, "message": "parallel task over resource pack limit"})])
client = KlingClient(session=session, max_retries=0)
with pytest.raises(KlingAPIError) as exc:
client.post("/v1/videos/text2video", {})
assert is_retryable_kling_error(exc.value)
assert "并发/资源包限制" in exc.value.message
def test_classic_create_and_poll_parse_result_paths(monkeypatch):
monkeypatch.setenv("KLING_API_KEY", "test-key")
session = FakeSession(
[
FakeResponse({"code": 0, "data": {"task_id": "task-1"}}),
FakeResponse(
{
"code": 0,
"data": {
"task_status": "succeed",
"task_result": {"videos": [{"url": "https://example.com/out.mp4"}]},
},
}
),
]
)
client = KlingClient(session=session)
task_id = client.create_classic_task("/v1/videos/text2video", {"prompt": "x"})
outputs = client.poll_classic("/v1/videos/text2video", task_id, "videos")
assert task_id == "task-1"
assert outputs == [{"url": "https://example.com/out.mp4"}]
def test_turbo_create_and_poll_parse_result_paths(monkeypatch):
monkeypatch.setenv("KLING_API_KEY", "test-key")
session = FakeSession(
[
FakeResponse({"code": 0, "data": {"id": "turbo-1"}}),
FakeResponse(
{
"code": 0,
"data": [
{
"id": "turbo-1",
"status": "succeeded",
"outputs": [{"url": "https://example.com/out.mp4"}],
}
],
}
),
]
)
client = KlingClient(session=session)
task_id = client.create_turbo("/text-to-video/kling-3.0-turbo", {"prompt": "x"})
outputs = client.poll_turbo(task_id)
assert task_id == "turbo-1"
assert outputs == [{"url": "https://example.com/out.mp4"}]
def test_schema_snapshot_contains_phase1_contract_fields():
fixture = PROJECT_ROOT / "tests/fixtures/kling_official/schema_snapshot.json"
data = json.loads(fixture.read_text())
assert data["build_id"] == "97939672"
assert "index-0m3slU3p.js" in data["chunk_names"]
assert "document-navigation-Dk7H_V3n.js" in data["chunk_names"]
assert data["api_base"]["auth_env"] == "KLING_API_KEY"
assert data["task_statuses"]["classic"] == ["submitted", "processing", "succeed", "failed"]
assert data["task_statuses"]["turbo"] == ["submitted", "processing", "succeeded", "failed"]
assert data["result_paths"]["classic_created_id"] == "data.task_id"
assert data["result_paths"]["turbo_created_id"] == "data.id"
assert "kling-v3" in data["models"]["video"]
assert "kling-v3" in data["models"]["image"]
assert data["endpoints"]["tts"]["path"] == "/v1/audio/tts"
assert data["endpoints"]["avatar_image_to_video"]["path"] == "/v1/videos/avatar/image2video"
assert data["endpoints"]["identify_face"]["path"] == "/v1/videos/identify-face"
assert data["endpoints"]["advanced_lip_sync"]["path"] == "/v1/videos/advanced-lip-sync"
assert data["endpoints"]["video_effects"]["path"] == "/v1/videos/effects"
assert data["result_paths"]["classic_audio_results"] == "data.task_result.audios[]"
assert data["result_paths"]["identify_face_session"] == "data.session_id"
assert data["result_paths"]["identify_face_results"] == "data.face_data[]"
assert data["core_field_enums"]["tts_voice_language"] == ["zh", "en"]
assert data["core_field_enums"]["avatar_mode"] == ["std", "pro"]
def test_optional_live_doc_snapshot_check():
if os.environ.get("RUN_KLING_DOC_LIVE_CHECK") != "1":
pytest.skip("Set RUN_KLING_DOC_LIVE_CHECK=1 to compare fixture against current Kling docs HTML.")
import re
import urllib.request
fixture = PROJECT_ROOT / "tests/fixtures/kling_official/schema_snapshot.json"
expected = json.loads(fixture.read_text())
with urllib.request.urlopen("https://kling.ai/document-api/api/video/3-0-turbo/text-to-video", timeout=20) as response:
html = response.read().decode("utf-8", errors="ignore")
match = re.search(r'<meta name="buildId" content="([^"]+)"', html)
assert match, "Kling official docs HTML no longer exposes buildId; refresh schema fixture."
assert match.group(1) == expected["build_id"], "Kling official docs buildId changed; refresh schema fixture before implementation."
def test_elements_helper_normalizes_and_records_metadata(tmp_path):
assert normalize_element_list([123, {"element_id": "456"}]) == [
{"element_id": 123},
{"element_id": 456},
]
artifact = write_elements_artifact(
tmp_path / "kling_elements.json",
[{"element_id": 123, "kind": "character", "name": "main-presenter"}],
)
data = json.loads(artifact.read_text())
assert data["provider"] == "kling_official"
assert data["elements"][0]["element_id"] == 123
try:
normalize_element_list([{"name": "missing-id"}])
except ValueError as exc:
assert "element_id" in str(exc)
else:
raise AssertionError("element_list items without element_id must be rejected")
def test_elements_helper_read_only_endpoints_do_not_enter_registry(isolated_tool_registry):
fake = HelperFakeClient()
assert get_custom_element(123, client=fake)["data"]["element_id"] == 123
assert list_custom_elements(client=fake)["data"][0]["element_id"] == 456
assert list_preset_elements(client=fake)["data"][0]["element_id"] == 1
assert fake.calls == [
("/v1/general/advanced-custom-elements/123", {}),
("/v1/general/advanced-custom-elements", {}),
("/v1/general/advanced-presets-elements", {}),
]
import tools._kling.elements as elements_module
assert not hasattr(elements_module, "create_element")
assert not hasattr(elements_module, "delete_element")
isolated_tool_registry.discover("tools")
assert isolated_tool_registry.get("kling_elements") is None
assert isolated_tool_registry.get("kling_account_usage") is None
def test_account_usage_helper_uses_endpoint_cache_and_throttle():
reset_account_usage_cache()
fake = HelperFakeClient()
first = get_account_costs(
start_time="2026-07-01",
end_time="2026-07-03",
client=fake,
now=100.0,
)
second = get_account_costs(
start_time="2026-07-01",
end_time="2026-07-03",
client=fake,
now=101.0,
)
throttled = get_account_costs(
resource_pack_name="different",
client=fake,
now=102.0,
)
assert fake.calls == [
("/account/costs", {"start_time": "2026-07-01", "end_time": "2026-07-03"})
]
assert first["throttle_status"] == "fresh"
assert second["cached"] is True
assert second["throttle_status"] == "cache_hit"
assert throttled["throttle_status"] == "throttled_no_cache"
def test_account_usage_cache_is_scoped_by_api_identity():
reset_account_usage_cache()
first_client = HelperFakeClient(api_key="account-a")
second_client = HelperFakeClient(api_key="account-b")
get_account_costs(client=first_client, now=100.0)
get_account_costs(client=second_client, now=111.0)
cached = get_account_costs(client=second_client, now=112.0)
assert first_client.calls == [("/account/costs", {})]
assert second_client.calls == [("/account/costs", {})]
assert cached["cached"] is True
def test_account_usage_helper_uses_kling_auth_header(monkeypatch):
reset_account_usage_cache()
monkeypatch.setenv("KLING_API_KEY", "test-key")
session = HelperFakeSession()
client = KlingClient(session=session, max_retries=0)
result = get_account_costs(
start_time="2026-07-01",
end_time="2026-07-03",
resource_pack_name="starter",
client=client,
now=200.0,
)
assert result["resource_pack_subscribe_infos"][0]["name"] == "pack-a"
method, url, kwargs = session.calls[0]
assert method == "get"
assert url.endswith("/account/costs")
assert kwargs["headers"]["Authorization"] == "Bearer test-key"
assert kwargs["params"] == {
"start_time": "2026-07-01",
"end_time": "2026-07-03",
"resource_pack_name": "starter",
}
def test_env_example_documents_kling_official_keys():
env = read(".env.example")
assert "KLING_API_KEY=" in env
assert "KLING_API_BASE_URL=" in env
def test_provider_docs_distinguish_fal_and_official_kling():
providers = read("docs/PROVIDERS.md")
assert "Kling Official" in providers
assert "kling_official_video" in providers
assert "kling_official_image" in providers
assert "kling_tts" in providers
assert "kling_avatar" in providers
assert "kling_lip_sync" in providers
assert "fal.ai" in providers
assert "provider=\"kling_official\"" in providers
assert "provider=\"kling\"" in providers
assert "Elements remain an internal Kling Official helper" in providers
assert "Account Usage is available as a low-frequency diagnostic helper" in providers
assert "callback_url" in providers
assert "audio effects and video effects are documented but intentionally not registered" in providers
def test_architecture_env_mapping_includes_kling_official():
architecture = read("docs/ARCHITECTURE.md")
assert "`KLING_API_KEY` | kling_official_video, kling_official_image, kling_tts, kling_avatar, kling_lip_sync" in architecture
assert "`KLING_API_BASE_URL` | kling_official_video, kling_official_image, kling_tts, kling_avatar, kling_lip_sync" in architecture
assert "Elements and Account" in architecture
assert "not separate pipeline stages" in architecture
assert "Kling Official also adds provider tools only where OpenMontage already has a" in architecture
def test_ai_video_skill_metadata_and_new_skill_link():
ai_video = read(".agents/skills/ai-video-gen/SKILL.md")
index = read("skills/INDEX.md")
creative = read("skills/creative/video-gen-prompting.md")
official_skill = PROJECT_ROOT / ".agents/skills/kling-official/SKILL.md"
assert "KLING_API_KEY" in ai_video
assert "kling_official_video" in ai_video
assert "kling_tts" in index
assert "avatar/lip-sync face selection" in index
assert ".agents/skills/kling-official/" in creative
assert official_skill.is_file()
official_skill_text = official_skill.read_text(encoding="utf-8")
assert "Omni References" in official_skill_text
assert "Callback Notes" in official_skill_text
assert "TTS Parameters" in official_skill_text
assert "Lip Sync Parameters" in official_skill_text
assert "Audio Effects And Video Effects" in official_skill_text
def test_provider_agent_skills_reference_kling_official():
assert "kling-official" in KlingOfficialVideo().agent_skills
assert "kling-official" in KlingOfficialImage().agent_skills
assert "kling-official" in KlingTTS().agent_skills
assert "kling-official" in KlingAvatar().agent_skills
assert "kling-official" in KlingLipSync().agent_skills
@pytest.mark.parametrize(
("tool", "output_variants"),
[
(
KlingOfficialVideo(),
{
"prompt": "changed prompt",
"operation": "image_to_video",
"api_family": "omni",
"model_name": "kling-v2-6",
"model_variant": "kling-v2-5-turbo",
"duration": "10",
"aspect_ratio": "9:16",
"resolution": "1080p",
"mode": "pro",
"sound": "on",
"negative_prompt": "blur",
"cfg_scale": 0.7,
"reference_image_url": "https://example.com/first.png",
"reference_image_path": "/tmp/first.png",
"reference_tail_image_url": "https://example.com/tail.png",
"reference_tail_image_path": "/tmp/tail.png",
"reference_image_urls": ["https://example.com/ref.png"],
"reference_image_paths": ["/tmp/ref.png"],
"reference_video_url": "https://example.com/ref.mp4",
"video_urls": ["https://example.com/ref-2.mp4"],
"image_list": [{"image_url": "https://example.com/list.png"}],
"video_list": [{"video_url": "https://example.com/list.mp4"}],
"element_list": [{"element_id": 123}],
"multi_shot": True,
"shot_type": "intelligence",
"multi_prompt": [{"prompt": "second shot", "duration": "3"}],
"camera_control": {"type": "simple", "config": {"horizontal": 1}},
"watermark": True,
},
),
(
KlingOfficialImage(),
{
"prompt": "changed prompt",
"negative_prompt": "blur",
"operation": "omni",
"api_family": "omni",
"model_name": "kling-image-o1",
"image_url": "https://example.com/source.png",
"image_path": "/tmp/source.png",
"image_urls": ["https://example.com/ref.png"],
"image_paths": ["/tmp/ref.png"],
"image_list": [{"image_url": "https://example.com/list.png"}],
"image_reference": "subject",
"image_fidelity": 0.7,
"human_fidelity": 0.8,
"resolution": "2k",
"aspect_ratio": "1:1",
"n": 2,
"result_type": "series",
"series_amount": "3",
"element_list": [{"element_id": 123}],
"watermark": True,
},
),
(
KlingAvatar(),
{
"image_url": "https://example.com/avatar.png",
"image_path": "/tmp/avatar.png",
"audio_id": "audio-a",
"sound_file": "inline-audio",
"sound_file_url": "https://example.com/audio.mp3",
"sound_file_path": "/tmp/audio.mp3",
"audio_path": "/tmp/audio-alias.mp3",
"prompt": "natural presenter motion",
"mode": "pro",
},
),
(
KlingLipSync(),
{
"operation": "full_lip_sync",
"video_id": "video-a",
"video_url": "https://example.com/video.mp4",
"session_id": "session-a",
"face_id": "face-a",
"face_choose": [{"face_id": "face-a", "audio_id": "audio-a"}],
"auto_select_face": True,
"audio_id": "audio-a",
"sound_file": "inline-audio",
"sound_file_url": "https://example.com/audio.mp3",
"sound_file_path": "/tmp/audio.mp3",
"audio_path": "/tmp/audio-alias.mp3",
"sound_start_time": 100,
"sound_end_time": 4100,
"sound_insert_time": 500,
"sound_volume": 1.2,
"original_audio_volume": 0.4,
},
),
],
)
def test_kling_output_inputs_change_idempotency_keys(tool, output_variants):
baseline = tool.idempotency_key({})
collisions = [
field
for field, value in output_variants.items()
if tool.idempotency_key({field: value}) == baseline
]
assert collisions == [], f"{tool.name} idempotency key ignores: {collisions}"
def test_phase3_does_not_register_audio_or_video_effect_tools(isolated_tool_registry):
isolated_tool_registry.discover("tools")
assert isolated_tool_registry.get("kling_audio") is None
assert isolated_tool_registry.get("kling_effects") is None

92
tests/contracts/test_kling_official_e2e_script.py Normal file
View File

@@ -0,0 +1,92 @@
"""Contract tests for the Kling official E2E smoke script."""
from __future__ import annotations
import importlib.util
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
SCRIPT_PATH = PROJECT_ROOT / "scripts" / "kling_official_animated_explainer_e2e.py"
def _load_script():
spec = importlib.util.spec_from_file_location("kling_official_animated_explainer_e2e", SCRIPT_PATH)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_env_status_redacts_secret_values():
script = _load_script()
status = script._env_status(
{
"KLING_API_KEY": "secret-token",
"KLING_API_BASE_URL": "https://api-beijing.klingai.com",
"FAL_KEY": "",
}
)
assert status["KLING_API_KEY"]["present"] is True
assert status["KLING_API_KEY"]["display"] == "<set:12 chars>"
assert "secret-token" not in repr(status)
assert status["KLING_API_BASE_URL"]["display"] == "https://api-beijing.klingai.com"
assert status["FAL_KEY"]["present"] is False
def test_cli_modes_are_explicit_and_non_paid_by_default():
script = _load_script()
assert script._execution_mode(script._parse_args([])) == "dry_run"
assert script._execution_mode(script._parse_args(["--live-tts"])) == "live_tts"
assert script._execution_mode(script._parse_args(["--live-full"])) == "live_full"
assert script._execution_mode(script._parse_args(["--live"])) == "live_full"
assert script._execution_mode(script._parse_args(["--live-avatar"])) == "live_avatar"
assert script._execution_mode(script._parse_args(["--live-all"])) == "live_all"
def test_video_duration_aligns_to_narration_within_kling_limits():
script = _load_script()
assert script._aligned_video_duration("3", 6.05) == "7"
assert script._aligned_video_duration("10", 6.05) == "10"
assert script._aligned_video_duration("3", None) == "3"
assert script._aligned_video_duration("3", 30.0) == "15"
def test_live_all_combines_core_and_avatar_results(monkeypatch, tmp_path):
script = _load_script()
core = {
"artifacts": {"narration": "narration.mp3", "final": "final.mp4"},
"ffprobe": {"final": {"duration": 6.0}},
"estimated_cost_usd": 0.25,
}
avatar_suite = {
"artifacts": {"avatar": "avatar.mp4", "lip_sync": "lip.mp4"},
"ffprobe": {"avatar": {"duration": 7.0}, "lip_sync": {"duration": 7.0}},
"estimated_cost_usd": 0.75,
}
monkeypatch.setattr(script, "_run_live_full", lambda *args, **kwargs: core)
monkeypatch.setattr(
script, "_run_live_avatar_suite", lambda *args, **kwargs: avatar_suite
)
result = script._run_live_all(
tmp_path,
voice_id="voice-a",
voice_language="en",
voice_speed=1.0,
text="hello",
timeout_seconds=30,
poll_interval=1.0,
include_account_usage=False,
video_duration="3",
)
assert result["core"] is core
assert result["avatar_suite"] is avatar_suite
assert result["artifacts"]["final"] == "final.mp4"
assert result["artifacts"]["lip_sync"] == "lip.mp4"
assert result["estimated_cost_usd"] == 1.0

294
tests/contracts/test_kling_official_image.py Normal file
View File

@@ -0,0 +1,294 @@
"""Contract tests for the Kling official image provider."""
from __future__ import annotations
import base64
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
from tools.graphics.kling_official_image import KlingOfficialImage
def test_registry_discovers_kling_official_image(monkeypatch, isolated_tool_registry):
monkeypatch.delenv("KLING_API_KEY", raising=False)
isolated_tool_registry.discover("tools")
tool = isolated_tool_registry.get("kling_official_image")
assert tool is not None
assert tool.capability == "image_generation"
assert tool.provider == "kling_official"
def test_image_schema_and_skill():
tool = KlingOfficialImage()
props = tool.input_schema["properties"]
assert "image_url" in props
assert "api_family" in props
assert "kling-official" in tool.agent_skills
def test_generation_payload():
tool = KlingOfficialImage()
request = tool._build_request(
{
"prompt": "portrait of a launch engineer",
"negative_prompt": "blurry",
"api_family": "generation",
"resolution": "2k",
"n": 2,
"watermark": False,
}
)
assert request["path"] == "/v1/images/generations"
assert request["payload"]["model_name"] == "kling-v3"
assert request["payload"]["negative_prompt"] == "blurry"
assert request["payload"]["resolution"] == "2k"
assert request["payload"]["n"] == 2
assert request["payload"]["watermark_info"] == {"enabled": False}
def test_edit_payload_converts_image_path_to_base64(tmp_path):
image_path = tmp_path / "subject.png"
image_path.write_bytes(b"subject")
tool = KlingOfficialImage()
request = tool._build_request(
{
"prompt": "keep the subject, change background",
"generation_mode": "edit",
"image_path": str(image_path),
"image_reference": "subject",
}
)
assert request["path"] == "/v1/images/generations"
assert request["payload"]["image"] == base64.b64encode(b"subject").decode("ascii")
assert request["payload"]["image_reference"] == "subject"
def test_omni_payload_uses_image_list(tmp_path):
image_path = tmp_path / "ref.png"
image_path.write_bytes(b"ref")
tool = KlingOfficialImage()
request = tool._build_request(
{
"prompt": "combine <<<image_1>>> with neon product lighting",
"api_family": "omni",
"image_urls": ["https://example.com/ref-a.png"],
"image_paths": [str(image_path)],
"result_type": "series",
"series_amount": "3",
}
)
assert request["path"] == "/v1/images/omni-image"
assert request["payload"]["model_name"] == "kling-image-o1"
assert request["payload"]["image_list"][0] == {"image": "https://example.com/ref-a.png"}
assert request["payload"]["image_list"][1] == {"image": base64.b64encode(b"ref").decode("ascii")}
assert request["payload"]["series_amount"] == "3"
assert request["references_used"][0]["placeholder"] == "<<<image_1>>>"
def test_omni_prompt_helper_adds_placeholders_and_validates_counts():
tool = KlingOfficialImage()
request = tool._build_request(
{
"prompt": "combine these into one scene",
"api_family": "omni",
"image_urls": ["https://example.com/a.png", "https://example.com/b.png"],
}
)
assert "<<<image_1>>> <<<image_2>>>" in request["payload"]["prompt"]
assert request["references_used"][1]["source"] == "https://example.com/b.png"
existing = tool._build_request(
{
"prompt": "keep <<<image_1>>> as the subject",
"api_family": "omni",
"image_urls": ["https://example.com/a.png"],
}
)
assert existing["payload"]["prompt"].count("<<<image_1>>>") == 1
try:
tool._build_request(
{
"prompt": "use <<<image_2>>>",
"api_family": "omni",
"image_urls": ["https://example.com/a.png"],
}
)
except ValueError as exc:
assert "only 1 image" in str(exc)
else:
raise AssertionError("Image Omni placeholders must match provided image count")
def test_image_omni_element_list_and_callback_payload():
tool = KlingOfficialImage()
request = tool._build_request(
{
"prompt": "render with element",
"api_family": "omni",
"element_list": [{"element_id": "321"}],
"callback_url": "https://example.com/callback",
}
)
assert request["payload"]["element_list"] == [{"element_id": 321}]
assert request["payload"]["callback_url"] == "https://example.com/callback"
assert request["element_ids"] == [321]
try:
tool._build_request(
{
"prompt": "bad callback",
"api_family": "omni",
"callback_url": "ftp://example.com/callback",
}
)
except ValueError as exc:
assert "callback_url" in str(exc)
else:
raise AssertionError("callback_url must be an absolute http(s) URL")
def test_image_model_must_match_api_family():
tool = KlingOfficialImage()
try:
tool._build_request(
{
"prompt": "generation with omni model",
"api_family": "generation",
"model_name": "kling-image-o1",
}
)
except ValueError as exc:
assert "api_family=generation" in str(exc)
else:
raise AssertionError("generation requests must reject omni image models")
try:
tool._build_request(
{
"prompt": "omni with generation model",
"api_family": "omni",
"model_name": "kling-v3",
}
)
except ValueError as exc:
assert "api_family=omni" in str(exc)
else:
raise AssertionError("omni requests must reject generation image models")
def test_execute_downloads_all_image_results(monkeypatch, tmp_path):
class FakeClient:
def create_classic_task(self, path, payload):
return "img-task-1"
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
return [
{"url": "https://example.com/a.png"},
{"url": "https://example.com/b.png"},
]
def download(self, url, output_path):
output_path.write_bytes(url.encode("utf-8"))
return output_path
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.graphics.kling_official_image.KlingClient", lambda: FakeClient())
output_path = tmp_path / "image.png"
result = KlingOfficialImage().execute({"prompt": "x", "n": 2, "output_path": str(output_path)})
assert result.success
assert result.data["provider"] == "kling_official"
assert result.data["task_id"] == "img-task-1"
assert result.data["remote_outputs"][0]["url"].endswith("a.png")
assert len(result.artifacts) == 2
assert Path(result.artifacts[0]).read_bytes() == b"https://example.com/a.png"
assert Path(result.artifacts[1]).read_bytes() == b"https://example.com/b.png"
assert result.cost_usd > 0
def test_execute_image_omni_series_records_references_callback_and_artifacts(monkeypatch, tmp_path):
class FakeClient:
def create_classic_task(self, path, payload):
self.path = path
self.payload = payload
return "omni-img-task-1"
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
return [
{"url": "https://example.com/series-a.png"},
{"url": "https://example.com/series-b.png"},
]
def download(self, url, output_path):
output_path.write_bytes(url.encode("utf-8"))
return output_path
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.graphics.kling_official_image.KlingClient", lambda: FakeClient())
result = KlingOfficialImage().execute(
{
"prompt": "series from references",
"api_family": "omni",
"image_urls": ["https://example.com/ref.png"],
"element_list": [654],
"result_type": "series",
"series_amount": "2",
"callback_url": "https://example.com/callback",
"output_path": str(tmp_path / "series.png"),
}
)
assert result.success
assert result.data["api_family"] == "omni"
assert result.data["remote_outputs"][1]["url"].endswith("series-b.png")
assert result.data["references_used"][0]["placeholder"] == "<<<image_1>>>"
assert result.data["element_ids"] == [654]
assert result.data["callback_requested"] is True
assert result.data["polling_used"] is True
assert len(result.artifacts) == 2
assert Path(result.artifacts[1]).name == "series_2.png"
def test_image_selector_prefers_official_provider(monkeypatch, isolated_tool_registry):
monkeypatch.setenv("KLING_API_KEY", "test-key")
isolated_tool_registry.discover("tools")
def fake_execute(self, inputs):
from tools.base_tool import ToolResult
return ToolResult(success=True, data={"output_path": "out.png"}, artifacts=["out.png"])
monkeypatch.setattr(KlingOfficialImage, "execute", fake_execute)
result = isolated_tool_registry.get("image_selector").execute(
{
"prompt": "official image",
"preferred_provider": "kling_official",
"allowed_providers": ["kling_official"],
"api_family": "omni",
"image_reference": "subject",
}
)
assert result.success
assert result.data["selected_provider"] == "kling_official"
def test_image_cost_estimate_is_not_zero():
tool = KlingOfficialImage()
assert tool.estimate_cost({"prompt": "x"}) > 0
base = tool.estimate_cost({"prompt": "x", "api_family": "omni"})
series = tool.estimate_cost(
{
"prompt": "x",
"api_family": "omni",
"result_type": "series",
"series_amount": "3",
"resolution": "4k",
"image_urls": ["https://example.com/a.png", "https://example.com/b.png"],
}
)
assert series > base
dry_run = tool.dry_run({"prompt": "x"})
assert dry_run["cost_estimate_confidence"] == "low"

419
tests/contracts/test_kling_official_video.py Normal file
View File

@@ -0,0 +1,419 @@
"""Contract tests for the Kling official video provider."""
from __future__ import annotations
import base64
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
from tools._kling.account import reset_account_usage_cache
from tools._kling.errors import KlingAPIError
from tools.video.kling_official_video import KlingOfficialVideo
def test_registry_discovers_kling_official_video(monkeypatch, isolated_tool_registry):
monkeypatch.delenv("KLING_API_KEY", raising=False)
isolated_tool_registry.discover("tools")
tool = isolated_tool_registry.get("kling_official_video")
assert tool is not None
assert tool.capability == "video_generation"
assert tool.provider == "kling_official"
def test_video_schema_has_no_top_level_image_url_and_has_skill():
tool = KlingOfficialVideo()
props = tool.input_schema["properties"]
assert "image_url" not in props
assert "reference_image_url" in props
assert "kling-official" in tool.agent_skills
assert "ai-video-gen" in tool.agent_skills
def test_classic_text_to_video_payload():
tool = KlingOfficialVideo()
request = tool._build_request(
{
"prompt": "cinematic robot walking through rain",
"api_family": "classic",
"operation": "text_to_video",
"duration": "5",
"aspect_ratio": "9:16",
"watermark": False,
}
)
assert request["path"] == "/v1/videos/text2video"
assert request["protocol"] == "classic"
assert request["payload"]["model_name"] == "kling-v3"
assert request["payload"]["prompt"].startswith("cinematic")
assert request["payload"]["aspect_ratio"] == "9:16"
assert request["payload"]["watermark_info"] == {"enabled": False}
def test_classic_image_to_video_uses_reference_image_path(tmp_path):
image_path = tmp_path / "ref.png"
image_path.write_bytes(b"fake-image")
tool = KlingOfficialVideo()
request = tool._build_request(
{
"prompt": "animate the frame",
"api_family": "classic",
"operation": "image_to_video",
"reference_image_path": str(image_path),
}
)
assert request["path"] == "/v1/videos/image2video"
assert request["payload"]["image"] == base64.b64encode(b"fake-image").decode("ascii")
assert "aspect_ratio" not in request["payload"]
def test_turbo_payloads():
tool = KlingOfficialVideo()
text_request = tool._build_request(
{
"prompt": "fast product reveal",
"api_family": "turbo",
"operation": "text_to_video",
"duration": "6",
"resolution": "1080p",
}
)
assert text_request["path"] == "/text-to-video/kling-3.0-turbo"
assert text_request["payload"]["settings"] == {
"resolution": "1080p",
"duration": 6,
"aspect_ratio": "16:9",
}
image_request = tool._build_request(
{
"prompt": "animate the product",
"api_family": "turbo",
"operation": "image_to_video",
"reference_image_url": "https://example.com/ref.png",
}
)
assert image_request["path"] == "/image-to-video/kling-3.0-turbo"
assert image_request["payload"]["contents"] == [
{"type": "prompt", "text": "animate the product"},
{"type": "first_frame", "url": "https://example.com/ref.png"},
]
def test_omni_reference_payload():
tool = KlingOfficialVideo()
request = tool._build_request(
{
"prompt": "match the motion and mood",
"api_family": "omni",
"operation": "reference_to_video",
"video_list": [{"video_url": "https://example.com/ref.mp4", "refer_type": "base"}],
}
)
assert request["path"] == "/v1/videos/omni-video"
assert request["payload"]["model_name"] == "kling-video-o1"
assert request["payload"]["video_list"][0]["video_url"].endswith("ref.mp4")
def test_video_omni_payload_supports_multi_refs_elements_and_multi_prompt(tmp_path):
image_path = tmp_path / "local.png"
image_path.write_bytes(b"local-ref")
tool = KlingOfficialVideo()
request = tool._build_request(
{
"prompt": "two-shot brand reveal",
"api_family": "omni",
"operation": "reference_to_video",
"image_list": [{"image_url": "https://example.com/start.png", "type": "first_frame"}],
"reference_tail_image_path": str(image_path),
"video_list": [
{
"video_url": "https://example.com/motion.mp4",
"refer_type": "feature",
"keep_original_sound": True,
}
],
"element_list": [123, {"element_id": "456"}],
"multi_shot": True,
"shot_type": "customize",
"multi_prompt": [
{"prompt": "wide product intro", "duration": "5"},
{"prompt": "close detail pass", "camera_control": {"type": "simple"}},
],
}
)
payload = request["payload"]
assert payload["image_list"][0] == {"image_url": "https://example.com/start.png", "type": "first_frame"}
assert payload["image_list"][1] == {
"image_url": base64.b64encode(b"local-ref").decode("ascii"),
"type": "end_frame",
}
assert payload["video_list"] == [
{
"video_url": "https://example.com/motion.mp4",
"refer_type": "feature",
"keep_original_sound": "yes",
}
]
assert payload["element_list"] == [{"element_id": 123}, {"element_id": 456}]
assert payload["multi_shot"] is True
assert payload["multi_prompt"][1]["camera_control"] == {"type": "simple"}
assert request["element_ids"] == [123, 456]
assert any(item["kind"] == "element" for item in request["references_used"])
def test_video_omni_requires_reference_input_and_rejects_local_video_paths():
tool = KlingOfficialVideo()
try:
tool._build_request(
{
"prompt": "needs a reference",
"api_family": "omni",
"operation": "reference_to_video",
}
)
except ValueError as exc:
assert "requires image_list, video_list, element_list" in str(exc)
else:
raise AssertionError("reference_to_video must require at least one Omni reference")
try:
tool._build_request(
{
"prompt": "local video",
"api_family": "omni",
"operation": "reference_to_video",
"reference_video_path": "/tmp/ref.mp4",
}
)
except ValueError as exc:
assert "local video paths cannot be silently uploaded" in str(exc)
else:
raise AssertionError("Video Omni must not silently upload local videos")
def test_video_callback_payloads_and_validation():
tool = KlingOfficialVideo()
classic = tool._build_request(
{
"prompt": "callback classic",
"api_family": "classic",
"operation": "text_to_video",
"callback_url": "https://example.com/kling/callback",
}
)
assert classic["payload"]["callback_url"] == "https://example.com/kling/callback"
turbo = tool._build_request(
{
"prompt": "callback turbo",
"api_family": "turbo",
"operation": "text_to_video",
"callback_url": "https://example.com/kling/callback",
}
)
assert turbo["payload"]["options"]["callback_url"] == "https://example.com/kling/callback"
omni = tool._build_request(
{
"prompt": "callback omni",
"api_family": "omni",
"operation": "reference_to_video",
"video_list": [{"video_url": "https://example.com/ref.mp4"}],
"callback_url": "https://example.com/kling/callback",
}
)
assert omni["payload"]["callback_url"] == "https://example.com/kling/callback"
try:
tool._build_request(
{
"prompt": "bad callback",
"api_family": "classic",
"operation": "text_to_video",
"callback_url": "not-a-url",
}
)
except ValueError as exc:
assert "callback_url" in str(exc)
else:
raise AssertionError("callback_url must be validated before sending")
def test_video_model_must_match_api_family():
tool = KlingOfficialVideo()
try:
tool._build_request(
{
"prompt": "classic request with omni model",
"api_family": "classic",
"operation": "text_to_video",
"model_name": "kling-video-o1",
}
)
except ValueError as exc:
assert "api_family=classic" in str(exc)
else:
raise AssertionError("classic requests must reject omni video models")
try:
tool._build_request(
{
"prompt": "omni request with classic model",
"api_family": "omni",
"operation": "reference_to_video",
"model_name": "kling-v3",
"video_list": [{"video_url": "https://example.com/ref.mp4"}],
}
)
except ValueError as exc:
assert "api_family=omni" in str(exc)
else:
raise AssertionError("omni requests must reject classic video models")
def test_execute_downloads_video_and_returns_artifact(monkeypatch, tmp_path):
class FakeClient:
def create_classic_task(self, path, payload):
self.path = path
self.payload = payload
return "task-1"
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
return [{"url": "https://example.com/out.mp4"}]
def download(self, url, output_path):
output_path.write_bytes(b"video")
return output_path
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.video.kling_official_video.KlingClient", lambda: FakeClient())
monkeypatch.setattr("tools.video.kling_official_video.probe_output", lambda path: {"duration_seconds": 5.0})
output_path = tmp_path / "out.mp4"
result = KlingOfficialVideo().execute({"prompt": "x", "output_path": str(output_path)})
assert result.success
assert result.data["provider"] == "kling_official"
assert result.data["task_id"] == "task-1"
assert result.artifacts == [str(output_path)]
assert output_path.read_bytes() == b"video"
assert result.cost_usd > 0
def test_execute_downloads_all_omni_video_outputs_and_records_metadata(monkeypatch, tmp_path):
reset_account_usage_cache()
class FakeClient:
def create_classic_task(self, path, payload):
self.path = path
self.payload = payload
return "omni-task-1"
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
return [
{"url": "https://example.com/a.mp4"},
{"url": "https://example.com/b.mp4"},
]
def download(self, url, output_path):
output_path.write_bytes(url.encode("utf-8"))
return output_path
def get(self, path, params=None):
assert path == "/account/costs"
return {"code": 0, "data": {"resource_pack_subscribe_infos": [{"name": "pack-a"}]}}
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.video.kling_official_video.KlingClient", lambda: FakeClient())
monkeypatch.setattr("tools.video.kling_official_video.probe_output", lambda path: {"duration_seconds": 5.0})
result = KlingOfficialVideo().execute(
{
"prompt": "omni",
"api_family": "omni",
"operation": "reference_to_video",
"video_list": [{"video_url": "https://example.com/ref.mp4", "refer_type": "base"}],
"element_list": [789],
"callback_url": "https://example.com/callback",
"include_account_usage": True,
"output_path": str(tmp_path / "out.mp4"),
}
)
assert result.success
assert result.data["api_family"] == "omni"
assert result.data["remote_outputs"][1]["url"].endswith("b.mp4")
assert result.data["element_ids"] == [789]
assert result.data["callback_requested"] is True
assert result.data["polling_used"] is True
assert result.data["account_usage"]["resource_pack_subscribe_infos"][0]["name"] == "pack-a"
assert result.data["cost_source"] == "estimate_with_account_usage_context"
assert result.cost_usd > 0
assert len(result.artifacts) == 2
assert Path(result.artifacts[1]).name == "out_2.mp4"
def test_video_selector_prefers_official_provider_without_fal_upload(
monkeypatch, tmp_path, isolated_tool_registry
):
monkeypatch.setenv("KLING_API_KEY", "test-key")
image_path = tmp_path / "ref.png"
image_path.write_bytes(b"fake")
isolated_tool_registry.discover("tools")
seen = {}
def fake_execute(self, inputs):
seen.update(inputs)
from tools.base_tool import ToolResult
return ToolResult(success=True, data={"output_path": "out.mp4"}, artifacts=["out.mp4"])
def fail_upload(path):
raise AssertionError("fal.ai upload should not be called for kling_official_video")
monkeypatch.setattr(KlingOfficialVideo, "execute", fake_execute)
monkeypatch.setattr("tools.video._shared.upload_image_fal", fail_upload)
result = isolated_tool_registry.get("video_selector").execute(
{
"prompt": "animate",
"operation": "image_to_video",
"preferred_provider": "kling_official",
"allowed_providers": ["kling_official"],
"reference_image_path": str(image_path),
}
)
assert result.success
assert result.data["selected_provider"] == "kling_official"
assert seen["reference_image_path"] == str(image_path)
def test_video_cost_estimate_is_not_zero():
tool = KlingOfficialVideo()
assert tool.estimate_cost({"prompt": "x"}) > 0
base = tool.estimate_cost({"prompt": "x", "api_family": "omni"})
expensive = tool.estimate_cost(
{
"prompt": "x",
"api_family": "omni",
"mode": "4k",
"sound": "on",
"video_list": [{"video_url": "https://example.com/ref.mp4"}],
"element_list": [1, 2],
"multi_prompt": [{"prompt": "a"}, {"prompt": "b"}],
}
)
assert expensive > base
dry_run = tool.dry_run({"prompt": "x"})
assert dry_run["cost_estimate_confidence"] == "low"
def test_video_account_resource_error_includes_diagnostic(monkeypatch):
class FakeClient:
def create_classic_task(self, path, payload):
raise KlingAPIError("resource pack exhausted", code=1102, request_id="req-1")
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.video.kling_official_video.KlingClient", lambda: FakeClient())
result = KlingOfficialVideo().execute({"prompt": "x"})
assert not result.success
assert result.data["account_usage_diagnostic"]["reason"] == "account_balance_or_resource_pack"

172
tests/contracts/test_kling_tts.py Normal file
View File

@@ -0,0 +1,172 @@
"""Contract tests for the Kling official TTS provider."""
from __future__ import annotations
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
from tools.audio.kling_tts import KlingTTS
def test_registry_discovers_kling_tts(monkeypatch, isolated_tool_registry):
monkeypatch.delenv("KLING_API_KEY", raising=False)
isolated_tool_registry.discover("tools")
tool = isolated_tool_registry.get("kling_tts")
assert tool is not None
assert tool.capability == "tts"
assert tool.provider == "kling_official"
def test_tts_schema_and_skill_metadata():
tool = KlingTTS()
props = tool.input_schema["properties"]
assert "voice_id" in props
assert props["voice_language"]["enum"] == ["zh", "en"]
assert "kling-official" in tool.agent_skills
assert "text-to-speech" in tool.agent_skills
assert tool.estimate_cost({"text": "hello", "voice_id": "voice-a"}) > 0
assert tool.dry_run({"text": "hello", "voice_id": "voice-a"})["cost_estimate_confidence"] == "low"
def test_tts_payload_and_validation():
tool = KlingTTS()
request = tool._build_request(
{
"text": "Hello from Kling",
"voice_id": "voice-a",
"voice_language": "en",
"voice_speed": 1.2,
"callback_url": "https://example.com/kling/callback",
}
)
assert request["path"] == "/v1/audio/tts"
assert request["payload"] == {
"text": "Hello from Kling",
"voice_id": "voice-a",
"voice_language": "en",
"voice_speed": 1.2,
"callback_url": "https://example.com/kling/callback",
}
for bad_inputs in (
{"text": "missing voice"},
{"text": "x", "voice_id": "voice-a", "voice_language": "fr"},
{"text": "x", "voice_id": "voice-a", "voice_speed": 9},
):
try:
tool._build_request(bad_inputs)
except ValueError:
pass
else:
raise AssertionError(f"Invalid TTS inputs should fail: {bad_inputs}")
def test_execute_downloads_all_audio_results(monkeypatch, tmp_path):
class FakeClient:
def create_classic_task(self, path, payload):
self.path = path
self.payload = payload
return "tts-task-1"
def poll_classic(self, path, task_id, result_key, timeout_seconds, poll_interval):
assert result_key == "audios"
return [
{"url": "https://example.com/a.mp3"},
{"audio_url": "https://example.com/b.wav"},
]
def download(self, url, output_path):
output_path.write_bytes(url.encode("utf-8"))
return output_path
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.audio.kling_tts.KlingClient", lambda: FakeClient())
monkeypatch.setattr("tools.audio.kling_tts.probe_duration", lambda path: 1.23)
output_path = tmp_path / "speech.mp3"
result = KlingTTS().execute(
{
"text": "Hello",
"voice_id": "voice-a",
"output_path": str(output_path),
}
)
assert result.success
assert result.data["provider"] == "kling_official"
assert result.data["task_id"] == "tts-task-1"
assert result.data["audio_duration_seconds"] == 1.23
assert len(result.artifacts) == 2
assert Path(result.artifacts[0]).name == "speech.mp3"
assert Path(result.artifacts[1]).name == "speech_2.wav"
assert result.cost_usd > 0
def test_execute_accepts_synchronous_create_response(monkeypatch, tmp_path):
class FakeClient:
def post(self, path, payload):
assert path == "/v1/audio/tts"
assert payload["voice_id"] == "voice-a"
return {
"code": 0,
"message": "SUCCEED",
"request_id": "req-1",
"data": {
"task_id": "tts-task-sync",
"task_status": "succeed",
"task_result": {
"audios": [
{"url": "https://example.com/sync.mp3"},
]
},
},
}
def poll_classic(self, *args, **kwargs):
raise AssertionError("synchronous TTS response should not poll")
def download(self, url, output_path):
output_path.write_bytes(url.encode("utf-8"))
return output_path
monkeypatch.setenv("KLING_API_KEY", "test-key")
monkeypatch.setattr("tools.audio.kling_tts.KlingClient", lambda: FakeClient())
monkeypatch.setattr("tools.audio.kling_tts.probe_duration", lambda path: 2.5)
result = KlingTTS().execute(
{
"text": "Hello",
"voice_id": "voice-a",
"output_path": str(tmp_path / "sync.mp3"),
}
)
assert result.success
assert result.data["task_id"] == "tts-task-sync"
assert result.data["remote_outputs"] == [{"url": "https://example.com/sync.mp3"}]
assert result.data["audio_duration_seconds"] == 2.5
def test_tts_selector_prefers_kling_official(monkeypatch, isolated_tool_registry):
monkeypatch.setenv("KLING_API_KEY", "test-key")
isolated_tool_registry.discover("tools")
def fake_execute(self, inputs):
from tools.base_tool import ToolResult
return ToolResult(success=True, data={"output_path": "out.mp3"}, artifacts=["out.mp3"])
monkeypatch.setattr(KlingTTS, "execute", fake_execute)
result = isolated_tool_registry.get("tts_selector").execute(
{
"text": "official speech",
"voice_id": "voice-a",
"preferred_provider": "kling_official",
"allowed_providers": ["kling_official"],
}
)
assert result.success
assert result.data["selected_provider"] == "kling_official"

1
tests/contracts/test_phase3_contracts.py
View File

@@ -683,6 +683,7 @@ class TestCapabilityMetadata:
"doubao",
"elevenlabs",
"google_tts",
"kling_official",
"openai",
"piper",
}

228
tests/fixtures/kling_official/schema_snapshot.json vendored Normal file
View File

@@ -0,0 +1,228 @@
{
"build_id": "97939672",
"source_urls": [
"https://kling.ai/document-api/api/get-started/authentication",
"https://kling.ai/document-api/api/get-started/error-codes",
"https://kling.ai/document-api/api/get-started/concurrency-rules",
"https://kling.ai/document-api/api/video/3-0-turbo/text-to-video",
"https://kling.ai/document-api/api/video/3-0-turbo/image-to-video",
"https://kling.ai/document-api/api/video/3-0-omni/text-to-video",
"https://kling.ai/document-api/api/video/3-0-omni/image-to-video",
"https://kling.ai/document-api/api/video/3-0-omni/video-omni",
"https://kling.ai/document-api/api/image/3-0-omni/image-generation",
"https://kling.ai/document-api/api/image/3-0-omni/image-omni",
"https://kling.ai/document-api/api/video/audio-generation/text-to-audio",
"https://kling.ai/document-api/api/video/audio-generation/video-to-audio",
"https://kling.ai/document-api/api/video/avatar",
"https://kling.ai/document-api/api/video/lip-sync",
"https://kling.ai/document-api/api/video/effects"
],
"chunk_names": [
"index-0m3slU3p.js",
"document-navigation-Dk7H_V3n.js"
],
"extracted_at": "2026-07-10T08:34:20Z",
"api_base": {
"default": "https://api-singapore.klingai.com",
"env_override": "KLING_API_BASE_URL",
"auth_env": "KLING_API_KEY",
"auth_header": "Authorization: Bearer <KLING_API_KEY>"
},
"endpoints": {
"classic_text_to_video": {
"method": "POST",
"path": "/v1/videos/text2video",
"poll": "GET /v1/videos/text2video/{id}"
},
"classic_image_to_video": {
"method": "POST",
"path": "/v1/videos/image2video",
"poll": "GET /v1/videos/image2video/{id}"
},
"turbo_text_to_video": {
"method": "POST",
"path": "/text-to-video/kling-3.0-turbo",
"poll": "GET /tasks?task_ids=<id>"
},
"turbo_image_to_video": {
"method": "POST",
"path": "/image-to-video/kling-3.0-turbo",
"poll": "GET /tasks?task_ids=<id>"
},
"video_omni": {
"method": "POST",
"path": "/v1/videos/omni-video",
"poll": "GET /v1/videos/omni-video/{id}"
},
"image_generation": {
"method": "POST",
"path": "/v1/images/generations",
"poll": "GET /v1/images/generations/{id}"
},
"image_omni": {
"method": "POST",
"path": "/v1/images/omni-image",
"poll": "GET /v1/images/omni-image/{id}"
},
"tts": {
"method": "POST",
"path": "/v1/audio/tts",
"poll": "GET /v1/audio/tts/{id}"
},
"text_to_audio": {
"method": "POST",
"path": "/v1/audio/text-to-audio",
"poll": "GET /v1/audio/text-to-audio/{id}"
},
"video_to_audio": {
"method": "POST",
"path": "/v1/audio/video-to-audio",
"poll": "GET /v1/audio/video-to-audio/{id}"
},
"avatar_image_to_video": {
"method": "POST",
"path": "/v1/videos/avatar/image2video",
"poll": "GET /v1/videos/avatar/image2video/{id}"
},
"identify_face": {
"method": "POST",
"path": "/v1/videos/identify-face"
},
"advanced_lip_sync": {
"method": "POST",
"path": "/v1/videos/advanced-lip-sync",
"poll": "GET /v1/videos/advanced-lip-sync/{id}"
},
"video_effects": {
"method": "POST",
"path": "/v1/videos/effects",
"poll": "GET /v1/videos/effects/{id}"
}
},
"models": {
"video": [
"kling-v1",
"kling-v1-5",
"kling-v1-6",
"kling-v2-master",
"kling-v2-1",
"kling-v2-1-master",
"kling-v2-5-turbo",
"kling-v2-6",
"kling-v3",
"kling-video-o1",
"kling-v3-omni"
],
"image": [
"kling-v1",
"kling-v1-5",
"kling-v2",
"kling-v2-new",
"kling-v2-1",
"kling-v3",
"kling-image-o1",
"kling-v3-omni"
]
},
"task_statuses": {
"classic": [
"submitted",
"processing",
"succeed",
"failed"
],
"turbo": [
"submitted",
"processing",
"succeeded",
"failed"
]
},
"result_paths": {
"classic_created_id": "data.task_id",
"turbo_created_id": "data.id",
"classic_video_results": "data.task_result.videos[]",
"classic_image_results": "data.task_result.images[]",
"classic_audio_results": "data.task_result.audios[]",
"identify_face_session": "data.session_id",
"identify_face_results": "data.face_data[]",
"turbo_results": "data[0].outputs[]"
},
"core_field_enums": {
"aspect_ratio": [
"16:9",
"9:16",
"1:1"
],
"video_duration": [
"3",
"4",
"5",
"6",
"7",
"8",
"9",
"10",
"11",
"12",
"13",
"14",
"15"
],
"video_resolution": [
"720p",
"1080p"
],
"image_resolution": [
"1k",
"2k",
"4k"
],
"mode": [
"std",
"pro",
"4k"
],
"sound": [
"on",
"off"
],
"image_aspect_ratio": [
"16:9",
"9:16",
"1:1",
"4:3",
"3:4",
"3:2",
"2:3",
"21:9",
"auto"
],
"image_reference": [
"subject",
"face"
],
"image_result_type": [
"single",
"series"
],
"tts_voice_language": [
"zh",
"en"
],
"avatar_mode": [
"std",
"pro"
],
"lip_sync_operation": [
"identify_face",
"advanced_lip_sync",
"full_lip_sync"
]
},
"notes": [
"Official docs were fetched as a SPA on 2026-07-10. Current HTML exposes buildId 97939672.",
"Advanced lip sync currently accepts one face_choose item. The item contains face_id, audio_id or sound_file, and sound_start_time, sound_end_time, and sound_insert_time.",
"The current entry/navigation bundles no longer expose the older api-*.js chunk names as literal references, so this fixture records the current entry assets and the Phase 1 core schema facts used by implementation and tests."
]
}

6
tools/_kling/__init__.py Normal file
View File

@@ -0,0 +1,6 @@
"""Shared helpers for Kling official API providers."""
from .client import KlingClient
from .errors import KlingAPIError, is_retryable_kling_error
__all__ = ["KlingAPIError", "KlingClient", "is_retryable_kling_error"]

121
tools/_kling/account.py Normal file
View File

@@ -0,0 +1,121 @@
"""Account usage diagnostics for Kling official API.
This module is a low-frequency helper, not an OpenMontage registry tool.
"""
from __future__ import annotations
import time
from hashlib import sha256
from typing import Any
from .client import KlingClient
from .errors import KlingAPIError
_CACHE: dict[tuple[tuple[str, str], ...], dict[str, Any]] = {}
_LAST_QUERY_AT = 0.0
def reset_account_usage_cache() -> None:
"""Clear in-process account usage cache. Intended for tests."""
global _LAST_QUERY_AT
_CACHE.clear()
_LAST_QUERY_AT = 0.0
def get_account_costs(
*,
start_time: str | None = None,
end_time: str | None = None,
resource_pack_name: str | None = None,
client: KlingClient | None = None,
ttl_seconds: float = 300.0,
min_interval_seconds: float = 10.0,
now: float | None = None,
) -> dict[str, Any]:
"""Read `/account/costs` with in-process cache and throttle protection."""
global _LAST_QUERY_AT
timestamp = time.time() if now is None else now
params = {
key: value
for key, value in {
"start_time": start_time,
"end_time": end_time,
"resource_pack_name": resource_pack_name,
}.items()
if value
}
api = client or KlingClient()
key = _cache_key(api, params)
cached = _CACHE.get(key)
if cached and timestamp - float(cached["fetched_at"]) <= ttl_seconds:
return {**cached["payload"], "cached": True, "throttle_status": "cache_hit"}
if _LAST_QUERY_AT and timestamp - _LAST_QUERY_AT < min_interval_seconds:
if cached:
return {**cached["payload"], "cached": True, "throttle_status": "throttled_cache"}
return {
"provider": "kling_official",
"queried_range": {
"start_time": start_time,
"end_time": end_time,
"resource_pack_name": resource_pack_name,
},
"cached": False,
"throttle_status": "throttled_no_cache",
"message": "Account Usage is rate-limited; retry after the local throttle window.",
}
raw = api.get("/account/costs", params=params)
data = raw.get("data") if isinstance(raw, dict) else {}
if not isinstance(data, dict):
data = {}
payload = {
"provider": "kling_official",
"queried_range": {
"start_time": start_time,
"end_time": end_time,
"resource_pack_name": resource_pack_name,
},
"resource_pack_subscribe_infos": data.get("resource_pack_subscribe_infos", []),
"raw": raw,
"cached": False,
"throttle_status": "fresh",
}
_CACHE[key] = {"fetched_at": timestamp, "payload": payload}
_LAST_QUERY_AT = timestamp
return payload
def _cache_key(client: Any, params: dict[str, Any]) -> tuple[tuple[str, str], ...]:
"""Scope Account Usage cache by request params and account endpoint identity."""
api_key = getattr(client, "api_key", None) or ""
api_key_hash = sha256(str(api_key).encode("utf-8")).hexdigest() if api_key else ""
scope = {
"base_url": getattr(client, "base_url", ""),
"api_key_sha256": api_key_hash,
**{name: str(value) for name, value in params.items()},
}
return tuple(sorted((name, str(value)) for name, value in scope.items()))
def account_usage_hint_for_error(error: KlingAPIError) -> dict[str, Any]:
"""Return a diagnostic hint for balance/resource-pack related errors."""
code = str(error.code) if error.code is not None else ""
if code not in {"1101", "1102"}:
return {}
return {
"provider": "kling_official",
"reason": "account_balance_or_resource_pack",
"message": (
"Kling returned an account/resource-pack error. Use tools._kling.account.get_account_costs() "
"for a low-frequency account usage diagnostic, or check the Kling Open Platform console."
),
"error_code": error.code,
"request_id": error.request_id,
}

17
tools/_kling/callbacks.py Normal file
View File

@@ -0,0 +1,17 @@
"""Callback validation helpers for Kling official providers."""
from __future__ import annotations
from urllib.parse import urlparse
def validate_callback_url(callback_url: str | None) -> str | None:
"""Return a normalized callback URL or raise for obviously invalid input."""
if not callback_url:
return None
value = str(callback_url).strip()
parsed = urlparse(value)
if parsed.scheme not in {"http", "https"} or not parsed.netloc:
raise ValueError("callback_url must be an absolute http(s) URL")
return value

216
tools/_kling/client.py Normal file
View File

@@ -0,0 +1,216 @@
"""HTTP client and task parsers for Kling official API providers."""
from __future__ import annotations
import os
import time
from pathlib import Path
from typing import Any
from urllib.parse import urljoin
import requests
from .errors import KlingAPIError, is_retryable_kling_error
from .schemas import (
CLASSIC_FAILURE_STATUS,
CLASSIC_PENDING_STATUSES,
CLASSIC_SUCCESS_STATUS,
DEFAULT_API_BASE_URL,
TURBO_FAILURE_STATUS,
TURBO_PENDING_STATUSES,
TURBO_SUCCESS_STATUS,
)
class KlingClient:
"""Small synchronous client for the official Kling API."""
def __init__(
self,
api_key: str | None = None,
base_url: str | None = None,
session: Any | None = None,
max_retries: int = 2,
) -> None:
self.api_key = api_key if api_key is not None else os.environ.get("KLING_API_KEY")
self.base_url = (base_url or os.environ.get("KLING_API_BASE_URL") or DEFAULT_API_BASE_URL).rstrip("/")
self.session = session or requests.Session()
self.max_retries = max_retries
@property
def headers(self) -> dict[str, str]:
if not self.api_key:
raise KlingAPIError(
"KLING_API_KEY is not set. Configure KLING_API_KEY for official Kling API access.",
http_status=401,
)
return {
"Authorization": f"Bearer {self.api_key}",
"Accept": "application/json",
"Content-Type": "application/json",
}
def post(self, path: str, payload: dict[str, Any]) -> dict[str, Any]:
return self._request("post", path, json=payload)
def get(self, path: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
return self._request("get", path, params=params)
def download(self, url: str, output_path: Path, timeout: int = 180) -> Path:
output_path.parent.mkdir(parents=True, exist_ok=True)
response = self.session.get(url, timeout=timeout)
self._raise_for_http_error(response)
content = getattr(response, "content", None)
if content is None and hasattr(response, "iter_content"):
content = b"".join(chunk for chunk in response.iter_content(chunk_size=1024 * 128) if chunk)
output_path.write_bytes(content or b"")
return output_path
def create_classic_task(self, path: str, payload: dict[str, Any]) -> str:
data = self.post(path, payload)
task_id = ((data.get("data") or {}).get("task_id"))
if not task_id:
raise KlingAPIError(f"Kling Classic create response missing data.task_id: {data}")
return str(task_id)
def poll_classic(
self,
path: str,
task_id: str,
result_key: str,
timeout_seconds: int = 900,
poll_interval: float = 5.0,
) -> list[dict[str, Any]]:
deadline = time.time() + timeout_seconds
while time.time() < deadline:
data = self.get(f"{path.rstrip('/')}/{task_id}")
payload = data.get("data") or {}
status = payload.get("task_status") or payload.get("status")
if status == CLASSIC_SUCCESS_STATUS:
task_result = payload.get("task_result") or {}
outputs = task_result.get(result_key) or []
if not isinstance(outputs, list):
raise KlingAPIError(f"Kling Classic result path data.task_result.{result_key} is not a list")
return outputs
if status == CLASSIC_FAILURE_STATUS:
message = payload.get("task_status_msg") or payload.get("message") or "Kling Classic task failed"
raise KlingAPIError(str(message), code=payload.get("task_status"), response=data)
if status not in CLASSIC_PENDING_STATUSES:
raise KlingAPIError(f"Unexpected Kling Classic task status {status!r}", response=data)
time.sleep(min(poll_interval, max(0.0, deadline - time.time())))
raise TimeoutError(f"Kling Classic task {task_id} timed out after {timeout_seconds}s")
def create_turbo(self, path: str, payload: dict[str, Any]) -> str:
data = self.post(path, payload)
task_id = ((data.get("data") or {}).get("id"))
if not task_id:
raise KlingAPIError(f"Kling Turbo create response missing data.id: {data}")
return str(task_id)
def poll_turbo(
self,
task_id: str,
timeout_seconds: int = 900,
poll_interval: float = 5.0,
) -> list[dict[str, Any]]:
deadline = time.time() + timeout_seconds
while time.time() < deadline:
data = self.get("/tasks", params={"task_ids": task_id})
records = data.get("data") or []
if not records:
raise KlingAPIError(f"Kling Turbo poll response missing data[0]: {data}")
record = records[0]
status = record.get("status") or record.get("task_status")
if status == TURBO_SUCCESS_STATUS:
outputs = record.get("outputs") or []
if not isinstance(outputs, list):
raise KlingAPIError("Kling Turbo result path data[0].outputs is not a list")
return outputs
if status == TURBO_FAILURE_STATUS:
message = record.get("message") or record.get("error") or "Kling Turbo task failed"
raise KlingAPIError(str(message), code=record.get("code"), request_id=record.get("request_id"), response=data)
if status not in TURBO_PENDING_STATUSES:
raise KlingAPIError(f"Unexpected Kling Turbo task status {status!r}", response=data)
time.sleep(min(poll_interval, max(0.0, deadline - time.time())))
raise TimeoutError(f"Kling Turbo task {task_id} timed out after {timeout_seconds}s")
def _request(self, method: str, path: str, **kwargs: Any) -> dict[str, Any]:
url = self._url(path)
last_error: KlingAPIError | None = None
for attempt in range(self.max_retries + 1):
try:
response = getattr(self.session, method)(url, headers=self.headers, timeout=30, **kwargs)
self._raise_for_http_error(response)
data = response.json()
self._raise_for_business_error(data)
return data
except KlingAPIError as error:
last_error = error
if attempt >= self.max_retries or not is_retryable_kling_error(error):
raise
time.sleep(min(2.0 * (attempt + 1), 8.0))
except requests.RequestException as exc:
last_error = KlingAPIError(str(exc))
if attempt >= self.max_retries:
raise last_error from exc
time.sleep(min(2.0 * (attempt + 1), 8.0))
raise last_error or KlingAPIError("Kling API request failed")
def _url(self, path: str) -> str:
if path.startswith("http://") or path.startswith("https://"):
return path
return urljoin(f"{self.base_url}/", path.lstrip("/"))
def _raise_for_http_error(self, response: Any) -> None:
status = getattr(response, "status_code", None)
if status is not None and 200 <= int(status) < 300:
return
code = None
message = None
request_id = None
body: dict[str, Any] | None = None
try:
body = response.json()
code = body.get("code")
message = body.get("message") or body.get("msg")
request_id = body.get("request_id") or body.get("requestId")
except Exception:
text = getattr(response, "text", "")
message = text[:500] if text else f"HTTP {status}"
raise self._format_error(
code=code,
message=message or f"HTTP {status}",
request_id=request_id,
http_status=int(status) if status is not None else None,
response=body,
)
def _raise_for_business_error(self, data: dict[str, Any]) -> None:
code = data.get("code")
if code in (None, 0, "0"):
return
raise self._format_error(
code=code,
message=str(data.get("message") or data.get("msg") or "Kling API returned an error"),
request_id=data.get("request_id") or data.get("requestId"),
response=data,
)
@staticmethod
def _format_error(
*,
code: str | int | None,
message: str,
request_id: str | None = None,
http_status: int | None = None,
response: dict[str, Any] | None = None,
) -> KlingAPIError:
if str(code) == "1303" and "并发/资源包限制" not in message:
message = f"{message} (并发/资源包限制: parallel task over resource pack limit)"
return KlingAPIError(
message=message,
code=code,
request_id=request_id,
http_status=http_status,
response=response,
)

86
tools/_kling/elements.py Normal file
View File

@@ -0,0 +1,86 @@
"""Element reference helpers for Kling official Omni providers.
These helpers intentionally do not inherit from BaseTool. Elements are an
internal provider reference mechanism in Phase 2, not a registry capability.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
from .client import KlingClient
def normalize_element_list(element_list: Any | None) -> list[dict[str, int]]:
"""Normalize official Kling element references to element_list objects."""
if not element_list:
return []
if not isinstance(element_list, list):
raise ValueError("element_list must be a list of element ids or objects")
normalized: list[dict[str, int]] = []
for item in element_list:
raw_id: Any
if isinstance(item, dict):
raw_id = item.get("element_id", item.get("id"))
else:
raw_id = item
if raw_id is None:
raise ValueError("each element_list item must include element_id")
try:
element_id = int(raw_id)
except (TypeError, ValueError) as exc:
raise ValueError(f"element_id must be an integer-compatible value: {raw_id!r}") from exc
if element_id <= 0:
raise ValueError("element_id must be positive")
normalized.append({"element_id": element_id})
return normalized
def element_ids(element_list: Any | None) -> list[int]:
"""Return normalized element ids from an element reference list."""
return [item["element_id"] for item in normalize_element_list(element_list)]
def get_custom_element(element_id: int, client: KlingClient | None = None) -> dict[str, Any]:
"""Fetch one custom element for validation or diagnostics."""
api = client or KlingClient()
return api.get(f"/v1/general/advanced-custom-elements/{int(element_id)}")
def list_custom_elements(
client: KlingClient | None = None,
params: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""List custom elements without creating or deleting assets."""
api = client or KlingClient()
return api.get("/v1/general/advanced-custom-elements", params=params)
def list_preset_elements(
client: KlingClient | None = None,
params: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""List official preset elements without entering the tool registry."""
api = client or KlingClient()
return api.get("/v1/general/advanced-presets-elements", params=params)
def write_elements_artifact(
artifact_path: str | Path,
elements: list[dict[str, Any]],
) -> Path:
"""Write element metadata in the Phase 2 reproducibility artifact shape."""
path = Path(artifact_path)
path.parent.mkdir(parents=True, exist_ok=True)
payload = {"provider": "kling_official", "elements": elements}
path.write_text(json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8")
return path

46
tools/_kling/errors.py Normal file
View File

@@ -0,0 +1,46 @@
"""Error types and retry policy for Kling official API calls."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
@dataclass
class KlingAPIError(Exception):
"""Structured error returned by the Kling official API."""
message: str
code: str | int | None = None
request_id: str | None = None
http_status: int | None = None
response: dict[str, Any] | None = None
def __str__(self) -> str:
parts = [self.message]
if self.code is not None:
parts.append(f"code={self.code}")
if self.request_id:
parts.append(f"request_id={self.request_id}")
if self.http_status is not None:
parts.append(f"http_status={self.http_status}")
return " | ".join(parts)
_RETRYABLE_CODES = {"1302", "1303", "5000", "5001", "5002"}
_RETRYABLE_HTTP = {500, 503, 504}
def _code_str(code: str | int | None) -> str | None:
if code is None:
return None
return str(code)
def is_retryable_kling_error(error: KlingAPIError) -> bool:
"""Return whether an official Kling error is safe for limited retry."""
code = _code_str(error.code)
if code in _RETRYABLE_CODES:
return True
return error.http_status in _RETRYABLE_HTTP

111
tools/_kling/media.py Normal file
View File

@@ -0,0 +1,111 @@
"""Media normalization and download helpers for Kling official providers."""
from __future__ import annotations
import base64
import mimetypes
from pathlib import Path
from urllib.parse import urlparse
def strip_data_uri_prefix(value: str | None) -> str | None:
"""Return raw base64/content by removing a data URI prefix if present."""
if value is None:
return None
marker = ";base64,"
if value.startswith("data:") and marker in value:
return value.split(marker, 1)[1]
return value
def image_file_to_raw_base64(path: str | Path) -> str:
"""Read a local image file and return raw base64 without data URI prefix."""
image_path = Path(path)
if not image_path.is_file():
raise FileNotFoundError(f"Image not found: {image_path}")
return base64.b64encode(image_path.read_bytes()).decode("ascii")
def file_to_raw_base64(path: str | Path, *, label: str = "File") -> str:
"""Read a local media file and return raw base64 without a data URI prefix."""
media_path = Path(path)
if not media_path.is_file():
raise FileNotFoundError(f"{label} not found: {media_path}")
return base64.b64encode(media_path.read_bytes()).decode("ascii")
def normalize_image_input(url: str | None = None, path: str | Path | None = None) -> str | None:
"""Normalize a Kling image input to either URL or raw base64."""
if url:
return strip_data_uri_prefix(url)
if path:
return image_file_to_raw_base64(path)
return None
def normalize_media_input(
url: str | None = None,
path: str | Path | None = None,
value: str | None = None,
*,
label: str = "Media file",
) -> str | None:
"""Normalize a generic Kling media input to URL, raw base64, or raw provided value."""
if value:
return strip_data_uri_prefix(value)
if url:
return strip_data_uri_prefix(url)
if path:
return file_to_raw_base64(path, label=label)
return None
def extension_from_url(url: str | None, default: str = ".png") -> str:
"""Infer a file extension from a URL path."""
if not url:
return default
suffix = Path(urlparse(url).path).suffix.lower()
if suffix in {
".png",
".jpg",
".jpeg",
".webp",
".gif",
".mp4",
".mov",
".m4v",
".mp3",
".wav",
".m4a",
".aac",
".ogg",
".opus",
}:
return suffix
return default
def extension_from_content_type(content_type: str | None, default: str = ".png") -> str:
if not content_type:
return default
ext = mimetypes.guess_extension(content_type.split(";", 1)[0].strip())
return ext or default
def output_path_with_suffix(path: str | Path, suffix: str) -> Path:
output_path = Path(path)
if output_path.suffix:
return output_path
return output_path.with_suffix(suffix)
def numbered_output_path(first_path: Path, index: int, suffix: str) -> Path:
if index == 0:
return output_path_with_suffix(first_path, suffix)
return first_path.with_name(f"{first_path.stem}_{index + 1}{suffix}")

40
tools/_kling/omni.py Normal file
View File

@@ -0,0 +1,40 @@
"""Omni reference helpers for Kling official providers."""
from __future__ import annotations
import re
from typing import Any
PLACEHOLDER_RE = re.compile(r"<<<image_(\d+)>>>")
def build_image_prompt_references(
prompt: str,
image_list: list[dict[str, Any]],
) -> tuple[str, list[dict[str, Any]]]:
"""Bind image_list entries to stable Image Omni placeholders."""
references = [
{
"index": index,
"placeholder": f"<<<image_{index}>>>",
"source": item.get("source") or item.get("image") or item.get("image_url"),
"source_type": item.get("source_type", "unknown"),
}
for index, item in enumerate(image_list, start=1)
]
if not references:
return prompt, []
existing_numbers = [int(value) for value in PLACEHOLDER_RE.findall(prompt)]
if existing_numbers:
if max(existing_numbers) > len(references):
raise ValueError(
f"prompt references <<<image_{max(existing_numbers)}>>> but only {len(references)} image(s) were provided"
)
if min(existing_numbers) < 1:
raise ValueError("Image Omni prompt placeholders must start at <<<image_1>>>")
return prompt, references
placeholders = " ".join(item["placeholder"] for item in references)
return f"{prompt}\nReferences: {placeholders}", references

121
tools/_kling/schemas.py Normal file
View File

@@ -0,0 +1,121 @@
"""Lightweight schema constants for Kling official providers."""
from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
from typing import Any
DEFAULT_API_BASE_URL = "https://api-singapore.klingai.com"
class KlingProtocol(str, Enum):
CLASSIC = "classic"
TURBO = "turbo"
CLASSIC_PENDING_STATUSES = {"submitted", "processing"}
CLASSIC_SUCCESS_STATUS = "succeed"
CLASSIC_FAILURE_STATUS = "failed"
CLASSIC_STATUSES = [
"submitted",
"processing",
"succeed",
"failed",
]
TURBO_PENDING_STATUSES = {"submitted", "processing"}
TURBO_SUCCESS_STATUS = "succeeded"
TURBO_FAILURE_STATUS = "failed"
TURBO_STATUSES = [
"submitted",
"processing",
"succeeded",
"failed",
]
VIDEO_MODELS = [
"kling-v1",
"kling-v1-5",
"kling-v1-6",
"kling-v2-master",
"kling-v2-1",
"kling-v2-1-master",
"kling-v2-5-turbo",
"kling-v2-6",
"kling-v3",
"kling-video-o1",
"kling-v3-omni",
]
CLASSIC_VIDEO_MODELS = [
"kling-v1",
"kling-v1-5",
"kling-v1-6",
"kling-v2-master",
"kling-v2-1",
"kling-v2-1-master",
"kling-v2-5-turbo",
"kling-v2-6",
"kling-v3",
]
OMNI_VIDEO_MODELS = ["kling-video-o1", "kling-v3-omni"]
IMAGE_MODELS = [
"kling-v1",
"kling-v1-5",
"kling-v2",
"kling-v2-new",
"kling-v2-1",
"kling-v3",
"kling-image-o1",
"kling-v3-omni",
]
IMAGE_GENERATION_MODELS = [
"kling-v1",
"kling-v1-5",
"kling-v2",
"kling-v2-new",
"kling-v2-1",
"kling-v3",
]
OMNI_IMAGE_MODELS = ["kling-image-o1", "kling-v3-omni"]
VIDEO_DURATIONS = [str(value) for value in range(3, 16)]
VIDEO_ASPECT_RATIOS = ["16:9", "9:16", "1:1"]
VIDEO_RESOLUTIONS = ["720p", "1080p"]
VIDEO_MODES = ["std", "pro", "4k"]
SOUND_VALUES = ["on", "off"]
IMAGE_RESOLUTIONS = ["1k", "2k", "4k"]
IMAGE_ASPECT_RATIOS = ["16:9", "9:16", "1:1", "4:3", "3:4", "3:2", "2:3", "21:9", "auto"]
IMAGE_REFERENCE_TYPES = ["subject", "face"]
IMAGE_RESULT_TYPES = ["single", "series"]
RESULT_PATHS = {
"classic_video": "data.task_result.videos[]",
"classic_image": "data.task_result.images[]",
"classic_audio": "data.task_result.audios[]",
"turbo": "data[0].outputs[]",
}
TTS_LANGUAGES = ["zh", "en"]
TTS_SPEED_MIN = 0.5
TTS_SPEED_MAX = 2.0
AVATAR_MODES = ["std", "pro"]
LIP_SYNC_OPERATIONS = ["identify_face", "advanced_lip_sync", "full_lip_sync"]
@dataclass
class ClassicTaskResult:
task_id: str
status: str
outputs: list[dict[str, Any]]
@dataclass
class TurboTaskResult:
task_id: str
status: str
outputs: list[dict[str, Any]]

340
tools/audio/kling_tts.py Normal file
View File

@@ -0,0 +1,340 @@
"""Kling official API text-to-speech provider."""
from __future__ import annotations
import time
from pathlib import Path
from typing import Any
from tools._kling.account import account_usage_hint_for_error, get_account_costs
from tools._kling.callbacks import validate_callback_url
from tools._kling.client import KlingClient
from tools._kling.errors import KlingAPIError
from tools._kling.media import extension_from_url, numbered_output_path, output_path_with_suffix
from tools._kling.schemas import TTS_LANGUAGES, TTS_SPEED_MAX, TTS_SPEED_MIN
from tools.analysis.audio_probe import probe_duration
from tools.base_tool import (
BaseTool,
DependencyError,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolTier,
)
class KlingTTS(BaseTool):
name = "kling_tts"
version = "0.1.0"
tier = ToolTier.VOICE
capability = "tts"
provider = "kling_official"
stability = ToolStability.EXPERIMENTAL
execution_mode = ExecutionMode.SYNC
determinism = Determinism.STOCHASTIC
runtime = ToolRuntime.API
dependencies = ["env:KLING_API_KEY"]
install_instructions = (
"Set KLING_API_KEY in .env for the official Kling API. "
"Pass voice_id explicitly; OpenMontage does not guess Kling voice IDs."
)
agent_skills = ["kling-official", "text-to-speech"]
capabilities = ["text_to_speech", "voice_selection", "multilingual"]
supports = {
"multilingual": True,
"voice_selection": True,
"offline": False,
"native_audio": True,
}
best_for = [
"official Kling text-to-speech",
"Chinese or English narration when a Kling voice_id is known",
"keeping narration provider provenance inside the Kling official account",
]
not_good_for = [
"fully offline narration",
"voice cloning without a configured official voice_id",
"auto-discovering voices",
]
fallback_tools = ["doubao_tts", "elevenlabs_tts", "openai_tts", "google_tts", "piper_tts"]
input_schema = {
"type": "object",
"required": ["text", "voice_id"],
"properties": {
"text": {"type": "string"},
"voice_id": {
"type": "string",
"description": "Official Kling voice ID. Required; do not rely on an unknown default.",
},
"voice_language": {"type": "string", "enum": TTS_LANGUAGES, "default": "en"},
"voice_speed": {
"type": "number",
"minimum": TTS_SPEED_MIN,
"maximum": TTS_SPEED_MAX,
"default": 1.0,
},
"callback_url": {"type": "string"},
"external_task_id": {"type": "string"},
"include_account_usage": {
"type": "boolean",
"default": False,
"description": "Optional low-frequency account usage diagnostic; not used by default.",
},
"timeout_seconds": {"type": "integer", "default": 300},
"poll_interval": {"type": "number", "default": 3.0},
"output_path": {"type": "string"},
},
}
output_schema = {
"type": "object",
"properties": {
"output": {"type": "string"},
"output_path": {"type": "string"},
"audio_paths": {"type": "array"},
"task_id": {"type": "string"},
},
}
resource_profile = ResourceProfile(
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=100, network_required=True
)
retry_policy = RetryPolicy(
max_retries=2,
backoff_seconds=2.0,
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
)
idempotency_key_fields = ["text", "voice_id", "voice_language", "voice_speed"]
side_effects = ["paid remote generation via official Kling API", "writes audio file to output_path"]
user_visible_verification = ["Listen to generated audio for voice, language, and pacing"]
quality_score = 0.78
latency_p50_seconds = 20.0
def estimate_cost(self, inputs: dict[str, Any]) -> float:
text_length = len(str(inputs.get("text") or ""))
return round(max(text_length, 1) * 0.000018, 4)
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
return 30.0
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
result = super().dry_run(inputs)
result.update(
{
"paid_api": True,
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative character-based OpenMontage estimate pending official account-usage reconciliation.",
}
)
return result
def execute(self, inputs: dict[str, Any]) -> ToolResult:
try:
self.check_dependencies()
except DependencyError as exc:
return ToolResult(success=False, error=str(exc))
start = time.time()
try:
request = self._build_request(inputs)
client = KlingClient()
task_id, outputs = self._create_and_collect_audios(client, request, inputs)
paths = self._download_audios(client, outputs, inputs)
audio_duration = probe_duration(paths[0])
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
data: dict[str, Any] = {"provider": self.provider}
if isinstance(exc, KlingAPIError):
data.update(
{
"error_code": exc.code,
"request_id": exc.request_id,
"http_status": exc.http_status,
"account_usage_diagnostic": account_usage_hint_for_error(exc),
}
)
return ToolResult(success=False, data=data, error=f"Kling official TTS failed: {exc}")
except Exception as exc:
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official TTS failed: {exc}")
return ToolResult(
success=True,
data={
"provider": self.provider,
"model": "kling-official-tts",
"task_id": task_id,
"operation": "text_to_speech",
"text_length": len(request["payload"]["text"]),
"voice_id": request["payload"]["voice_id"],
"voice_language": request["payload"].get("voice_language"),
"voice_speed": request["payload"].get("voice_speed"),
"remote_outputs": outputs,
"output": str(paths[0]),
"output_path": str(paths[0]),
"audio_paths": [str(path) for path in paths],
"format": paths[0].suffix.lstrip(".") or "mp3",
"audio_duration_seconds": round(audio_duration, 2) if audio_duration else None,
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
**self._account_usage_result(inputs, client),
**self._callback_result_data(inputs, task_id),
},
artifacts=[str(path) for path in paths],
cost_usd=self.estimate_cost(inputs),
duration_seconds=round(time.time() - start, 2),
model="kling-official-tts",
)
def _build_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
text = str(inputs.get("text") or "").strip()
if not text:
raise ValueError("text is required")
if len(text) > 5000:
raise ValueError("text exceeds Kling TTS safety limit of 5000 characters")
voice_id = str(inputs.get("voice_id") or "").strip()
if not voice_id:
raise ValueError("voice_id is required for Kling official TTS")
voice_language = str(inputs.get("voice_language") or "en")
if voice_language not in TTS_LANGUAGES:
raise ValueError(f"voice_language must be one of: {', '.join(TTS_LANGUAGES)}")
voice_speed = float(inputs.get("voice_speed", 1.0))
if voice_speed < TTS_SPEED_MIN or voice_speed > TTS_SPEED_MAX:
raise ValueError(f"voice_speed must be between {TTS_SPEED_MIN} and {TTS_SPEED_MAX}")
payload: dict[str, Any] = {
"text": text,
"voice_id": voice_id,
"voice_language": voice_language,
"voice_speed": voice_speed,
}
self._copy_common_task_fields(inputs, payload)
return {
"protocol": "classic",
"path": "/v1/audio/tts",
"payload": payload,
"operation": "text_to_speech",
"model": "kling-official-tts",
}
@staticmethod
def _create_and_collect_audios(
client: KlingClient,
request: dict[str, Any],
inputs: dict[str, Any],
) -> tuple[str, list[dict[str, Any]]]:
"""Create a TTS task and return audio outputs.
Official TTS may return a completed task and task_result.audios[]
directly from POST /v1/audio/tts. Older/async behavior still requires
polling GET /v1/audio/tts/{task_id}, so support both shapes.
"""
if hasattr(client, "post"):
data = client.post(request["path"], request["payload"])
payload = data.get("data") or {}
task_id = payload.get("task_id")
if not task_id:
raise KlingAPIError(f"Kling TTS create response missing data.task_id: {data}")
task_result = payload.get("task_result") or {}
outputs = task_result.get("audios")
if outputs is not None:
if not isinstance(outputs, list):
raise KlingAPIError("Kling TTS result path data.task_result.audios is not a list")
return str(task_id), outputs
status = payload.get("task_status") or payload.get("status")
if status == "failed":
message = payload.get("task_status_msg") or payload.get("message") or "Kling TTS task failed"
raise KlingAPIError(str(message), code=payload.get("task_status"), response=data)
return str(task_id), client.poll_classic(
request["path"],
str(task_id),
"audios",
timeout_seconds=int(inputs.get("timeout_seconds", 300)),
poll_interval=float(inputs.get("poll_interval", 3.0)),
)
task_id = client.create_classic_task(request["path"], request["payload"])
return task_id, client.poll_classic(
request["path"],
task_id,
"audios",
timeout_seconds=int(inputs.get("timeout_seconds", 300)),
poll_interval=float(inputs.get("poll_interval", 3.0)),
)
def _download_audios(
self,
client: KlingClient,
outputs: list[dict[str, Any]],
inputs: dict[str, Any],
) -> list[Path]:
if not outputs:
raise ValueError("Kling TTS response contained no audios")
base_path = Path(inputs.get("output_path", "kling_tts.mp3"))
paths: list[Path] = []
for index, item in enumerate(outputs):
url = self._output_url(item)
suffix = extension_from_url(url, ".mp3")
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
client.download(url, output_path)
paths.append(output_path)
return paths
@staticmethod
def _output_url(item: dict[str, Any]) -> str:
url = item.get("url") or item.get("audio_url") or item.get("resource_url")
if url:
return str(url)
resource = item.get("resource") or {}
if isinstance(resource, dict) and resource.get("url"):
return str(resource["url"])
raise ValueError(f"Kling TTS response item contained no downloadable URL: {item}")
@staticmethod
def _copy_common_task_fields(inputs: dict[str, Any], payload: dict[str, Any]) -> None:
callback_url = validate_callback_url(inputs.get("callback_url"))
if callback_url:
payload["callback_url"] = callback_url
if inputs.get("external_task_id"):
payload["external_task_id"] = inputs["external_task_id"]
@staticmethod
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
callback_url = inputs.get("callback_url")
if not callback_url:
return {}
return {
"callback_url": str(callback_url),
"callback_requested": True,
"polling_used": True,
"task_id": task_id,
}
@staticmethod
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
if not inputs.get("include_account_usage"):
return {}
try:
usage = get_account_costs(client=client)
return {
"account_usage": usage,
"cost_source": "estimate_with_account_usage_context",
"reconciled_cost_usd": None,
}
except Exception as exc:
return {
"account_usage_error": str(exc),
"cost_source": "estimate",
"reconciled_cost_usd": None,
}

11
tools/audio/tts_selector.py
View File

@@ -45,6 +45,17 @@ class TTSSelector(BaseTool):
"type": "string",
"description": "Provider-specific voice ID. Passed through to the selected TTS provider.",
},
"voice_language": {
"type": "string",
"enum": ["zh", "en"],
"description": "Kling official voice language. Passed through when selected provider supports it.",
},
"voice_speed": {
"type": "number",
"minimum": 0.5,
"maximum": 2.0,
"description": "Kling official voice speed. Use speed for OpenAI/ElevenLabs-style controls.",
},
"model_id": {
"type": "string",
"description": "TTS model to use (e.g. eleven_multilingual_v2). Passed through to provider.",

342
tools/avatar/kling_avatar.py Normal file
View File

@@ -0,0 +1,342 @@
"""Kling official API avatar image-to-video provider."""
from __future__ import annotations
import time
from pathlib import Path
from typing import Any
from tools._kling.account import account_usage_hint_for_error, get_account_costs
from tools._kling.callbacks import validate_callback_url
from tools._kling.client import KlingClient
from tools._kling.errors import KlingAPIError
from tools._kling.media import (
extension_from_url,
normalize_image_input,
normalize_media_input,
numbered_output_path,
output_path_with_suffix,
)
from tools._kling.schemas import AVATAR_MODES
from tools.base_tool import (
BaseTool,
DependencyError,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolTier,
)
from tools.video._shared import probe_output
class KlingAvatar(BaseTool):
name = "kling_avatar"
version = "0.1.0"
tier = ToolTier.GENERATE
capability = "avatar"
provider = "kling_official"
stability = ToolStability.EXPERIMENTAL
execution_mode = ExecutionMode.SYNC
determinism = Determinism.STOCHASTIC
runtime = ToolRuntime.API
dependencies = ["env:KLING_API_KEY"]
install_instructions = (
"Set KLING_API_KEY in .env for the official Kling API. "
"Provide an avatar image plus either audio_id or sound_file/audio_path."
)
agent_skills = ["kling-official", "avatar-video"]
capabilities = ["photo_to_video", "avatar_video", "audio_driven_avatar"]
supports = {
"photo_to_video": True,
"audio_driven_animation": True,
"offline": False,
"cloud_render": True,
}
best_for = [
"official Kling cloud avatar presenter clips",
"high-quality image-to-video avatar generation from a supplied portrait",
"projects already using a Kling official account and resource pack",
]
not_good_for = [
"fully offline avatar generation",
"free local drafts",
"silently replacing the local talking_head provider",
]
fallback_tools = ["talking_head", "lip_sync"]
input_schema = {
"type": "object",
"anyOf": [
{"required": ["image_url"]},
{"required": ["image_path"]},
],
"allOf": [
{
"anyOf": [
{"required": ["audio_id"]},
{"required": ["sound_file"]},
{"required": ["sound_file_url"]},
{"required": ["sound_file_path"]},
{"required": ["audio_path"]},
]
}
],
"properties": {
"image_url": {"type": "string"},
"image_path": {"type": "string"},
"audio_id": {"type": "string"},
"sound_file": {
"type": "string",
"description": "Official Kling sound_file value or raw base64 audio.",
},
"sound_file_url": {"type": "string"},
"sound_file_path": {"type": "string"},
"audio_path": {
"type": "string",
"description": "Alias for sound_file_path for compatibility with local avatar tools.",
},
"prompt": {"type": "string"},
"mode": {"type": "string", "enum": AVATAR_MODES, "default": "std"},
"callback_url": {"type": "string"},
"external_task_id": {"type": "string"},
"include_account_usage": {
"type": "boolean",
"default": False,
"description": "Optional low-frequency account usage diagnostic; not used by default.",
},
"timeout_seconds": {"type": "integer", "default": 900},
"poll_interval": {"type": "number", "default": 5.0},
"output_path": {"type": "string"},
},
}
resource_profile = ResourceProfile(
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
)
retry_policy = RetryPolicy(
max_retries=2,
backoff_seconds=2.0,
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
)
idempotency_key_fields = [
"image_url",
"image_path",
"audio_id",
"sound_file",
"sound_file_url",
"sound_file_path",
"audio_path",
"prompt",
"mode",
]
side_effects = ["paid remote generation via official Kling API", "writes avatar video to output_path"]
user_visible_verification = ["Watch generated avatar video for identity preservation and mouth motion"]
quality_score = 0.82
latency_p50_seconds = 240.0
def estimate_cost(self, inputs: dict[str, Any]) -> float:
base = 0.35
if inputs.get("mode") == "pro":
base *= 1.7
if inputs.get("sound_file_path") or inputs.get("audio_path"):
base += 0.04
return round(base, 4)
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
return 240.0
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
result = super().dry_run(inputs)
result.update(
{
"paid_api": True,
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative OpenMontage avatar estimate pending official account-usage reconciliation.",
}
)
return result
def execute(self, inputs: dict[str, Any]) -> ToolResult:
try:
self.check_dependencies()
except DependencyError as exc:
return ToolResult(success=False, error=str(exc))
start = time.time()
try:
request = self._build_request(inputs)
client = KlingClient()
task_id = client.create_classic_task(request["path"], request["payload"])
outputs = client.poll_classic(
request["path"],
task_id,
"videos",
timeout_seconds=int(inputs.get("timeout_seconds", 900)),
poll_interval=float(inputs.get("poll_interval", 5.0)),
)
paths = self._download_videos(client, outputs, inputs)
probed = probe_output(paths[0])
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
data: dict[str, Any] = {"provider": self.provider}
if isinstance(exc, KlingAPIError):
data.update(
{
"error_code": exc.code,
"request_id": exc.request_id,
"http_status": exc.http_status,
"account_usage_diagnostic": account_usage_hint_for_error(exc),
}
)
return ToolResult(success=False, data=data, error=f"Kling official avatar generation failed: {exc}")
except Exception as exc:
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official avatar generation failed: {exc}")
return ToolResult(
success=True,
data={
"provider": self.provider,
"model": "kling-official-avatar",
"task_id": task_id,
"operation": "image_to_avatar_video",
"mode": request["payload"].get("mode"),
"prompt": request["payload"].get("prompt"),
"avatar_source": request["avatar_source"],
"audio_source": request["audio_source"],
"remote_outputs": outputs,
"output": str(paths[0]),
"output_path": str(paths[0]),
"video_paths": [str(path) for path in paths],
"format": "mp4",
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
**self._account_usage_result(inputs, client),
**self._callback_result_data(inputs, task_id),
**probed,
},
artifacts=[str(path) for path in paths],
cost_usd=self.estimate_cost(inputs),
duration_seconds=round(time.time() - start, 2),
model="kling-official-avatar",
)
def _build_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
image = normalize_image_input(inputs.get("image_url"), inputs.get("image_path"))
if not image:
raise ValueError("Kling avatar requires image_url or image_path")
mode = str(inputs.get("mode") or "std")
if mode not in AVATAR_MODES:
raise ValueError(f"mode must be one of: {', '.join(AVATAR_MODES)}")
payload: dict[str, Any] = {
"image": image,
"mode": mode,
}
if inputs.get("prompt"):
payload["prompt"] = str(inputs["prompt"])
audio_source = self._copy_audio_input(inputs, payload)
self._copy_common_task_fields(inputs, payload)
return {
"protocol": "classic",
"path": "/v1/videos/avatar/image2video",
"payload": payload,
"operation": "image_to_avatar_video",
"model": "kling-official-avatar",
"avatar_source": inputs.get("image_url") or inputs.get("image_path"),
"audio_source": audio_source,
}
@staticmethod
def _copy_audio_input(inputs: dict[str, Any], payload: dict[str, Any]) -> dict[str, Any]:
audio_id = str(inputs.get("audio_id") or "").strip()
if audio_id:
payload["audio_id"] = audio_id
return {"type": "audio_id", "value": audio_id}
sound_path = inputs.get("sound_file_path") or inputs.get("audio_path")
sound_file = normalize_media_input(
url=inputs.get("sound_file_url"),
path=sound_path,
value=inputs.get("sound_file"),
label="Avatar audio file",
)
if not sound_file:
raise ValueError("Kling avatar requires audio_id, sound_file, sound_file_url, sound_file_path, or audio_path")
payload["sound_file"] = sound_file
return {
"type": "sound_file",
"source": inputs.get("sound_file_url") or sound_path or "inline",
}
def _download_videos(
self,
client: KlingClient,
outputs: list[dict[str, Any]],
inputs: dict[str, Any],
) -> list[Path]:
if not outputs:
raise ValueError("Kling avatar response contained no videos")
base_path = Path(inputs.get("output_path", "kling_avatar.mp4"))
paths: list[Path] = []
for index, item in enumerate(outputs):
url = self._output_url(item)
suffix = extension_from_url(url, ".mp4")
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
client.download(url, output_path)
paths.append(output_path)
return paths
@staticmethod
def _output_url(item: dict[str, Any]) -> str:
url = item.get("url") or item.get("video_url") or item.get("resource_url")
if url:
return str(url)
resource = item.get("resource") or {}
if isinstance(resource, dict) and resource.get("url"):
return str(resource["url"])
raise ValueError(f"Kling avatar response item contained no downloadable URL: {item}")
@staticmethod
def _copy_common_task_fields(inputs: dict[str, Any], payload: dict[str, Any]) -> None:
callback_url = validate_callback_url(inputs.get("callback_url"))
if callback_url:
payload["callback_url"] = callback_url
if inputs.get("external_task_id"):
payload["external_task_id"] = inputs["external_task_id"]
@staticmethod
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
callback_url = inputs.get("callback_url")
if not callback_url:
return {}
return {
"callback_url": str(callback_url),
"callback_requested": True,
"polling_used": True,
"task_id": task_id,
}
@staticmethod
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
if not inputs.get("include_account_usage"):
return {}
try:
usage = get_account_costs(client=client)
return {
"account_usage": usage,
"cost_source": "estimate_with_account_usage_context",
"reconciled_cost_usd": None,
}
except Exception as exc:
return {
"account_usage_error": str(exc),
"cost_source": "estimate",
"reconciled_cost_usd": None,
}

739
tools/avatar/kling_lip_sync.py Normal file
View File

@@ -0,0 +1,739 @@
"""Kling official API lip-sync provider."""
from __future__ import annotations
import json
import time
from pathlib import Path
from typing import Any
from tools._kling.account import account_usage_hint_for_error, get_account_costs
from tools._kling.callbacks import validate_callback_url
from tools._kling.client import KlingClient
from tools._kling.errors import KlingAPIError
from tools._kling.media import (
extension_from_url,
normalize_media_input,
numbered_output_path,
output_path_with_suffix,
)
from tools._kling.schemas import LIP_SYNC_OPERATIONS
from tools.base_tool import (
BaseTool,
DependencyError,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolTier,
)
from tools.video._shared import probe_output
class KlingLipSync(BaseTool):
name = "kling_lip_sync"
version = "0.1.0"
tier = ToolTier.GENERATE
capability = "avatar"
provider = "kling_official"
stability = ToolStability.EXPERIMENTAL
execution_mode = ExecutionMode.SYNC
determinism = Determinism.STOCHASTIC
runtime = ToolRuntime.API
dependencies = ["env:KLING_API_KEY"]
install_instructions = (
"Set KLING_API_KEY in .env for the official Kling API. "
"Use identify_face first for multi-person clips, then pass face_choose or face_id."
)
agent_skills = ["kling-official", "avatar-video"]
capabilities = ["lip_sync", "identify_face", "audio_video_alignment"]
supports = {
"lip_sync": True,
"face_selection": True,
"offline": False,
"cloud_render": True,
}
best_for = [
"official Kling cloud lip-sync for existing presenter video",
"dubbing workflows that can use Kling face identification",
"manual or explicit automatic face selection before paid lip-sync generation",
]
not_good_for = [
"fully offline lip-sync",
"silent first-face selection in multi-person footage",
"replacing local lip_sync behavior implicitly",
]
fallback_tools = ["lip_sync"]
input_schema = {
"type": "object",
"properties": {
"operation": {"type": "string", "enum": LIP_SYNC_OPERATIONS, "default": "advanced_lip_sync"},
"video_id": {"type": "string"},
"video_url": {"type": "string"},
"video_path": {
"type": "string",
"description": "Not silently uploaded. Provide video_url unless an official upload path is added.",
},
"session_id": {"type": "string"},
"face_id": {"type": "string"},
"face_choose": {"type": "array"},
"auto_select_face": {
"type": "boolean",
"default": False,
"description": "Explicitly allow largest-face automatic selection after identify_face.",
},
"audio_id": {"type": "string"},
"sound_file": {
"type": "string",
"description": "Official Kling sound_file value or raw base64 audio.",
},
"sound_file_url": {"type": "string"},
"sound_file_path": {"type": "string"},
"audio_path": {
"type": "string",
"description": "Alias for sound_file_path for compatibility with local lip_sync.",
},
"sound_start_time": {
"type": "integer",
"minimum": 0,
"description": "Audio crop start in milliseconds.",
},
"sound_end_time": {
"type": "integer",
"minimum": 0,
"description": "Audio crop end in milliseconds; inferred for full_lip_sync when possible.",
},
"sound_insert_time": {
"type": "integer",
"minimum": 0,
"description": "Video timeline insertion point in milliseconds.",
},
"sound_volume": {"type": "number", "minimum": 0, "maximum": 2},
"original_audio_volume": {"type": "number", "minimum": 0, "maximum": 2},
"faces_artifact_path": {"type": "string"},
"callback_url": {"type": "string"},
"external_task_id": {"type": "string"},
"include_account_usage": {
"type": "boolean",
"default": False,
"description": "Optional low-frequency account usage diagnostic; not used by default.",
},
"timeout_seconds": {"type": "integer", "default": 900},
"poll_interval": {"type": "number", "default": 5.0},
"output_path": {"type": "string"},
},
}
resource_profile = ResourceProfile(
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
)
retry_policy = RetryPolicy(
max_retries=2,
backoff_seconds=2.0,
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
)
idempotency_key_fields = [
"operation",
"video_id",
"video_url",
"session_id",
"face_id",
"face_choose",
"auto_select_face",
"audio_id",
"sound_file",
"sound_file_url",
"sound_file_path",
"audio_path",
"sound_start_time",
"sound_end_time",
"sound_insert_time",
"sound_volume",
"original_audio_volume",
]
side_effects = [
"paid remote generation via official Kling API",
"writes face selection artifact",
"writes lip-synced video to output_path",
]
user_visible_verification = ["Watch output video to verify the selected face matches the new audio"]
quality_score = 0.80
latency_p50_seconds = 240.0
def estimate_cost(self, inputs: dict[str, Any]) -> float:
operation = str(inputs.get("operation", "advanced_lip_sync"))
if operation == "identify_face":
return 0.02
if operation == "full_lip_sync":
return 0.34
return 0.32
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
if inputs.get("operation") == "identify_face":
return 15.0
return 240.0
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
result = super().dry_run(inputs)
result.update(
{
"paid_api": True,
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative OpenMontage estimate for identify-face plus advanced lip-sync.",
}
)
return result
def execute(self, inputs: dict[str, Any]) -> ToolResult:
try:
self.check_dependencies()
except DependencyError as exc:
return ToolResult(success=False, error=str(exc))
operation = str(inputs.get("operation") or "advanced_lip_sync")
start = time.time()
client = KlingClient()
try:
if operation == "identify_face":
identify = self._identify_faces(client, inputs)
return self._identify_result(inputs, identify, start)
if operation == "full_lip_sync":
identify = self._identify_faces(client, inputs)
artifact_path = self._write_faces_artifact(inputs, identify)
face_choose, selection = self._face_selection(identify["faces"], inputs)
artifact_path = self._write_faces_artifact(inputs, identify, selection=selection)
if selection["selection_method"] == "requires_user_selection":
return ToolResult(
success=False,
data={
"provider": self.provider,
"operation": operation,
"session_id": identify["session_id"],
"faces": identify["faces"],
"requires_face_selection": True,
"selection_reason": selection["selection_reason"],
"faces_artifact_path": str(artifact_path),
},
artifacts=[str(artifact_path)],
error="Multiple faces detected. Pass face_id/face_choose or set auto_select_face=True.",
cost_usd=self.estimate_cost({"operation": "identify_face"}),
duration_seconds=round(time.time() - start, 2),
model="kling-official-lip-sync",
)
merged = {**inputs, "session_id": identify["session_id"], "face_choose": face_choose}
selected_face = self._selected_face_record(identify["faces"], face_choose)
self._apply_face_timing_defaults(merged, selected_face)
request = self._build_advanced_request(merged)
result = self._run_advanced_lip_sync(client, merged, request, start)
result.data["faces_artifact_path"] = str(artifact_path)
result.data["face_selection"] = selection
result.artifacts.append(str(artifact_path))
return result
if operation == "advanced_lip_sync":
request = self._build_advanced_request(inputs)
return self._run_advanced_lip_sync(client, inputs, request, start)
raise ValueError(f"Unsupported Kling lip-sync operation: {operation}")
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
data: dict[str, Any] = {"provider": self.provider}
if isinstance(exc, KlingAPIError):
data.update(
{
"error_code": exc.code,
"request_id": exc.request_id,
"http_status": exc.http_status,
"account_usage_diagnostic": account_usage_hint_for_error(exc),
}
)
return ToolResult(success=False, data=data, error=f"Kling official lip-sync failed: {exc}")
except Exception as exc:
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official lip-sync failed: {exc}")
def _identify_faces(self, client: KlingClient, inputs: dict[str, Any]) -> dict[str, Any]:
request = self._build_identify_request(inputs)
data = client.post(request["path"], request["payload"])
payload = data.get("data") or {}
session_id = payload.get("session_id")
if not session_id:
raise ValueError(f"Kling identify-face response missing data.session_id: {data}")
faces = (
payload.get("face_data")
or payload.get("faces")
or payload.get("face_list")
or payload.get("face_infos")
or payload.get("faces_info")
or []
)
if not isinstance(faces, list):
raise ValueError("Kling identify-face response face list is not a list")
if not faces:
raise ValueError("Kling identify-face response contained no faces")
return {
"session_id": str(session_id),
"faces": faces,
"raw_response": data,
"request": request,
}
def _identify_result(self, inputs: dict[str, Any], identify: dict[str, Any], start: float) -> ToolResult:
artifact_path = self._write_faces_artifact(inputs, identify)
return ToolResult(
success=True,
data={
"provider": self.provider,
"model": "kling-official-lip-sync",
"operation": "identify_face",
"session_id": identify["session_id"],
"faces": identify["faces"],
"face_count": len(identify["faces"]),
"faces_artifact_path": str(artifact_path),
},
artifacts=[str(artifact_path)],
cost_usd=self.estimate_cost({"operation": "identify_face"}),
duration_seconds=round(time.time() - start, 2),
model="kling-official-lip-sync",
)
def _run_advanced_lip_sync(
self,
client: KlingClient,
inputs: dict[str, Any],
request: dict[str, Any],
start: float,
) -> ToolResult:
task_id = client.create_classic_task(request["path"], request["payload"])
outputs = client.poll_classic(
request["path"],
task_id,
"videos",
timeout_seconds=int(inputs.get("timeout_seconds", 900)),
poll_interval=float(inputs.get("poll_interval", 5.0)),
)
paths = self._download_videos(client, outputs, inputs)
probed = probe_output(paths[0])
return ToolResult(
success=True,
data={
"provider": self.provider,
"model": "kling-official-lip-sync",
"task_id": task_id,
"operation": request["operation"],
"session_id": request["payload"]["session_id"],
"face_choose": self._face_choose_result_metadata(
request["payload"]["face_choose"]
),
"audio_source": request["audio_source"],
"remote_outputs": outputs,
"output": str(paths[0]),
"output_path": str(paths[0]),
"video_paths": [str(path) for path in paths],
"format": "mp4",
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
**self._account_usage_result(inputs, client),
**self._callback_result_data(inputs, task_id),
**probed,
},
artifacts=[str(path) for path in paths],
cost_usd=self.estimate_cost(inputs),
duration_seconds=round(time.time() - start, 2),
model="kling-official-lip-sync",
)
def _build_identify_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
if inputs.get("video_path") and not (inputs.get("video_url") or inputs.get("video_id")):
raise ValueError("Kling identify_face requires video_url or video_id; local video paths cannot be silently uploaded.")
payload: dict[str, Any] = {}
if inputs.get("video_id"):
payload["video_id"] = str(inputs["video_id"])
if inputs.get("video_url"):
payload["video_url"] = str(inputs["video_url"])
if not payload:
raise ValueError("Kling identify_face requires video_id or video_url")
return {
"path": "/v1/videos/identify-face",
"payload": payload,
"operation": "identify_face",
}
def _build_advanced_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
session_id = str(inputs.get("session_id") or "").strip()
if not session_id:
raise ValueError("advanced_lip_sync requires session_id")
face_choose = self._normalize_face_choose(inputs)
if not face_choose:
raise ValueError("advanced_lip_sync requires face_choose or face_id")
if len(face_choose) != 1:
raise ValueError("advanced_lip_sync currently supports exactly one face_choose item")
face_item = face_choose[0]
audio_source = self._copy_audio_input(inputs, face_item)
self._copy_timing_fields(inputs, face_item)
payload: dict[str, Any] = {
"session_id": session_id,
"face_choose": face_choose,
}
self._copy_common_task_fields(inputs, payload)
return {
"protocol": "classic",
"path": "/v1/videos/advanced-lip-sync",
"payload": payload,
"operation": "advanced_lip_sync",
"model": "kling-official-lip-sync",
"audio_source": audio_source,
}
@staticmethod
def _normalize_face_choose(inputs: dict[str, Any]) -> list[dict[str, Any]]:
if inputs.get("face_choose"):
raw = inputs["face_choose"]
if isinstance(raw, dict):
raw = [raw]
if not isinstance(raw, list):
raise ValueError("face_choose must be a list of face choice objects")
normalized: list[dict[str, Any]] = []
for item in raw:
if isinstance(item, str):
normalized.append({"face_id": item})
elif isinstance(item, dict):
if not (item.get("face_id") or item.get("id")):
raise ValueError("face_choose items must include face_id")
record = dict(item)
if "face_id" not in record and record.get("id"):
record["face_id"] = record.pop("id")
normalized.append(record)
else:
raise ValueError("face_choose items must be strings or objects")
return normalized
if inputs.get("face_id"):
return [{"face_id": str(inputs["face_id"])}]
return []
def _face_selection(
self,
faces: list[dict[str, Any]],
inputs: dict[str, Any],
) -> tuple[list[dict[str, Any]], dict[str, Any]]:
explicit = self._normalize_face_choose(inputs)
if explicit:
return explicit, {
"selection_method": "user_selected",
"selection_reason": "face_choose or face_id was provided",
"selected_face": explicit,
}
if len(faces) == 1:
choice = [self._face_to_choice(faces[0])]
return choice, {
"selection_method": "single_face",
"selection_reason": "Only one face was returned by identify_face",
"selected_face": choice,
}
if not inputs.get("auto_select_face"):
return [], {
"selection_method": "requires_user_selection",
"selection_reason": "Multiple faces detected and auto_select_face was not enabled",
"face_count": len(faces),
}
selected = max(faces, key=self._face_area)
choice = [self._face_to_choice(selected)]
return choice, {
"selection_method": "auto_selected",
"selection_reason": "auto_select_face=True selected the largest detected face area",
"selected_face": choice,
}
@staticmethod
def _face_to_choice(face: dict[str, Any]) -> dict[str, Any]:
face_id = face.get("face_id") or face.get("id")
if not face_id:
raise ValueError(f"Cannot select face without face_id/id: {face}")
return {"face_id": str(face_id)}
@staticmethod
def _face_area(face: dict[str, Any]) -> float:
for key in ("bbox", "box"):
value = face.get(key)
if isinstance(value, list) and len(value) >= 4:
third = float(value[2])
fourth = float(value[3])
width_height_area = max(third, 0.0) * max(fourth, 0.0)
corner_width = third - float(value[0])
corner_height = fourth - float(value[1])
corner_area = (
corner_width * corner_height
if corner_width > 0 and corner_height > 0
else 0.0
)
if corner_area and width_height_area:
return min(corner_area, width_height_area)
return corner_area or width_height_area
if isinstance(value, dict):
width = value.get("width") or value.get("w")
height = value.get("height") or value.get("h")
if width is not None and height is not None:
return max(float(width), 0.0) * max(float(height), 0.0)
width = face.get("width") or face.get("w")
height = face.get("height") or face.get("h")
if width is not None and height is not None:
return max(float(width), 0.0) * max(float(height), 0.0)
return 0.0
@staticmethod
def _selected_face_record(
faces: list[dict[str, Any]], face_choose: list[dict[str, Any]]
) -> dict[str, Any]:
selected_id = str(face_choose[0].get("face_id") or "")
for face in faces:
if str(face.get("face_id") or face.get("id") or "") == selected_id:
return face
raise ValueError(f"Selected face_id {selected_id!r} was not returned by identify_face")
def _apply_face_timing_defaults(
self, inputs: dict[str, Any], face: dict[str, Any]
) -> None:
face_start = int(face.get("start_time") or 0)
face_end = int(face.get("end_time") or 0)
face_choose = self._normalize_face_choose(inputs)
face_item = face_choose[0] if face_choose else {}
if inputs.get("sound_start_time") is None and face_item.get("sound_start_time") is None:
inputs["sound_start_time"] = 0
if inputs.get("sound_insert_time") is None and face_item.get("sound_insert_time") is None:
inputs["sound_insert_time"] = face_start
if inputs.get("sound_end_time") is not None or face_item.get("sound_end_time") is not None:
return
candidates: list[int] = []
audio_duration = self._local_audio_duration_ms(inputs)
if audio_duration:
candidates.append(audio_duration)
if face_end > face_start:
candidates.append(face_end - face_start)
if not candidates:
raise ValueError(
"full_lip_sync could not infer sound_end_time; provide it explicitly"
)
inputs["sound_end_time"] = min(candidates)
@staticmethod
def _local_audio_duration_ms(inputs: dict[str, Any]) -> int | None:
sound_path = inputs.get("sound_file_path") or inputs.get("audio_path")
if not sound_path:
return None
path = Path(sound_path)
if not path.is_file():
return None
seconds = probe_output(path).get("duration_seconds")
if not seconds:
return None
return int(round(float(seconds) * 1000))
@staticmethod
def _copy_timing_fields(inputs: dict[str, Any], face_item: dict[str, Any]) -> None:
for key, default in (("sound_start_time", 0), ("sound_insert_time", 0)):
nested = face_item.get(key)
top_level = inputs.get(key)
if nested is not None and top_level is not None and int(nested) != int(top_level):
raise ValueError(
f"Conflicting {key} values between top-level input and face_choose[0]"
)
value = nested if nested is not None else top_level
if value is None:
value = default
face_item[key] = int(value)
nested_end = face_item.get("sound_end_time")
top_level_end = inputs.get("sound_end_time")
if (
nested_end is not None
and top_level_end is not None
and int(nested_end) != int(top_level_end)
):
raise ValueError(
"Conflicting sound_end_time values between top-level input and face_choose[0]"
)
sound_end = nested_end if nested_end is not None else top_level_end
if sound_end is None:
raise ValueError("advanced_lip_sync requires sound_end_time")
face_item["sound_end_time"] = int(sound_end)
if face_item["sound_end_time"] - face_item["sound_start_time"] < 2000:
raise ValueError("advanced_lip_sync requires at least 2000ms of cropped audio")
for key in ("sound_volume", "original_audio_volume"):
nested = face_item.get(key)
top_level = inputs.get(key)
if nested is not None and top_level is not None and float(nested) != float(top_level):
raise ValueError(
f"Conflicting {key} values between top-level input and face_choose[0]"
)
value = nested if nested is not None else top_level
if value is not None:
face_item[key] = float(value)
@staticmethod
def _face_choose_result_metadata(
face_choose: list[dict[str, Any]],
) -> list[dict[str, Any]]:
metadata: list[dict[str, Any]] = []
for item in face_choose:
record = {key: value for key, value in item.items() if key != "sound_file"}
if item.get("sound_file"):
record["sound_file_provided"] = True
metadata.append(record)
return metadata
@staticmethod
def _copy_audio_input(inputs: dict[str, Any], payload: dict[str, Any]) -> dict[str, Any]:
nested_audio_id = str(payload.get("audio_id") or "").strip()
nested_sound_file = payload.get("sound_file")
if nested_audio_id and nested_sound_file:
raise ValueError(
"Conflicting audio input in face_choose[0]; provide audio_id or sound_file, not both"
)
top_level_audio_id = str(inputs.get("audio_id") or "").strip()
top_level_sound_requested = any(
inputs.get(key)
for key in ("sound_file", "sound_file_url", "sound_file_path", "audio_path")
)
if nested_audio_id:
if (
(top_level_audio_id and top_level_audio_id != nested_audio_id)
or top_level_sound_requested
):
raise ValueError(
"Conflicting audio input between top-level fields and face_choose[0]"
)
payload["audio_id"] = nested_audio_id
return {"type": "audio_id", "value": nested_audio_id}
if nested_sound_file:
if top_level_audio_id:
raise ValueError(
"Conflicting audio input between top-level fields and face_choose[0]"
)
if top_level_sound_requested:
top_level_sound_file = normalize_media_input(
url=inputs.get("sound_file_url"),
path=inputs.get("sound_file_path") or inputs.get("audio_path"),
value=inputs.get("sound_file"),
label="Lip-sync audio file",
)
if top_level_sound_file != nested_sound_file:
raise ValueError(
"Conflicting audio input between top-level fields and face_choose[0]"
)
return {"type": "sound_file", "source": "face_choose[0]"}
if top_level_audio_id:
payload["audio_id"] = top_level_audio_id
return {"type": "audio_id", "value": top_level_audio_id}
sound_path = inputs.get("sound_file_path") or inputs.get("audio_path")
sound_file = normalize_media_input(
url=inputs.get("sound_file_url"),
path=sound_path,
value=inputs.get("sound_file"),
label="Lip-sync audio file",
)
if not sound_file:
raise ValueError("advanced_lip_sync requires audio_id, sound_file, sound_file_url, sound_file_path, or audio_path")
payload["sound_file"] = sound_file
return {
"type": "sound_file",
"source": inputs.get("sound_file_url") or sound_path or "inline",
}
def _download_videos(
self,
client: KlingClient,
outputs: list[dict[str, Any]],
inputs: dict[str, Any],
) -> list[Path]:
if not outputs:
raise ValueError("Kling lip-sync response contained no videos")
base_path = Path(inputs.get("output_path", "kling_lip_sync.mp4"))
paths: list[Path] = []
for index, item in enumerate(outputs):
url = self._output_url(item)
suffix = extension_from_url(url, ".mp4")
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
client.download(url, output_path)
paths.append(output_path)
return paths
@staticmethod
def _output_url(item: dict[str, Any]) -> str:
url = item.get("url") or item.get("video_url") or item.get("resource_url")
if url:
return str(url)
resource = item.get("resource") or {}
if isinstance(resource, dict) and resource.get("url"):
return str(resource["url"])
raise ValueError(f"Kling lip-sync response item contained no downloadable URL: {item}")
@staticmethod
def _copy_common_task_fields(inputs: dict[str, Any], payload: dict[str, Any]) -> None:
callback_url = validate_callback_url(inputs.get("callback_url"))
if callback_url:
payload["callback_url"] = callback_url
if inputs.get("external_task_id"):
payload["external_task_id"] = inputs["external_task_id"]
@staticmethod
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
callback_url = inputs.get("callback_url")
if not callback_url:
return {}
return {
"callback_url": str(callback_url),
"callback_requested": True,
"polling_used": True,
"task_id": task_id,
}
@staticmethod
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
if not inputs.get("include_account_usage"):
return {}
try:
usage = get_account_costs(client=client)
return {
"account_usage": usage,
"cost_source": "estimate_with_account_usage_context",
"reconciled_cost_usd": None,
}
except Exception as exc:
return {
"account_usage_error": str(exc),
"cost_source": "estimate",
"reconciled_cost_usd": None,
}
def _write_faces_artifact(
self,
inputs: dict[str, Any],
identify: dict[str, Any],
selection: dict[str, Any] | None = None,
) -> Path:
if inputs.get("faces_artifact_path"):
artifact_path = Path(inputs["faces_artifact_path"])
elif inputs.get("output_path"):
artifact_path = Path(inputs["output_path"]).with_name("kling_lip_sync_faces.json")
else:
artifact_path = Path("kling_lip_sync_faces.json")
artifact_path.parent.mkdir(parents=True, exist_ok=True)
artifact = {
"provider": self.provider,
"operation": "identify_face",
"session_id": identify["session_id"],
"faces": identify["faces"],
"face_count": len(identify["faces"]),
"selection": selection,
}
artifact_path.write_text(json.dumps(artifact, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
return artifact_path

60
tools/graphics/image_selector.py
View File

@@ -61,6 +61,14 @@ class ImageSelector(BaseTool):
"type": "string",
"description": "Resolution tier for providers that support named resolutions.",
},
"api_family": {
"type": "string",
"description": "Provider-specific API family hint passed through when supported.",
},
"model_name": {
"type": "string",
"description": "Provider-specific model name passed through when supported.",
},
"generation_mode": {
"type": "string",
"enum": ["generate", "edit"],
@@ -79,6 +87,46 @@ class ImageSelector(BaseTool):
"items": {"type": "string"},
"description": "Multiple local source image paths for compositing edits.",
},
"image_list": {
"type": "array",
"description": "Provider-specific image reference list, e.g. Kling Official Image Omni.",
},
"element_list": {
"type": "array",
"description": "Provider-specific element references, e.g. Kling Official element_id objects.",
},
"image_reference": {
"type": "string",
"description": "Provider-specific reference type, e.g. subject or face.",
},
"image_fidelity": {
"type": "number",
"description": "Provider-specific reference image fidelity hint.",
},
"human_fidelity": {
"type": "number",
"description": "Provider-specific human or face fidelity hint.",
},
"result_type": {
"type": "string",
"description": "Provider-specific result type, e.g. single or series.",
},
"series_amount": {
"type": "string",
"description": "Provider-specific series amount for image series generation.",
},
"watermark": {
"type": "boolean",
"description": "Provider-specific watermark toggle passed through when supported.",
},
"callback_url": {
"type": "string",
"description": "Provider-specific callback URL. Current OpenMontage providers still poll by default.",
},
"external_task_id": {
"type": "string",
"description": "Provider-specific idempotency/provenance task id.",
},
"preferred_provider": {
"type": "string",
"description": "Provider name or 'auto'. Valid values are discovered at runtime from the registry.",
@@ -216,6 +264,18 @@ class ImageSelector(BaseTool):
"image_path",
"image_urls",
"image_paths",
"image_list",
"element_list",
"api_family",
"model_name",
"image_reference",
"image_fidelity",
"human_fidelity",
"result_type",
"series_amount",
"watermark",
"callback_url",
"external_task_id",
"workflow_json",
"workflow_path",
"output_node",

449
tools/graphics/kling_official_image.py Normal file
View File

@@ -0,0 +1,449 @@
"""Kling official API image generation provider."""
from __future__ import annotations
import time
from pathlib import Path
from typing import Any
from tools._kling.account import account_usage_hint_for_error, get_account_costs
from tools._kling.callbacks import validate_callback_url
from tools._kling.client import KlingClient
from tools._kling.elements import element_ids, normalize_element_list
from tools._kling.errors import KlingAPIError
from tools._kling.media import (
extension_from_url,
normalize_image_input,
numbered_output_path,
output_path_with_suffix,
)
from tools._kling.omni import build_image_prompt_references
from tools._kling.schemas import (
IMAGE_ASPECT_RATIOS,
IMAGE_GENERATION_MODELS,
IMAGE_MODELS,
IMAGE_REFERENCE_TYPES,
IMAGE_RESOLUTIONS,
IMAGE_RESULT_TYPES,
OMNI_IMAGE_MODELS,
)
from tools.base_tool import (
BaseTool,
DependencyError,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolTier,
)
class KlingOfficialImage(BaseTool):
name = "kling_official_image"
version = "0.1.0"
tier = ToolTier.GENERATE
capability = "image_generation"
provider = "kling_official"
stability = ToolStability.EXPERIMENTAL
execution_mode = ExecutionMode.SYNC
determinism = Determinism.STOCHASTIC
runtime = ToolRuntime.API
dependencies = ["env:KLING_API_KEY"]
install_instructions = (
"Set KLING_API_KEY in .env for the official Kling API. "
"Optionally set KLING_API_BASE_URL to override the default Singapore endpoint."
)
agent_skills = ["kling-official"]
capabilities = ["generate_image", "text_to_image", "image_edit"]
supports = {
"text_to_image": True,
"image_edit": True,
"negative_prompt": True,
"aspect_ratio": True,
}
best_for = [
"official Kling image generation",
"subject or face reference generation",
"Omni multi-reference image workflows",
]
not_good_for = ["offline generation", "free generation", "non-Kling model families"]
fallback_tools = ["flux_image", "google_imagen", "openai_image", "recraft_image"]
input_schema = {
"type": "object",
"required": ["prompt"],
"properties": {
"prompt": {"type": "string"},
"negative_prompt": {"type": "string"},
"operation": {"type": "string", "enum": ["generate", "omni"], "default": "generate"},
"generation_mode": {"type": "string", "enum": ["generate", "edit"], "default": "generate"},
"api_family": {"type": "string", "enum": ["generation", "omni"], "default": "generation"},
"model_name": {"type": "string", "enum": IMAGE_MODELS, "default": "kling-v3"},
"image_url": {"type": "string"},
"image_path": {"type": "string"},
"image_urls": {"type": "array", "items": {"type": "string"}},
"image_paths": {"type": "array", "items": {"type": "string"}},
"image_list": {"type": "array"},
"image_reference": {"type": "string", "enum": IMAGE_REFERENCE_TYPES},
"image_fidelity": {"type": "number", "default": 0.5},
"human_fidelity": {"type": "number", "default": 0.45},
"resolution": {"type": "string", "enum": IMAGE_RESOLUTIONS, "default": "1k"},
"aspect_ratio": {"type": "string", "enum": IMAGE_ASPECT_RATIOS, "default": "16:9"},
"n": {"type": "integer", "default": 1},
"result_type": {"type": "string", "enum": IMAGE_RESULT_TYPES, "default": "single"},
"series_amount": {"type": "string"},
"element_list": {"type": "array"},
"watermark": {"type": "boolean", "default": False},
"callback_url": {"type": "string"},
"external_task_id": {"type": "string"},
"include_account_usage": {
"type": "boolean",
"default": False,
"description": "Optional low-frequency account usage diagnostic; not used by default.",
},
"timeout_seconds": {"type": "integer", "default": 600},
"poll_interval": {"type": "number", "default": 3.0},
"output_path": {"type": "string"},
},
}
resource_profile = ResourceProfile(
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=200, network_required=True
)
retry_policy = RetryPolicy(
max_retries=2,
backoff_seconds=2.0,
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
)
idempotency_key_fields = [
"prompt",
"negative_prompt",
"operation",
"api_family",
"model_name",
"image_url",
"image_path",
"image_urls",
"image_paths",
"image_list",
"image_reference",
"image_fidelity",
"human_fidelity",
"aspect_ratio",
"resolution",
"n",
"result_type",
"series_amount",
"element_list",
"watermark",
]
side_effects = [
"paid remote generation via official Kling API",
"writes image file(s) to output_path",
]
user_visible_verification = ["Inspect generated image for quality, prompt adherence, and reference fidelity"]
def estimate_cost(self, inputs: dict[str, Any]) -> float:
n = int(inputs.get("n", 1) or 1)
resolution = str(inputs.get("resolution", "1k"))
api_family = str(inputs.get("api_family", "generation"))
base = 0.04 if api_family == "generation" else 0.08
if resolution == "2k":
base *= 1.8
if resolution == "4k":
base *= 3.5
if inputs.get("result_type") == "series":
base *= 1.5
amount = inputs.get("series_amount")
if amount and str(amount).isdigit():
base *= max(int(str(amount)), 1)
if api_family == "omni":
reference_count = sum(len(inputs.get(key) or []) for key in ("image_list", "image_urls", "image_paths", "element_list"))
if inputs.get("image_url") or inputs.get("image_path"):
reference_count += 1
base *= 1 + (0.08 * reference_count)
return round(base * max(n, 1), 4)
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
return 90.0
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
result = super().dry_run(inputs)
result.update(
{
"paid_api": True,
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative OpenMontage estimate; official account usage reconciliation is planned for Phase 2.",
}
)
return result
def execute(self, inputs: dict[str, Any]) -> ToolResult:
try:
self.check_dependencies()
except DependencyError as exc:
return ToolResult(success=False, error=str(exc))
start = time.time()
try:
request = self._build_request(inputs)
client = KlingClient()
task_id = client.create_classic_task(request["path"], request["payload"])
outputs = client.poll_classic(
request["path"],
task_id,
"images",
timeout_seconds=int(inputs.get("timeout_seconds", 600)),
poll_interval=float(inputs.get("poll_interval", 3.0)),
)
paths = self._download_images(client, outputs, inputs)
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
data: dict[str, Any] = {"provider": self.provider}
if isinstance(exc, KlingAPIError):
data.update(
{
"error_code": exc.code,
"request_id": exc.request_id,
"http_status": exc.http_status,
}
)
data["account_usage_diagnostic"] = account_usage_hint_for_error(exc)
return ToolResult(success=False, data=data, error=f"Kling official image generation failed: {exc}")
except Exception as exc:
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official image generation failed: {exc}")
return ToolResult(
success=True,
data={
"provider": self.provider,
"model": request["model"],
"task_id": task_id,
"api_family": request["api_family"],
"operation": request["operation"],
"prompt": request["payload"]["prompt"],
"remote_outputs": outputs,
"output": str(paths[0]),
"output_path": str(paths[0]),
"image_paths": [str(path) for path in paths],
"format": paths[0].suffix.lstrip(".") or "png",
"references_used": request.get("references_used", []),
"element_ids": request.get("element_ids", []),
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
**self._account_usage_result(inputs, client),
**self._callback_result_data(inputs, task_id),
},
artifacts=[str(path) for path in paths],
cost_usd=self.estimate_cost(inputs),
duration_seconds=round(time.time() - start, 2),
model=request["model"],
)
def _build_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
api_family = str(inputs.get("api_family", "generation"))
if inputs.get("operation") == "omni":
api_family = "omni"
if api_family == "omni":
return self._build_omni_request(inputs)
return self._build_generation_request(inputs)
def _build_generation_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
prompt = self._prompt(inputs)
model_name = str(inputs.get("model_name") or "kling-v3")
if model_name not in IMAGE_GENERATION_MODELS:
raise ValueError(f"model_name {model_name!r} is not supported for api_family=generation")
payload: dict[str, Any] = {
"model_name": model_name,
"prompt": prompt,
"resolution": inputs.get("resolution", "1k"),
"n": int(inputs.get("n", 1) or 1),
"aspect_ratio": inputs.get("aspect_ratio", "16:9"),
}
if len(prompt) > 2500:
raise ValueError("prompt exceeds Kling image generation limit of 2500 characters")
if inputs.get("negative_prompt"):
payload["negative_prompt"] = inputs["negative_prompt"]
image = normalize_image_input(inputs.get("image_url"), inputs.get("image_path"))
if image:
payload["image"] = image
if inputs.get("image_reference"):
payload["image_reference"] = inputs["image_reference"]
for key in ("image_fidelity", "human_fidelity"):
if inputs.get(key) is not None:
payload[key] = inputs[key]
elements = normalize_element_list(inputs.get("element_list"))
if elements:
payload["element_list"] = elements
self._copy_common_task_fields(inputs, payload)
return {
"protocol": "classic",
"path": "/v1/images/generations",
"payload": payload,
"api_family": "generation",
"operation": "generate",
"model": payload["model_name"],
"references_used": self._reference_metadata_from_generation_payload(payload),
"element_ids": element_ids(payload.get("element_list")),
}
def _build_omni_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
model_name = str(inputs.get("model_name") or "kling-image-o1")
if model_name not in OMNI_IMAGE_MODELS:
raise ValueError(f"model_name {model_name!r} is not supported for api_family=omni")
image_items, references_used = self._normalize_omni_image_list(inputs)
prompt, prompt_references = build_image_prompt_references(self._prompt(inputs), image_items)
references_used = prompt_references or references_used
elements = normalize_element_list(inputs.get("element_list"))
payload: dict[str, Any] = {
"model_name": model_name,
"prompt": prompt,
"resolution": inputs.get("resolution", "1k"),
"n": int(inputs.get("n", 1) or 1),
"result_type": inputs.get("result_type", "single"),
"aspect_ratio": inputs.get("aspect_ratio", "16:9"),
}
if image_items:
payload["image_list"] = [{"image": item["image"]} for item in image_items]
if elements:
payload["element_list"] = elements
if inputs.get("series_amount"):
payload["series_amount"] = inputs["series_amount"]
self._copy_common_task_fields(inputs, payload)
return {
"protocol": "classic",
"path": "/v1/images/omni-image",
"payload": payload,
"api_family": "omni",
"operation": "generate",
"model": model_name,
"references_used": references_used,
"element_ids": [item["element_id"] for item in elements],
}
def _normalize_omni_image_list(
self,
inputs: dict[str, Any],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
image_list: list[dict[str, Any]] = []
references_used: list[dict[str, Any]] = []
def add_image(value: str | None, *, source: str | None, source_type: str) -> None:
if not value:
return
image_list.append({"image": value, "source": source or value, "source_type": source_type})
references_used.append(
{
"kind": "image",
"source": source or value,
"source_type": source_type,
"placeholder": f"<<<image_{len(image_list)}>>>",
}
)
for item in inputs.get("image_list") or []:
if not isinstance(item, dict):
raise ValueError("image_list items must be objects")
source = item.get("image") or item.get("image_url") or item.get("image_path")
value = normalize_image_input(item.get("image") or item.get("image_url"), item.get("image_path"))
if not value:
raise ValueError("image_list items must include image, image_url, or image_path")
add_image(value, source=source, source_type="image_list")
for url in inputs.get("image_urls") or []:
add_image(normalize_image_input(url=url), source=url, source_type="image_urls")
for path in inputs.get("image_paths") or []:
add_image(normalize_image_input(path=path), source=str(path), source_type="image_paths")
if inputs.get("image_url") or inputs.get("image_path"):
add_image(
normalize_image_input(inputs.get("image_url"), inputs.get("image_path")),
source=inputs.get("image_url") or inputs.get("image_path"),
source_type="image",
)
return image_list, references_used
def _download_images(self, client: KlingClient, outputs: list[dict[str, Any]], inputs: dict[str, Any]) -> list[Path]:
if not outputs:
raise ValueError("Kling image response contained no images")
base_path = Path(inputs.get("output_path", "kling_official_image.png"))
paths: list[Path] = []
for index, item in enumerate(outputs):
url = self._output_url(item)
suffix = extension_from_url(url, ".png")
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
client.download(url, output_path)
paths.append(output_path)
return paths
@staticmethod
def _output_url(item: dict[str, Any]) -> str:
url = item.get("url") or item.get("image_url") or item.get("resource_url")
if url:
return str(url)
resource = item.get("resource") or {}
if isinstance(resource, dict) and resource.get("url"):
return str(resource["url"])
raise ValueError(f"Kling image response item contained no downloadable URL: {item}")
@staticmethod
def _prompt(inputs: dict[str, Any]) -> str:
prompt = str(inputs.get("prompt") or "").strip()
if not prompt:
raise ValueError("prompt is required")
return prompt
@staticmethod
def _copy_common_task_fields(inputs: dict[str, Any], payload: dict[str, Any]) -> None:
if "watermark" in inputs:
payload["watermark_info"] = {"enabled": bool(inputs.get("watermark"))}
callback_url = validate_callback_url(inputs.get("callback_url"))
if callback_url:
payload["callback_url"] = callback_url
if inputs.get("external_task_id"):
payload["external_task_id"] = inputs["external_task_id"]
@staticmethod
def _reference_metadata_from_generation_payload(payload: dict[str, Any]) -> list[dict[str, Any]]:
references: list[dict[str, Any]] = []
if payload.get("image"):
references.append({"kind": "image", "source_type": "image"})
if payload.get("element_list"):
references.extend(
{"kind": "element", "element_id": item["element_id"]}
for item in normalize_element_list(payload.get("element_list"))
)
return references
@staticmethod
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
callback_url = inputs.get("callback_url")
if not callback_url:
return {}
return {
"callback_url": str(callback_url),
"callback_requested": True,
"polling_used": True,
"task_id": task_id,
}
@staticmethod
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
if not inputs.get("include_account_usage"):
return {}
try:
usage = get_account_costs(client=client)
return {
"account_usage": usage,
"cost_source": "estimate_with_account_usage_context",
"reconciled_cost_usd": None,
}
except Exception as exc:
return {
"account_usage_error": str(exc),
"cost_source": "estimate",
"reconciled_cost_usd": None,
}

704
tools/video/kling_official_video.py Normal file
View File

@@ -0,0 +1,704 @@
"""Kling official API video generation provider."""
from __future__ import annotations
import time
from pathlib import Path
from typing import Any
from tools._kling.account import account_usage_hint_for_error, get_account_costs
from tools._kling.callbacks import validate_callback_url
from tools._kling.client import KlingClient
from tools._kling.elements import element_ids, normalize_element_list
from tools._kling.errors import KlingAPIError
from tools._kling.media import (
extension_from_url,
normalize_image_input,
numbered_output_path,
output_path_with_suffix,
)
from tools._kling.schemas import (
CLASSIC_VIDEO_MODELS,
OMNI_VIDEO_MODELS,
SOUND_VALUES,
VIDEO_ASPECT_RATIOS,
VIDEO_DURATIONS,
VIDEO_MODES,
VIDEO_MODELS,
VIDEO_RESOLUTIONS,
)
from tools.base_tool import (
BaseTool,
DependencyError,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolTier,
)
from tools.video._shared import probe_output
class KlingOfficialVideo(BaseTool):
name = "kling_official_video"
version = "0.1.0"
tier = ToolTier.GENERATE
capability = "video_generation"
provider = "kling_official"
stability = ToolStability.EXPERIMENTAL
execution_mode = ExecutionMode.SYNC
determinism = Determinism.STOCHASTIC
runtime = ToolRuntime.API
dependencies = ["env:KLING_API_KEY"]
install_instructions = (
"Set KLING_API_KEY in .env for the official Kling API. "
"Optionally set KLING_API_BASE_URL to override the default Singapore endpoint."
)
agent_skills = ["ai-video-gen", "kling-official"]
capabilities = ["text_to_video", "image_to_video", "reference_to_video"]
supports = {
"text_to_video": True,
"image_to_video": True,
"reference_to_video": True,
"reference_image": True,
"negative_prompt": True,
"aspect_ratio": True,
}
best_for = [
"official Kling direct API access",
"text-to-video and image-to-video with Kling model controls",
"projects that need provider provenance separate from fal.ai Kling",
]
not_good_for = ["offline generation", "free generation", "non-Kling model families"]
fallback_tools = ["kling_video", "seedance_video", "veo_video", "minimax_video"]
input_schema = {
"type": "object",
"required": ["prompt"],
"properties": {
"prompt": {"type": "string"},
"operation": {
"type": "string",
"enum": ["text_to_video", "image_to_video", "reference_to_video", "omni_video"],
"default": "text_to_video",
},
"api_family": {
"type": "string",
"enum": ["classic", "turbo", "omni"],
"default": "classic",
},
"model_name": {"type": "string", "enum": VIDEO_MODELS, "default": "kling-v3"},
"model_variant": {"type": "string", "description": "Compatibility alias for model_name."},
"duration": {"type": "string", "enum": VIDEO_DURATIONS, "default": "5"},
"aspect_ratio": {"type": "string", "enum": VIDEO_ASPECT_RATIOS, "default": "16:9"},
"resolution": {"type": "string", "enum": VIDEO_RESOLUTIONS, "default": "720p"},
"mode": {"type": "string", "enum": VIDEO_MODES, "default": "std"},
"sound": {"type": "string", "enum": SOUND_VALUES, "default": "off"},
"negative_prompt": {"type": "string"},
"cfg_scale": {"type": "number", "default": 0.5},
"reference_image_url": {"type": "string"},
"reference_image_path": {"type": "string"},
"reference_tail_image_url": {"type": "string"},
"reference_tail_image_path": {"type": "string"},
"reference_image_urls": {"type": "array", "items": {"type": "string"}},
"reference_image_paths": {"type": "array", "items": {"type": "string"}},
"reference_video_url": {"type": "string"},
"reference_video_path": {"type": "string"},
"video_urls": {"type": "array", "items": {"type": "string"}},
"video_paths": {"type": "array", "items": {"type": "string"}},
"image_list": {"type": "array"},
"video_list": {"type": "array"},
"element_list": {"type": "array"},
"multi_shot": {"type": "boolean"},
"shot_type": {"type": "string", "enum": ["customize", "intelligence"]},
"multi_prompt": {"type": "array"},
"camera_control": {"type": "object"},
"watermark": {"type": "boolean", "default": False},
"callback_url": {"type": "string"},
"external_task_id": {"type": "string"},
"include_account_usage": {
"type": "boolean",
"default": False,
"description": "Optional low-frequency account usage diagnostic; not used by default.",
},
"timeout_seconds": {"type": "integer", "default": 900},
"poll_interval": {"type": "number", "default": 5.0},
"output_path": {"type": "string"},
},
}
resource_profile = ResourceProfile(
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
)
retry_policy = RetryPolicy(
max_retries=2,
backoff_seconds=2.0,
retryable_errors=["1302", "1303", "5000", "5001", "5002"],
)
idempotency_key_fields = [
"prompt",
"negative_prompt",
"operation",
"api_family",
"model_name",
"model_variant",
"duration",
"aspect_ratio",
"resolution",
"mode",
"sound",
"cfg_scale",
"camera_control",
"reference_image_url",
"reference_image_path",
"reference_tail_image_url",
"reference_tail_image_path",
"reference_image_urls",
"reference_image_paths",
"reference_video_url",
"video_urls",
"image_list",
"video_list",
"element_list",
"multi_shot",
"shot_type",
"multi_prompt",
"watermark",
]
side_effects = [
"paid remote generation via official Kling API",
"writes video file to output_path",
]
user_visible_verification = ["Watch generated clip for motion coherence and prompt adherence"]
def estimate_cost(self, inputs: dict[str, Any]) -> float:
duration = int(str(inputs.get("duration", "5")))
mode = str(inputs.get("mode", "std"))
api_family = str(inputs.get("api_family", "classic"))
base = 0.18
if api_family == "turbo":
base = 0.22
if api_family == "omni":
base = 0.30
if mode == "pro":
base *= 1.6
if mode == "4k":
base *= 3.0
if inputs.get("sound") == "on":
base += 0.05
if api_family == "omni":
reference_count = self._estimate_reference_count(inputs)
base *= 1 + (0.12 * reference_count)
multi_prompt = inputs.get("multi_prompt") or []
if multi_prompt:
base *= 1 + (0.10 * len(multi_prompt))
return round(base * max(duration, 3) / 5, 4)
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
return 180.0
def dry_run(self, inputs: dict[str, Any]) -> dict[str, Any]:
result = super().dry_run(inputs)
result.update(
{
"paid_api": True,
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative OpenMontage estimate; official account usage reconciliation is planned for Phase 2.",
}
)
return result
def execute(self, inputs: dict[str, Any]) -> ToolResult:
try:
self.check_dependencies()
except DependencyError as exc:
return ToolResult(success=False, error=str(exc))
start = time.time()
try:
request = self._build_request(inputs)
client = KlingClient()
if request["protocol"] == "turbo":
task_id = client.create_turbo(request["path"], request["payload"])
outputs = client.poll_turbo(
task_id,
timeout_seconds=int(inputs.get("timeout_seconds", 900)),
poll_interval=float(inputs.get("poll_interval", 5.0)),
)
else:
task_id = client.create_classic_task(request["path"], request["payload"])
outputs = client.poll_classic(
request["path"],
task_id,
"videos",
timeout_seconds=int(inputs.get("timeout_seconds", 900)),
poll_interval=float(inputs.get("poll_interval", 5.0)),
)
paths = self._download_videos(client, outputs, inputs)
video_url = self._first_output_url(outputs)
probed = probe_output(paths[0])
except (KlingAPIError, TimeoutError, ValueError, KeyError, FileNotFoundError) as exc:
data: dict[str, Any] = {"provider": self.provider}
if isinstance(exc, KlingAPIError):
data.update(
{
"error_code": exc.code,
"request_id": exc.request_id,
"http_status": exc.http_status,
}
)
data["account_usage_diagnostic"] = account_usage_hint_for_error(exc)
return ToolResult(success=False, data=data, error=f"Kling official video generation failed: {exc}")
except Exception as exc:
return ToolResult(success=False, data={"provider": self.provider}, error=f"Kling official video generation failed: {exc}")
return ToolResult(
success=True,
data={
"provider": self.provider,
"model": request["model"],
"task_id": task_id,
"operation": request["operation"],
"api_family": request["api_family"],
"prompt": inputs["prompt"],
"remote_url": video_url,
"remote_outputs": outputs,
"output": str(paths[0]),
"output_path": str(paths[0]),
"video_paths": [str(path) for path in paths],
"format": "mp4",
"references_used": request.get("references_used", []),
"element_ids": request.get("element_ids", []),
"cost_estimate_confidence": "low",
"cost_estimate_basis": "Conservative estimate pending official account-usage reconciliation.",
**self._account_usage_result(inputs, client),
**self._callback_result_data(inputs, task_id),
**probed,
},
artifacts=[str(path) for path in paths],
cost_usd=self.estimate_cost(inputs),
duration_seconds=round(time.time() - start, 2),
model=request["model"],
)
def _build_request(self, inputs: dict[str, Any]) -> dict[str, Any]:
operation = str(inputs.get("operation", "text_to_video"))
api_family = str(inputs.get("api_family", "classic"))
if operation == "omni_video":
operation = "reference_to_video"
api_family = "omni"
if api_family == "turbo":
return self._build_turbo_request(inputs, operation)
if api_family == "omni":
return self._build_omni_request(inputs, operation)
return self._build_classic_request(inputs, operation)
def _build_classic_request(self, inputs: dict[str, Any], operation: str) -> dict[str, Any]:
if operation == "text_to_video":
payload = self._base_classic_payload(inputs)
payload["prompt"] = self._prompt(inputs)
if inputs.get("negative_prompt"):
payload["negative_prompt"] = inputs["negative_prompt"]
if inputs.get("aspect_ratio"):
payload["aspect_ratio"] = inputs.get("aspect_ratio", "16:9")
self._copy_multi_shot_fields(inputs, payload)
path = "/v1/videos/text2video"
elif operation == "image_to_video":
image = normalize_image_input(inputs.get("reference_image_url"), inputs.get("reference_image_path"))
if not image:
raise ValueError("image_to_video requires reference_image_url or reference_image_path")
payload = self._base_classic_payload(inputs)
payload["image"] = image
if inputs.get("prompt"):
payload["prompt"] = inputs["prompt"]
if inputs.get("negative_prompt"):
payload["negative_prompt"] = inputs["negative_prompt"]
tail = normalize_image_input(inputs.get("reference_tail_image_url"), inputs.get("reference_tail_image_path"))
if tail:
payload["image_tail"] = tail
if inputs.get("element_list"):
payload["element_list"] = normalize_element_list(inputs.get("element_list"))
self._copy_multi_shot_fields(inputs, payload)
path = "/v1/videos/image2video"
else:
raise ValueError(f"Unsupported classic video operation: {operation}")
return {
"protocol": "classic",
"path": path,
"payload": payload,
"operation": operation,
"api_family": "classic",
"model": payload["model_name"],
"references_used": self._reference_metadata_from_classic_payload(payload),
"element_ids": element_ids(payload.get("element_list")),
}
def _build_turbo_request(self, inputs: dict[str, Any], operation: str) -> dict[str, Any]:
settings = {
"resolution": inputs.get("resolution", "720p"),
"duration": int(str(inputs.get("duration", "5"))),
}
options = self._options_payload(inputs)
if operation == "text_to_video":
settings["aspect_ratio"] = inputs.get("aspect_ratio", "16:9")
payload = {"prompt": self._prompt(inputs), "settings": settings}
if options:
payload["options"] = options
path = "/text-to-video/kling-3.0-turbo"
elif operation == "image_to_video":
if inputs.get("reference_image_path") and not inputs.get("reference_image_url"):
raise ValueError("Turbo image_to_video requires reference_image_url; local paths cannot be silently uploaded.")
image_url = inputs.get("reference_image_url")
if not image_url:
raise ValueError("image_to_video requires reference_image_url for api_family=turbo")
contents = [{"type": "prompt", "text": self._prompt(inputs)}, {"type": "first_frame", "url": image_url}]
payload = {"contents": contents, "settings": settings}
if options:
payload["options"] = options
path = "/image-to-video/kling-3.0-turbo"
else:
raise ValueError(f"Unsupported turbo video operation: {operation}")
return {
"protocol": "turbo",
"path": path,
"payload": payload,
"operation": operation,
"api_family": "turbo",
"model": "kling-3.0-turbo",
}
def _build_omni_request(self, inputs: dict[str, Any], operation: str) -> dict[str, Any]:
explicit_model = inputs.get("model_name") or inputs.get("model_variant")
model_name = str(explicit_model or "kling-video-o1")
if model_name not in OMNI_VIDEO_MODELS:
raise ValueError(f"model_name {model_name!r} is not supported for api_family=omni")
payload, references_used, element_id_values = self._build_omni_payload(inputs, operation, model_name)
return {
"protocol": "classic",
"path": "/v1/videos/omni-video",
"payload": payload,
"operation": operation,
"api_family": "omni",
"model": model_name,
"references_used": references_used,
"element_ids": element_id_values,
}
def _build_omni_payload(
self,
inputs: dict[str, Any],
operation: str,
model_name: str,
) -> tuple[dict[str, Any], list[dict[str, Any]], list[int]]:
payload: dict[str, Any] = {
"model_name": model_name,
"prompt": self._prompt(inputs),
"mode": inputs.get("mode", "pro"),
"duration": str(inputs.get("duration", "5")),
}
if inputs.get("sound"):
payload["sound"] = inputs["sound"]
if inputs.get("aspect_ratio"):
payload["aspect_ratio"] = inputs["aspect_ratio"]
self._copy_common_task_fields(inputs, payload)
self._copy_multi_shot_fields(inputs, payload)
references_used: list[dict[str, Any]] = []
image_list, image_refs = self._normalize_omni_image_list(inputs)
references_used.extend(image_refs)
if image_list:
payload["image_list"] = image_list
video_list, video_refs = self._normalize_omni_video_list(inputs)
references_used.extend(video_refs)
if video_list:
payload["video_list"] = video_list
elements = normalize_element_list(inputs.get("element_list"))
element_id_values = [item["element_id"] for item in elements]
if elements:
payload["element_list"] = elements
references_used.extend(
{"kind": "element", "element_id": item["element_id"]}
for item in elements
)
if operation == "reference_to_video" and not any(payload.get(k) for k in ("image_list", "video_list", "element_list")):
raise ValueError("reference_to_video with api_family=omni requires image_list, video_list, element_list, or reference image URLs.")
return payload, references_used, element_id_values
def _base_classic_payload(self, inputs: dict[str, Any]) -> dict[str, Any]:
model_name = str(inputs.get("model_name") or inputs.get("model_variant") or "kling-v3")
if model_name not in CLASSIC_VIDEO_MODELS:
raise ValueError(f"model_name {model_name!r} is not supported for api_family=classic")
payload: dict[str, Any] = {
"model_name": model_name,
"duration": str(inputs.get("duration", "5")),
"mode": inputs.get("mode", "std"),
"sound": inputs.get("sound", "off"),
}
if inputs.get("cfg_scale") is not None:
payload["cfg_scale"] = inputs["cfg_scale"]
if inputs.get("camera_control"):
payload["camera_control"] = inputs["camera_control"]
self._copy_common_task_fields(inputs, payload)
return payload
def _options_payload(self, inputs: dict[str, Any]) -> dict[str, Any]:
options: dict[str, Any] = {}
callback_url = validate_callback_url(inputs.get("callback_url"))
if callback_url:
options["callback_url"] = callback_url
if inputs.get("external_task_id"):
options["external_task_id"] = inputs["external_task_id"]
if "watermark" in inputs:
options["watermark_info"] = {"enabled": bool(inputs.get("watermark"))}
return options
def _copy_common_task_fields(self, inputs: dict[str, Any], payload: dict[str, Any]) -> None:
if "watermark" in inputs:
payload["watermark_info"] = {"enabled": bool(inputs.get("watermark"))}
callback_url = validate_callback_url(inputs.get("callback_url"))
if callback_url:
payload["callback_url"] = callback_url
if inputs.get("external_task_id"):
payload["external_task_id"] = inputs["external_task_id"]
def _copy_multi_shot_fields(self, inputs: dict[str, Any], payload: dict[str, Any]) -> None:
if inputs.get("multi_shot") is None and not inputs.get("multi_prompt"):
return
payload["multi_shot"] = bool(inputs.get("multi_shot", True))
shot_type = str(inputs.get("shot_type") or "customize")
if shot_type not in {"customize", "intelligence"}:
raise ValueError("shot_type must be one of: customize, intelligence")
payload["shot_type"] = shot_type
if inputs.get("multi_prompt"):
if not isinstance(inputs["multi_prompt"], list):
raise ValueError("multi_prompt must be a list")
normalized: list[dict[str, Any]] = []
for item in inputs["multi_prompt"]:
if not isinstance(item, dict) or not item.get("prompt"):
raise ValueError("each multi_prompt item must be an object with prompt")
allowed = {
key: item[key]
for key in ("prompt", "duration", "camera_control", "image_refs", "element_refs")
if key in item
}
normalized.append(allowed)
payload["multi_prompt"] = normalized
def _normalize_omni_image_list(
self,
inputs: dict[str, Any],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
image_list: list[dict[str, Any]] = []
references_used: list[dict[str, Any]] = []
def add_image(value: str | None, *, kind: str, item_type: str | None = None) -> None:
if not value:
return
record = {"image_url": value}
if item_type:
record["type"] = item_type
image_list.append(record)
references_used.append(
{
"kind": "image",
"source": value,
"source_type": kind,
"type": item_type,
}
)
for item in inputs.get("image_list") or []:
if not isinstance(item, dict):
raise ValueError("image_list items must be objects")
value = normalize_image_input(item.get("image_url") or item.get("image"), item.get("image_path"))
if not value:
raise ValueError("image_list items must include image_url, image, or image_path")
record = {"image_url": value}
if item.get("type"):
record["type"] = item["type"]
image_list.append(record)
references_used.append(
{
"kind": "image",
"source": item.get("image_url") or item.get("image_path") or item.get("image"),
"source_type": "image_list",
"type": item.get("type"),
}
)
add_image(
normalize_image_input(inputs.get("reference_image_url"), inputs.get("reference_image_path")),
kind="reference_image",
item_type="first_frame",
)
add_image(
normalize_image_input(inputs.get("reference_tail_image_url"), inputs.get("reference_tail_image_path")),
kind="reference_tail_image",
item_type="end_frame",
)
for url in inputs.get("reference_image_urls") or []:
add_image(normalize_image_input(url=url), kind="reference_image_urls")
for path in inputs.get("reference_image_paths") or []:
add_image(normalize_image_input(path=path), kind="reference_image_paths")
return image_list, references_used
def _normalize_omni_video_list(
self,
inputs: dict[str, Any],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
if inputs.get("reference_video_path") or inputs.get("video_paths"):
raise ValueError("Video Omni requires video URLs; local video paths cannot be silently uploaded.")
video_list: list[dict[str, Any]] = []
references_used: list[dict[str, Any]] = []
def add_video(item: dict[str, Any], source_type: str) -> None:
if item.get("video_path"):
raise ValueError("Video Omni requires video URLs; local video paths cannot be silently uploaded.")
video_url = item.get("video_url") or item.get("url")
if not video_url:
raise ValueError("video_list items must include video_url")
record = {"video_url": video_url}
if item.get("refer_type"):
record["refer_type"] = item["refer_type"]
if "keep_original_sound" in item:
value = item["keep_original_sound"]
record["keep_original_sound"] = "yes" if value is True else "no" if value is False else value
video_list.append(record)
references_used.append(
{
"kind": "video",
"source": video_url,
"source_type": source_type,
"refer_type": record.get("refer_type"),
"keep_original_sound": record.get("keep_original_sound"),
}
)
for item in inputs.get("video_list") or []:
if not isinstance(item, dict):
raise ValueError("video_list items must be objects")
add_video(item, "video_list")
if inputs.get("reference_video_url"):
add_video({"video_url": inputs["reference_video_url"]}, "reference_video_url")
for url in inputs.get("video_urls") or []:
add_video({"video_url": url}, "video_urls")
return video_list, references_used
def _download_videos(
self,
client: KlingClient,
outputs: list[dict[str, Any]],
inputs: dict[str, Any],
) -> list[Path]:
if not outputs:
raise ValueError("Kling video response contained no videos")
base_path = Path(inputs.get("output_path", "kling_official_video.mp4"))
paths: list[Path] = []
for index, item in enumerate(outputs):
url = self._output_url(item)
suffix = extension_from_url(url, ".mp4")
output_path = numbered_output_path(output_path_with_suffix(base_path, suffix), index, suffix)
client.download(url, output_path)
paths.append(output_path)
return paths
@staticmethod
def _output_url(item: dict[str, Any]) -> str:
url = item.get("url") or item.get("video_url") or item.get("resource_url")
if url:
return str(url)
resource = item.get("resource") or {}
if isinstance(resource, dict) and resource.get("url"):
return str(resource["url"])
raise ValueError(f"Kling video response contained no downloadable URL: {item}")
@staticmethod
def _reference_metadata_from_classic_payload(payload: dict[str, Any]) -> list[dict[str, Any]]:
references: list[dict[str, Any]] = []
if payload.get("image"):
references.append({"kind": "image", "source_type": "reference_image"})
if payload.get("image_tail"):
references.append({"kind": "image", "source_type": "reference_tail_image"})
if payload.get("element_list"):
references.extend(
{"kind": "element", "element_id": item["element_id"]}
for item in normalize_element_list(payload.get("element_list"))
)
return references
@staticmethod
def _callback_result_data(inputs: dict[str, Any], task_id: str) -> dict[str, Any]:
callback_url = inputs.get("callback_url")
if not callback_url:
return {}
return {
"callback_url": str(callback_url),
"callback_requested": True,
"polling_used": True,
"task_id": task_id,
}
@staticmethod
def _account_usage_result(inputs: dict[str, Any], client: KlingClient) -> dict[str, Any]:
if not inputs.get("include_account_usage"):
return {}
try:
usage = get_account_costs(client=client)
return {
"account_usage": usage,
"cost_source": "estimate_with_account_usage_context",
"reconciled_cost_usd": None,
}
except Exception as exc:
return {
"account_usage_error": str(exc),
"cost_source": "estimate",
"reconciled_cost_usd": None,
}
@staticmethod
def _estimate_reference_count(inputs: dict[str, Any]) -> int:
count = 0
for key in (
"image_list",
"video_list",
"element_list",
"reference_image_urls",
"reference_image_paths",
"video_urls",
):
count += len(inputs.get(key) or [])
for key in (
"reference_image_url",
"reference_image_path",
"reference_tail_image_url",
"reference_tail_image_path",
"reference_video_url",
):
if inputs.get(key):
count += 1
return count
@staticmethod
def _prompt(inputs: dict[str, Any]) -> str:
prompt = str(inputs.get("prompt") or "").strip()
if not prompt:
raise ValueError("prompt is required")
return prompt
@staticmethod
def _first_output_url(outputs: list[dict[str, Any]]) -> str:
for item in outputs:
try:
return KlingOfficialVideo._output_url(item)
except ValueError:
continue
raise ValueError(f"Kling video response contained no downloadable URL: {outputs}")

60
tools/video/video_selector.py
View File

@@ -107,6 +107,38 @@ class VideoSelector(BaseTool):
"items": {"type": "string"},
"description": "Local reference image paths for providers that support reference-conditioned video.",
},
"reference_video_url": {
"type": "string",
"description": "Reference video URL for providers that support video-conditioned generation.",
},
"reference_video_path": {
"type": "string",
"description": "Local reference video path. Providers that require URLs should reject this clearly.",
},
"image_list": {
"type": "array",
"description": "Provider-specific list of image references, e.g. Kling Official Video Omni.",
},
"video_list": {
"type": "array",
"description": "Provider-specific list of video references, e.g. Kling Official Video Omni.",
},
"element_list": {
"type": "array",
"description": "Provider-specific element references, e.g. Kling Official element_id objects.",
},
"multi_shot": {
"type": "boolean",
"description": "Provider-specific multi-shot mode.",
},
"shot_type": {
"type": "string",
"description": "Provider-specific multi-shot type.",
},
"multi_prompt": {
"type": "array",
"description": "Structured multi-shot prompts; not inferred from prose.",
},
"image_url": {
"type": "string",
"description": "Alias for reference_image_url (used by some providers like Kling via fal.ai).",
@@ -115,6 +147,34 @@ class VideoSelector(BaseTool):
"type": "string",
"description": "Resolution hint for providers that support named output resolutions.",
},
"api_family": {
"type": "string",
"description": "Provider-specific API family hint passed through when supported, e.g. classic/turbo/omni.",
},
"model_name": {
"type": "string",
"description": "Provider-specific model name passed through when supported.",
},
"mode": {
"type": "string",
"description": "Provider-specific quality mode passed through when supported.",
},
"sound": {
"type": "string",
"description": "Provider-specific native audio toggle passed through when supported.",
},
"watermark": {
"type": "boolean",
"description": "Provider-specific watermark toggle passed through when supported.",
},
"callback_url": {
"type": "string",
"description": "Provider-specific callback URL. Current OpenMontage providers still poll by default.",
},
"external_task_id": {
"type": "string",
"description": "Provider-specific idempotency/provenance task id.",
},
"workflow_json": {
"type": "string",
"description": (