From 8e1128400b40eb5b17c73785a525d63ee35bfcda Mon Sep 17 00:00:00 2001 From: xucailiang <74602715+xucailiang@users.noreply.github.com> Date: Tue, 7 Jul 2026 15:34:23 +0800 Subject: [PATCH] Clean up Kling official PR artifacts --- .agents/skills/kling-official/SKILL.md | 4 +- .agents/skills/video_toolkit/SKILL.md | 456 ------- diagram.png | Bin 5336 -> 5915 bytes docs/ARCHITECTURE.md | 14 +- docs/kling-official-integration-plan.md | 1133 ----------------- docs/kling-official-phase-1-core.md | 784 ------------ .../kling-official-phase-2-omni-operations.md | 464 ------- ...g-official-phase-3-media-avatar-effects.md | 561 -------- tests/contracts/test_kling_official_core.py | 2 +- 9 files changed, 10 insertions(+), 3408 deletions(-) delete mode 100644 .agents/skills/video_toolkit/SKILL.md delete mode 100644 docs/kling-official-integration-plan.md delete mode 100644 docs/kling-official-phase-1-core.md delete mode 100644 docs/kling-official-phase-2-omni-operations.md delete mode 100644 docs/kling-official-phase-3-media-avatar-effects.md diff --git a/.agents/skills/kling-official/SKILL.md b/.agents/skills/kling-official/SKILL.md index f2ffd86e..a1939e95 100644 --- a/.agents/skills/kling-official/SKILL.md +++ b/.agents/skills/kling-official/SKILL.md @@ -69,7 +69,7 @@ Local image references may be normalized through `tools/_kling/media.py`. Local Image Omni accepts `image_list[]` with official `image` values. Prompt placeholders such as `<<>>` must map stably to the provided image order. If the prompt already contains placeholders, validate that the referenced images exist and do not insert duplicates. -## Phase 3 Capability Boundaries +## Capability Boundaries TTS, avatar, and lip sync are provider additions to existing OpenMontage capabilities. Audio effects and video effects are official Kling endpoints, but they are not registered as default OpenMontage tools until a pipeline has a stable capability slot for them. @@ -148,7 +148,7 @@ Local video paths must not be silently uploaded through fal.ai or any other prov ## Audio Effects And Video Effects -Official Kling audio effects (`/v1/audio/text-to-audio`, `/v1/audio/video-to-audio`) and video effects (`/v1/videos/effects`) are intentionally not default OpenMontage selector tools in Phase 3. Record the non-mapping reason in docs/tests instead of registering tools that current pipelines might misuse. +Official Kling audio effects (`/v1/audio/text-to-audio`, `/v1/audio/video-to-audio`) and video effects (`/v1/videos/effects`) are intentionally not default OpenMontage selector tools. Record the non-mapping reason in docs/tests instead of registering tools that current pipelines might misuse. ## Elements Helper diff --git a/.agents/skills/video_toolkit/SKILL.md b/.agents/skills/video_toolkit/SKILL.md deleted file mode 100644 index 88d337f0..00000000 --- a/.agents/skills/video_toolkit/SKILL.md +++ /dev/null @@ -1,456 +0,0 @@ ---- -name: video-toolkit -description: Create professional videos autonomously using Codex-video-toolkit — AI voiceovers, image generation, music, talking heads, and Remotion rendering. -metadata: - openclaw: - emoji: "🎬" - skillKey: "video-toolkit" - os: ["darwin", "linux"] - requires: - bins: ["node", "python3", "ffmpeg", "npm"] ---- - -# Video Toolkit - -Create professional explainer videos from a text brief. The toolkit uses open-source AI models on cloud GPUs (Modal or RunPod) for voiceover, image generation, music, and talking head animation. Remotion (React) handles composition and rendering. - -## CRITICAL: Toolkit Path - -The toolkit lives at a fixed path. **ALWAYS `cd` here before running any tool command.** - -```bash -TOOLKIT=~/.openclaw/workspace/Codex-video-toolkit -cd $TOOLKIT -``` - -**NEVER run tool commands from inside a project directory.** Tools resolve paths relative to the toolkit root. - -## Setup - -### Step 1: Check Current State - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/verify_setup.py -``` - -If everything shows `[x]`, skip to "Quick Test" below. Otherwise continue setup. - -### Step 2: Install Python Dependencies - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -pip3 install --break-system-packages -r tools/requirements.txt -``` - -Note: `--break-system-packages` is needed on Debian/Ubuntu with managed Python (PEP 668). Safe inside containers. - -### Step 3: Configure Cloud GPU Endpoints - -The toolkit needs cloud GPU endpoint URLs in `.env`. Check if `.env` exists and has Modal endpoints: - -```bash -cat ~/.openclaw/workspace/Codex-video-toolkit/.env | grep MODAL -``` - -If Modal endpoints are configured, you're ready. If not, **ask the user to provide Modal endpoint URLs** or set up Modal: - -```bash -pip3 install --break-system-packages modal -python3 -m modal setup # Opens browser for authentication - -# Deploy each tool — capture the endpoint URL from output -cd ~/.openclaw/workspace/Codex-video-toolkit -modal deploy docker/modal-qwen3-tts/app.py -modal deploy docker/modal-flux2/app.py -modal deploy docker/modal-music-gen/app.py -modal deploy docker/modal-sadtalker/app.py -modal deploy docker/modal-image-edit/app.py -modal deploy docker/modal-upscale/app.py -modal deploy docker/modal-propainter/app.py -modal deploy docker/modal-ltx2/app.py # Requires: modal secret create huggingface-token HF_TOKEN=hf_... -``` - -**LTX-2 prerequisite:** Before deploying LTX-2, create a HuggingFace secret and accept the [Gemma 3 license](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized): -```bash -modal secret create huggingface-token HF_TOKEN=hf_your_read_access_token -``` - -Add each URL to `.env`: -``` -MODAL_QWEN3_TTS_ENDPOINT_URL=https://...modal.run -MODAL_FLUX2_ENDPOINT_URL=https://...modal.run -MODAL_MUSIC_GEN_ENDPOINT_URL=https://...modal.run -MODAL_SADTALKER_ENDPOINT_URL=https://...modal.run -MODAL_IMAGE_EDIT_ENDPOINT_URL=https://...modal.run -MODAL_UPSCALE_ENDPOINT_URL=https://...modal.run -MODAL_DEWATERMARK_ENDPOINT_URL=https://...modal.run -MODAL_LTX2_ENDPOINT_URL=https://...modal.run -``` - -Optional but recommended — Cloudflare R2 for reliable file transfer: -``` -R2_ACCOUNT_ID=... -R2_ACCESS_KEY_ID=... -R2_SECRET_ACCESS_KEY=... -R2_BUCKET_NAME=video-toolkit -``` - -### Step 4: Verify and Quick Test - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/verify_setup.py -``` - -All tools should show `[x]`. Then run a quick test to confirm the GPU pipeline works: - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/qwen3_tts.py --text "Hello, this is a test." --speaker Ryan --tone warm --output /tmp/video-toolkit-test.mp3 --cloud modal -``` - -If you get a valid .mp3 file, setup is complete. If it fails, check: -- `.env` has the correct `MODAL_QWEN3_TTS_ENDPOINT_URL` -- Run `python3 tools/verify_setup.py --json` and check `modal_tools` for which endpoints are missing - -**Cost:** Modal includes $30/month free compute. A typical 60s video costs $1-3. - ---- - -## Creating a Video - -### Step 1: Create Project - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -cp -r templates/product-demo projects/PROJECT_NAME -cd projects/PROJECT_NAME -npm install -``` - -Templates: `product-demo` (marketing/explainer), `sprint-review`, `sprint-review-v2` (composable scenes). - -### Step 2: Write Config - -Edit `projects/PROJECT_NAME/src/config/demo-config.ts`: - -```typescript -export const demoConfig: ProductDemoConfig = { - product: { - name: 'My Product', - tagline: 'What it does in one line', - website: 'example.com', - }, - scenes: [ - { type: 'title', durationSeconds: 9, content: { headline: '...', subheadline: '...' } }, - { type: 'problem', durationSeconds: 14, content: { headline: '...', problems: ['...', '...'] } }, - { type: 'solution', durationSeconds: 13, content: { headline: '...', highlights: ['...', '...'] } }, - { type: 'stats', durationSeconds: 12, content: { stats: [{value: '99%', label: '...'}, ...] } }, - { type: 'cta', durationSeconds: 10, content: { headline: '...', links: ['...'] } }, - ], - audio: { - backgroundMusicFile: 'audio/bg-music.mp3', - backgroundMusicVolume: 0.12, - }, -}; -``` - -Scene types: `title`, `problem`, `solution`, `demo`, `feature`, `stats`, `cta`. - -**Duration rule:** Estimate `durationSeconds` as `ceil(word_count / 2.5) + 2`. You will adjust this after generating audio in Step 4. - -### Step 3: Write Voiceover Script - -Create `projects/PROJECT_NAME/VOICEOVER-SCRIPT.md`: - -```markdown -## Scene 1: Title (9s, ~17 words) -Build videos with AI. The product name toolkit makes it easy. - -## Scene 2: Problem (14s, ~30 words) -The problem statement goes here. Keep it punchy and relatable. -``` - -**Word budget per scene:** `(durationSeconds - 2) * 2.5` words. The -2 accounts for 1s audio delay + 1s padding. - -### Step 4: Generate Assets - -**CRITICAL: All commands below MUST be run from the toolkit root, not the project directory.** - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -``` - -#### 4a. Background Music - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/music_gen.py \ - --preset corporate-bg \ - --duration 90 \ - --output projects/PROJECT_NAME/public/audio/bg-music.mp3 \ - --cloud modal -``` - -Presets: `corporate-bg`, `upbeat-tech`, `ambient`, `dramatic`, `tension`, `hopeful`, `cta`, `lofi`. - -#### 4b. Voiceover (per-scene) - -Generate ONE .mp3 file PER SCENE. Do NOT generate a single voiceover file. - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit - -# Scene 01 -python3 tools/qwen3_tts.py \ - --text "The voiceover text for scene one." \ - --speaker Ryan --tone warm \ - --output projects/PROJECT_NAME/public/audio/scenes/01.mp3 \ - --cloud modal - -# Scene 02 -python3 tools/qwen3_tts.py \ - --text "The voiceover text for scene two." \ - --speaker Ryan --tone warm \ - --output projects/PROJECT_NAME/public/audio/scenes/02.mp3 \ - --cloud modal - -# ... repeat for each scene -``` - -**Speakers:** `Ryan`, `Aiden`, `Vivian`, `Serena`, `Uncle_Fu`, `Dylan`, `Eric`, `Ono_Anna`, `Sohee` -**Tones:** `neutral`, `warm`, `professional`, `excited`, `calm`, `serious`, `storyteller`, `tutorial` - -For voice cloning (needs a reference recording): -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/qwen3_tts.py \ - --text "Text to speak" \ - --ref-audio assets/voices/reference.m4a \ - --ref-text "Exact transcript of the reference audio" \ - --output projects/PROJECT_NAME/public/audio/scenes/01.mp3 \ - --cloud modal -``` - -#### 4c. Scene Images - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/flux2.py \ - --prompt "Dark tech background with blue geometric grid, cinematic lighting" \ - --width 1920 --height 1080 \ - --output projects/PROJECT_NAME/public/images/title-bg.png \ - --cloud modal -``` - -Image presets (use `--preset` instead of `--prompt --width --height`): -`title-bg`, `problem`, `solution`, `demo-bg`, `stats-bg`, `cta`, `thumbnail`, `portrait-bg` - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/flux2.py \ - --preset title-bg \ - --output projects/PROJECT_NAME/public/images/title-bg.png \ - --cloud modal -``` - -#### 4d. Video Clips — B-Roll & Animated Backgrounds (optional) - -Generate AI video clips for b-roll cutaways, animated slide backgrounds, or intro/outro sequences: - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit - -# B-roll clip from text -python3 tools/ltx2.py \ - --prompt "Aerial drone shot over a European city at golden hour, cinematic wide angle" \ - --output projects/PROJECT_NAME/public/videos/broll-europe.mp4 \ - --cloud modal - -# Animate a slide/screenshot (image-to-video) -python3 tools/ltx2.py \ - --prompt "Gentle particle effects, soft ambient light shifts, very slight camera drift" \ - --input projects/PROJECT_NAME/public/images/title-bg.png \ - --output projects/PROJECT_NAME/public/videos/animated-title.mp4 \ - --cloud modal - -# Abstract intro/outro background -python3 tools/ltx2.py \ - --prompt "Dark moody abstract background with flowing blue light streaks, bokeh particles, cinematic" \ - --output projects/PROJECT_NAME/public/videos/intro-bg.mp4 \ - --cloud modal -``` - -Use in Remotion compositions with ``: -```tsx - -``` - -**LTX-2 rules:** -- Max ~8 seconds per clip (193 frames at 24fps). Default is ~5s (121 frames). -- Width/height must be divisible by 64. Default: 768x512. -- ~$0.20-0.25 per clip, ~2.5 min generation time. -- Cold start ~60-90s. Subsequent clips on warm GPU are faster. -- Generated audio is ambient only — use voiceover/music tools for speech and music. -- ~30% of generations may have training data artifacts (logos/text). Re-run with `--seed` to vary. - -#### 4e. Talking Head Narrator (optional) - -Generate a presenter portrait, then animate per-scene clips: - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit - -# 1. Generate portrait -python3 tools/flux2.py \ - --prompt "Professional presenter portrait, clean style, dark background, facing camera, upper body" \ - --width 1024 --height 576 \ - --output projects/PROJECT_NAME/public/images/presenter.png \ - --cloud modal - -# 2. Generate per-scene narrator clips (one per scene, NOT one long video) -python3 tools/sadtalker.py \ - --image projects/PROJECT_NAME/public/images/presenter.png \ - --audio projects/PROJECT_NAME/public/audio/scenes/01.mp3 \ - --preprocess full --still --expression-scale 0.8 \ - --output projects/PROJECT_NAME/public/narrator-01.mp4 \ - --cloud modal - -# Repeat for each scene that needs a narrator -``` - -**SadTalker rules — follow these exactly:** -- **ALWAYS** use `--preprocess full` (default `crop` outputs a square, wrong aspect ratio) -- **ALWAYS** use `--still` (reduces head movement, looks professional) -- **ALWAYS** generate per-scene clips (6-15s each), NEVER one long video -- Processing: ~3-4 min per 10s of audio on Modal A10G -- `--expression-scale 0.8` keeps expressions subtle (range 0.0-1.5) - -#### 4e. Image Editing (optional) - -Create scene variants from existing images: - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/image_edit.py \ - --input projects/PROJECT_NAME/public/images/title-bg.png \ - --prompt "Make it darker with red tones, more ominous" \ - --output projects/PROJECT_NAME/public/images/problem-bg.png \ - --cloud modal -``` - -#### 4f. Upscaling (optional) - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -python3 tools/upscale.py \ - --input projects/PROJECT_NAME/public/images/some-image.png \ - --output projects/PROJECT_NAME/public/images/some-image-4x.png \ - --scale 4 --cloud modal -``` - -### Step 5: Sync Timing - -**ALWAYS do this after generating voiceover.** Audio duration differs from estimates. - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit -for f in projects/PROJECT_NAME/public/audio/scenes/*.mp3; do - echo "$(basename $f): $(ffprobe -v error -show_entries format=duration -of csv=p=0 "$f")s" -done -``` - -Update each scene's `durationSeconds` in `demo-config.ts` to: `ceil(actual_audio_duration + 2)`. - -Example: if `01.mp3` is 6.8s, set scene 1 `durationSeconds` to `9` (ceil(6.8 + 2) = 9). - -### Step 6: Review Still Frames - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit/projects/PROJECT_NAME -npx remotion still src/index.ts ProductDemo --frame=100 --output=/tmp/review-scene1.png -npx remotion still src/index.ts ProductDemo --frame=400 --output=/tmp/review-scene2.png -``` - -Check: text truncation, animation timing, narrator PiP positioning, background contrast. - -### Step 7: Render - -```bash -cd ~/.openclaw/workspace/Codex-video-toolkit/projects/PROJECT_NAME -npm run render -``` - -**Output:** `out/ProductDemo.mp4` - ---- - -## Composition Patterns - -### Per-Scene Audio - -Use per-scene audio with a 1-second delay (`from={30}` = 30 frames = 1s at 30fps): - -```tsx - - -``` - -### Per-Scene Narrator PiP - -```tsx - - - -``` - -**ALWAYS use ``, NEVER `