Merge commit 'f72efb53d3cea7cce0943d5dc4e55e33e4e59af3' into codex/repair-pr-353

# Conflicts:
#	AGENT_GUIDE.md
#	docs/PROVIDERS.md
This commit is contained in:
calesthio
2026-08-13 11:40:09 -07:00
12 changed files with 910 additions and 15 deletions

View File

@@ -0,0 +1,110 @@
---
name: azure-text-to-speech
description: Generate neural narration audio using Azure AI Speech (REST text-to-speech). Use when synthesizing voiceovers or narration in OpenMontage. Optional cloud TTS provider — preferred when AZURE_SPEECH_KEY is configured; the local piper_tts remains the default offline path. Shares one Speech resource with azure_stt.
license: MIT
compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
---
# Azure AI Speech — Text-to-Speech
Generate narration with **Azure neural TTS** — high-quality multilingual voices,
SSML prosody control, and express-as styles, served synchronously by the REST
`/cognitiveservices/v1` endpoint (no token exchange, Blob storage, or job
polling). In OpenMontage this is exposed through the `azure_tts` tool
(`capability=tts`, `provider=azure`). It is an **optional cloud TTS provider**
when `AZURE_SPEECH_KEY` is configured, prefer it for high-quality cloud
narration. The local `piper_tts` remains the **default offline path** and the
fallback when Azure is unavailable; `elevenlabs_tts` remains the choice for
voice cloning.
> Docs: [REST text to speech](https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech) · [Voice gallery](https://speech.microsoft.com/portal/voicegallery)
## Setup
Same Speech resource as `azure_stt`**one key/region unlocks both directions**
(STT and TTS). Create a **Speech** resource in the
[Azure portal](https://portal.azure.com); copy the key and region from its
**Keys and Endpoint** page.
```bash
export AZURE_SPEECH_KEY=your_speech_resource_key
export AZURE_SPEECH_REGION=eastus # your resource's region
# export AZURE_TTS_ENDPOINT=https://... # optional: full custom TTS host
# (the TTS host is https://<region>.tts.speech.microsoft.com — a different
# subdomain than the STT endpoint, hence the separate override var)
```
`azure_tts` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
`AZURE_SPEECH_REGION` or `AZURE_TTS_ENDPOINT` are set.
## Using it in a pipeline
Route through `tts_selector` as usual (it auto-discovers `azure_tts`), or call
the provider tool directly when the user has approved Azure:
```python
from tools.tool_registry import registry
registry.discover()
tts = registry._tools["azure_tts"]
result = tts.execute({
"text": "Every design decision in this dashboard has a reason.",
"voice": "andrew", # alias or full Azure short name
"rate": "-4%", # slightly slower for narration
# "style": "narration-professional", # for voices that support styles
"output_path": "projects/my-video/assets/audio/seg_001.mp3",
"output_format": "mp3", # or "wav" (48kHz PCM) for mixing
})
```
If `azure_tts` is unavailable (no key) or errors, fall back per its declared
chain: `elevenlabs_tts``openai_tts``piper_tts`.
## Voice selection
Curated shortlist (aliases accepted by the `voice` param):
| Alias | Voice | Character |
|-------|-------|-----------|
| `andrew` | en-US-AndrewMultilingualNeural | warm, confident, conversational — the default; founder/explainer register |
| `brandon` | en-US-BrandonMultilingualNeural | deeper, measured |
| `ava` | en-US-AvaMultilingualNeural | confident, bright female |
| `guy` | en-US-GuyNeural | authoritative |
| `jenny` | en-US-JennyNeural | friendly, clear |
Any valid Azure voice short name may be passed verbatim (e.g.
`de-DE-KatjaNeural`); the *Multilingual* voices handle non-English text well —
set `locale` to match the text's language for correct SSML.
## Parameters that matter
- **`rate` / `pitch`** — SSML prosody. Narration usually reads best slightly
slowed (`"-4%"` to `"-8%"`); leave pitch at `"0%"` unless correcting a voice.
- **`style`** — express-as style for voices that support it
(`narration-professional`, `calm`, `newscast`). Unsupported styles are
silently ignored by Azure, so listen to a sample before batch runs.
- **`output_format`** — `mp3` (48kHz/192kbit) for delivery, `wav` (48kHz PCM)
when the segment feeds `audio_mixer` for further processing.
- Determinism: a fixed voice + SSML re-renders effectively identical audio —
safe to regenerate individual segments without re-recording the whole set.
## Cost
Azure neural TTS Standard tier bills roughly **$16 per 1M characters** (~$0.016
per 1k chars; a 150-word narration segment ≈ $0.015). The tool reports
per-call `cost_usd` for the cost tracker. See
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
## Limits & tips
- One `execute` call = one narration segment. Generate per script section (the
asset stage convention) rather than one giant paragraph — smaller segments
align cleanly to scene timings and are cheap to regenerate.
- The synchronous endpoint caps a request at 10 minutes of audio — far above
any segment OpenMontage generates.
- Text is XML-escaped automatically; do not pre-escape or wrap in SSML — pass
plain text plus the `rate`/`pitch`/`style` params.
- Verify quality: listen to the first generated segment before batch-running a
full script (voice/style fit is a creative decision — surface it at the
proposal stage per the Decision Communication Contract).

View File

@@ -0,0 +1,110 @@
---
name: azure-text-to-speech
description: Generate neural narration audio using Azure AI Speech (REST text-to-speech). Use when synthesizing voiceovers or narration in OpenMontage. Optional cloud TTS provider — preferred when AZURE_SPEECH_KEY is configured; the local piper_tts remains the default offline path. Shares one Speech resource with azure_stt.
license: MIT
compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
---
# Azure AI Speech — Text-to-Speech
Generate narration with **Azure neural TTS** — high-quality multilingual voices,
SSML prosody control, and express-as styles, served synchronously by the REST
`/cognitiveservices/v1` endpoint (no token exchange, Blob storage, or job
polling). In OpenMontage this is exposed through the `azure_tts` tool
(`capability=tts`, `provider=azure`). It is an **optional cloud TTS provider**
when `AZURE_SPEECH_KEY` is configured, prefer it for high-quality cloud
narration. The local `piper_tts` remains the **default offline path** and the
fallback when Azure is unavailable; `elevenlabs_tts` remains the choice for
voice cloning.
> Docs: [REST text to speech](https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech) · [Voice gallery](https://speech.microsoft.com/portal/voicegallery)
## Setup
Same Speech resource as `azure_stt`**one key/region unlocks both directions**
(STT and TTS). Create a **Speech** resource in the
[Azure portal](https://portal.azure.com); copy the key and region from its
**Keys and Endpoint** page.
```bash
export AZURE_SPEECH_KEY=your_speech_resource_key
export AZURE_SPEECH_REGION=eastus # your resource's region
# export AZURE_TTS_ENDPOINT=https://... # optional: full custom TTS host
# (the TTS host is https://<region>.tts.speech.microsoft.com — a different
# subdomain than the STT endpoint, hence the separate override var)
```
`azure_tts` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
`AZURE_SPEECH_REGION` or `AZURE_TTS_ENDPOINT` are set.
## Using it in a pipeline
Route through `tts_selector` as usual (it auto-discovers `azure_tts`), or call
the provider tool directly when the user has approved Azure:
```python
from tools.tool_registry import registry
registry.discover()
tts = registry._tools["azure_tts"]
result = tts.execute({
"text": "Every design decision in this dashboard has a reason.",
"voice": "andrew", # alias or full Azure short name
"rate": "-4%", # slightly slower for narration
# "style": "narration-professional", # for voices that support styles
"output_path": "projects/my-video/assets/audio/seg_001.mp3",
"output_format": "mp3", # or "wav" (48kHz PCM) for mixing
})
```
If `azure_tts` is unavailable (no key) or errors, fall back per its declared
chain: `elevenlabs_tts``openai_tts``piper_tts`.
## Voice selection
Curated shortlist (aliases accepted by the `voice` param):
| Alias | Voice | Character |
|-------|-------|-----------|
| `andrew` | en-US-AndrewMultilingualNeural | warm, confident, conversational — the default; founder/explainer register |
| `brandon` | en-US-BrandonMultilingualNeural | deeper, measured |
| `ava` | en-US-AvaMultilingualNeural | confident, bright female |
| `guy` | en-US-GuyNeural | authoritative |
| `jenny` | en-US-JennyNeural | friendly, clear |
Any valid Azure voice short name may be passed verbatim (e.g.
`de-DE-KatjaNeural`); the *Multilingual* voices handle non-English text well —
set `locale` to match the text's language for correct SSML.
## Parameters that matter
- **`rate` / `pitch`** — SSML prosody. Narration usually reads best slightly
slowed (`"-4%"` to `"-8%"`); leave pitch at `"0%"` unless correcting a voice.
- **`style`** — express-as style for voices that support it
(`narration-professional`, `calm`, `newscast`). Unsupported styles are
silently ignored by Azure, so listen to a sample before batch runs.
- **`output_format`** — `mp3` (48kHz/192kbit) for delivery, `wav` (48kHz PCM)
when the segment feeds `audio_mixer` for further processing.
- Determinism: a fixed voice + SSML re-renders effectively identical audio —
safe to regenerate individual segments without re-recording the whole set.
## Cost
Azure neural TTS Standard tier bills roughly **$16 per 1M characters** (~$0.016
per 1k chars; a 150-word narration segment ≈ $0.015). The tool reports
per-call `cost_usd` for the cost tracker. See
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
## Limits & tips
- One `execute` call = one narration segment. Generate per script section (the
asset stage convention) rather than one giant paragraph — smaller segments
align cleanly to scene timings and are cheap to regenerate.
- The synchronous endpoint caps a request at 10 minutes of audio — far above
any segment OpenMontage generates.
- Text is XML-escaped automatically; do not pre-escape or wrap in SSML — pass
plain text plus the `rate`/`pitch`/`style` params.
- Verify quality: listen to the first generated segment before batch-running a
full script (voice/style fit is a creative decision — surface it at the
proposal stage per the Decision Communication Contract).

View File

@@ -114,14 +114,13 @@ UNSPLASH_ACCESS_KEY=
# --- Analysis ---
# HuggingFace token — enables speaker diarization in transcriber.
HF_TOKEN=
# Speech-to-text: optional Azure AI Speech (Fast Transcription). When set, the
# agent prefers azure_stt for cloud STT; the local faster-whisper transcriber
# remains the default offline path.
# Azure AI Speech resource key ('Keys and Endpoint' page).
# Speech: optional Azure AI Speech. One key/region unlocks both directions —
# azure_stt (Fast Transcription cloud STT) and azure_tts (neural cloud TTS).
# The local faster-whisper transcriber / piper_tts remain the default offline paths.
AZURE_SPEECH_KEY=
# Speech resource region, e.g. eastus.
AZURE_SPEECH_REGION=
# AZURE_SPEECH_ENDPOINT= # Optional: full custom endpoint URL (overrides region)
# AZURE_SPEECH_ENDPOINT= # Optional: full custom STT endpoint URL (overrides region)
# AZURE_TTS_ENDPOINT= # Optional: full custom TTS host (e.g. https://<region>.tts.speech.microsoft.com)
# --- Avatar (local installs) ---
# WAV2LIP_PATH= # Path to cloned Wav2Lip repo (for lip sync)

View File

@@ -505,7 +505,7 @@ Three selector tools abstract multi-provider capabilities. **Selectors auto-disc
| Selector | Routes to | How it discovers |
|----------|-----------|-----------------|
| `tts_selector` | All tools with `capability="tts"` (ElevenLabs, Google TTS, OpenAI, Piper, fish.audio) | `registry.get_by_capability("tts")` |
| `tts_selector` | All tools with `capability="tts"` (ElevenLabs, Google TTS, OpenAI, Piper, Azure, fish.audio) | `registry.get_by_capability("tts")` |
| `image_selector` | All tools with `capability="image_generation"` (FLUX, Google Imagen, GPT Image, Recraft, etc.) | `registry.get_by_capability("image_generation")` |
| `video_selector` | All tools with `capability="video_generation"` | `registry.get_by_capability("video_generation")` |
@@ -682,7 +682,7 @@ The `.agents/skills/` directory is large. When you're not coming in through a to
| **Character animation** | `character-rigging`, `svg-character-animation`, `pose-library-design`, `canvas-procedural-animation`, `character-animation-qa` |
| **Image generation** | `bfl-api`, `flux-best-practices` |
| **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` |
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` |
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `azure-text-to-speech` (optional cloud TTS — tool `azure_tts`, same Speech key as `azure_stt`), `setup-api-key` |
| **Speech-to-text** | `speech-to-text` (whisper `transcriber` — default, offline), `azure-speech-to-text` (optional cloud STT — tool `azure_stt`, preferred when `AZURE_SPEECH_KEY` is set) |
| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `agents` |
| **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) |

View File

@@ -45,7 +45,7 @@ OpenMontage/
│ ├── tool_registry.py # Auto-discovery singleton registry
│ ├── cost_tracker.py # Budget governance (estimate → reserve → reconcile)
│ ├── analysis/ # Transcription, scene detection, frame sampling, video understanding
│ ├── audio/ # TTS (ElevenLabs, OpenAI, Piper), music gen, mixing, enhancement
│ ├── audio/ # TTS (ElevenLabs, OpenAI, Piper, Azure, Google), music gen, mixing, enhancement
│ ├── avatar/ # Talking head animation, lip sync
│ ├── enhancement/ # Upscale, bg removal, face enhance/restore, color grading
│ ├── graphics/ # Image gen (FLUX, GPT Image, Recraft, local diffusion), stock, diagrams, code snippets, math animation
@@ -150,9 +150,9 @@ Selectors route based on: user preference when explicitly set, then scored ranki
### Tool Inventory by Category
**Analysis (4):** transcriber (WhisperX), scene_detect, frame_sampler, video_understand (CLIP/BLIP-2)
**Analysis (5):** transcriber (WhisperX), azure_stt, scene_detect, frame_sampler, video_understand (CLIP/BLIP-2)
**Audio (8):** elevenlabs_tts, google_tts, openai_tts, piper_tts, tts_selector, music_gen, audio_mixer, audio_enhance
**Audio (9):** elevenlabs_tts, google_tts, openai_tts, piper_tts, azure_tts, tts_selector, music_gen, audio_mixer, audio_enhance
**Avatar (2):** talking_head (SadTalker/MuseTalk), lip_sync (Wav2Lip)
@@ -383,6 +383,7 @@ All config is validated via Pydantic models in `lib/config_model.py`.
| Variable | Used By | Purpose |
|----------|---------|---------|
| `ELEVENLABS_API_KEY` | elevenlabs_tts, music_gen | TTS, music, sound effects |
| `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | azure_stt, azure_tts | Azure AI Speech cloud transcription + neural TTS (one resource, both directions) |
| `OPENAI_API_KEY` | openai_tts, openai_image | TTS fallback, GPT Image 2 |
| `XAI_API_KEY` | grok_image, grok_video | Grok image editing/generation, Grok video generation |
| `FAL_KEY` | flux_image, kling_video, veo_video, minimax_video, recraft_image | fal.ai hosted models (FLUX, Veo, Kling, MiniMax, Recraft) |

View File

@@ -47,8 +47,8 @@ DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin nar
DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type
DASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps)
# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default)
AZURE_SPEECH_KEY= # Azure AI Speech — Fast Transcription (word-level timestamps)
# AZURE AI SPEECH (optional cloud STT + TTS; one key unlocks both directions)
AZURE_SPEECH_KEY= # Azure AI Speech — azure_stt (Fast Transcription) + azure_tts (neural narration)
AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
# MULTI-MODEL GATEWAY (one key, 6+ tools)
@@ -625,6 +625,64 @@ allowance). OpenMontage estimates cost from the transcribed audio duration. See
---
### Azure AI Speech — Text-to-Speech
> **Cloud neural narration.** Azure neural TTS delivers high-quality multilingual voices with SSML prosody control and express-as styles — same Speech resource as `azure_stt`, so one key/region unlocks both directions. Optional: the local `piper_tts` remains the default offline TTS path. When `AZURE_SPEECH_KEY` is set, the agent may prefer `azure_tts` for cloud narration.
**Tools unlocked:** `azure_tts`
**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_TTS_ENDPOINT`)
#### Setup
Identical to the STT setup above — the same Speech resource key and region work
for both. If you already configured `azure_stt`, `azure_tts` is available now.
```bash
AZURE_SPEECH_KEY=your-speech-resource-key
AZURE_SPEECH_REGION=eastus
# AZURE_TTS_ENDPOINT=https://<region>.tts.speech.microsoft.com # optional, overrides region
```
Note: the TTS host (`<region>.tts.speech.microsoft.com`) differs from the STT
endpoint, so the optional override var is `AZURE_TTS_ENDPOINT`, not
`AZURE_SPEECH_ENDPOINT`.
#### API Notes
OpenMontage uses the synchronous REST v1 endpoint with an SSML body — no token
exchange, Blob storage, or job polling:
```text
POST https://{region}.tts.speech.microsoft.com/cognitiveservices/v1
Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY}
Content-Type: application/ssml+xml
X-Microsoft-OutputFormat: audio-48khz-192kbitrate-mono-mp3
```
Voice shortlist aliases: `andrew` (default — warm, confident), `brandon`
(deeper), `ava` (bright female), `guy` (authoritative), `jenny` (friendly). Any
Azure voice short name is accepted verbatim. See the `azure-text-to-speech`
skill for SSML `rate`/`pitch`/`style` guidance.
#### What It Is Best For
- High-quality neural narration on existing Azure credentials
- Calm, confident explainer / founder-register delivery
- Multilingual narration via the *Multilingual* voice family
- Deterministic re-renders (fixed voice + SSML → identical audio)
Not for: fully offline production (use `piper_tts`) or voice cloning (use
`elevenlabs_tts`).
#### Pricing
Azure neural TTS Standard (S0) bills roughly **$16 per 1M characters** (a free
F0 tier includes a limited monthly allowance). A 150-word narration segment
costs about $0.015. OpenMontage estimates cost from character count. See
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
---
### Google — TTS + Imagen + Music + Video (Shared Key)
> **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. `google_imagen` supports both Imagen 4 and Gemini 2.5 Flash Image, including projects without Imagen catalog access. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips.
@@ -1166,6 +1224,7 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
| **Pexels** | `PEXELS_API_KEY` | `pexels_image`, `pexels_video` | Free |
| **Pixabay** | `PIXABAY_API_KEY` | `pixabay_image`, `pixabay_video` | Free |
| **Piper** | — (install only) | `piper_tts` | Free |
| **Azure AI Speech** | `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | `azure_stt`, `azure_tts` | Free tier + paid |
| **Google** | `GOOGLE_API_KEY` (or `GEMINI_API_KEY`) | `google_tts`, `google_imagen`, `google_music`, `gemini_omni_video`, `veo_video` | Free tier (TTS) + paid |
| **ElevenLabs** | `ELEVENLABS_API_KEY` | `elevenlabs_tts`, `music_gen` | Free tier + paid |
| **fish.audio** | `FISH_AUDIO_API_KEY` | `fish_audio_tts` | Free tier (s2.1-pro-free) + paid |
@@ -1193,7 +1252,7 @@ How many providers cover each capability:
|-----------|----------------|-----------------|--------------|
| **Image Generation** | FLUX, Kling Official, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
| **Video Generation** | Grok, Kling Official, Kling via fal.ai, Seedance via Volcengine Ark, Runway, Veo, Gemini Omni, Higgsfield, MiniMax, HeyGen, Tencent Hunyuan | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
| **Text-to-Speech** | ElevenLabs, fish.audio, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier, fish.audio s2.1-pro-free |
| **Text-to-Speech** | Azure AI Speech, ElevenLabs, fish.audio, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier, Azure free tier, fish.audio s2.1-pro-free |
| **Music Generation** | ElevenLabs, Suno, Google Lyria | — | ElevenLabs free tier |
| **Post-Production** | — | FFmpeg (compose, stitch, trim, mix, enhance, grade) | All free |
| **Analysis** | — | WhisperX, Scene Detect, Frame Sampler, CLIP/BLIP-2 | All free |

View File

@@ -89,6 +89,7 @@ Key capability families to look for in the output:
| HyperFrames | `core/hyperframes.md` | HTML/CSS/GSAP composition runtime — kinetic typography, music-to-video, product promos, website capture. Vendored at v0.7.17 (2026-06-27). | `hyperframes` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type/narration), `hyperframes-media` (TTS/BGM/SFX/captions), `hyperframes-animation` (all motion), `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video` (beats-driven), `website-to-video`, `remotion-to-hyperframes` (migration), `gsap-core`, `gsap-timeline` |
| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps — default STT (offline, free) | `speech-to-text` |
| Azure STT | (tool: `azure_stt`) | Optional cloud speech-to-text, word-level timestamps — preferred when `AZURE_SPEECH_KEY` is set | `azure-speech-to-text` |
| Azure TTS | (tool: `azure_tts`) | Optional cloud neural narration (SSML prosody, express-as styles) — same Speech key as `azure_stt` | `azure-text-to-speech` |
| Subtitle Sync | `core/subtitle-sync.md` | Subtitle timing and alignment | `remotion-best-practices` |
| Color Grading | `core/color-grading.md` | FFmpeg color profiles, LUT workflow, accessibility | `ffmpeg` |

View File

@@ -93,6 +93,7 @@ For each script section:
- OpenAI: `instructions` only with `model: "gpt-4o-mini-tts"`; use `response_format` for output format
- Google TTS: `input_type: "ssml"` when using `<break>` tags, plus `speaking_rate` in `0.25..2.0` and `pitch` in `-20..20`
- ElevenLabs: `stability`, `similarity_boost`, `style`, `speed`, and `use_speaker_boost`
- Azure: voice aliases (`andrew`, `brandon`, `ava`, `guy`, `jenny`) or any Azure short name, SSML `rate` (e.g. `"-4%"`), `pitch` (e.g. `"+1st"`), and `style` (e.g. `"narration-professional"`) — see the `azure-text-to-speech` skill
7. Generate using `tts_selector` — it auto-routes to the best available TTS provider based on user preference and availability. Check the registry's `best_for` fields to understand each provider's strengths.
8. Record the applied `voice_performance` metadata on each narration asset
9. Verify the audio file exists and duration matches expected timing (±15%)

View File

@@ -691,6 +691,7 @@ class TestCapabilityMetadata:
assert "tts" in catalog
providers = {item["provider"] for item in catalog["tts"] if item["provider"] != "selector"}
assert providers == {
"azure",
"dashscope",
"doubao",
"elevenlabs",

View File

@@ -0,0 +1,295 @@
"""Focused tests for the Azure AI Speech neural TTS tool.
No live API calls: the network layer is monkeypatched. Covers the tool
contract, registry discovery, status behavior, voice resolution, SSML
construction, and execute() guardrails.
"""
import sys
import xml.etree.ElementTree as ET
from pathlib import Path
import pytest
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
from tools.base_tool import BaseTool, ToolStatus, ToolTier, ToolRuntime
from tools.tool_registry import ToolRegistry
from tools.audio.azure_tts import AzureTTS
FAKE_MP3 = b"\xff\xfb\x90\x00" + b"\x00" * 64
class _FakeResponse:
def __init__(self, content=FAKE_MP3, status_code=200, text=""):
self.content = content
self.status_code = status_code
self.text = text
@pytest.fixture
def azure_env(monkeypatch):
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
monkeypatch.setenv("AZURE_SPEECH_REGION", "eastus")
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
# ---- Contract ----
class TestContract:
def test_inherits_base_tool(self):
assert issubclass(AzureTTS, BaseTool)
def test_identity(self):
t = AzureTTS()
assert t.name == "azure_tts"
assert t.capability == "tts"
assert t.provider == "azure"
assert t.runtime == ToolRuntime.API
assert t.tier == ToolTier.VOICE
assert t.fallback == "piper_tts"
assert "azure-text-to-speech" in t.agent_skills
assert len(t.capabilities) > 0
def test_get_info_valid(self):
info = AzureTTS().get_info()
assert info["name"] == "azure_tts"
assert info["capability"] == "tts"
assert "text" in info["input_schema"]["properties"]
def test_estimate_cost_by_characters(self):
t = AzureTTS()
# Standard tier ≈ $16 per 1M characters.
assert t.estimate_cost({"text": "x" * 1_000_000}) == pytest.approx(16.0)
assert t.estimate_cost({}) == 0.0
# ---- Registry discovery ----
class TestDiscovery:
def test_discoverable(self):
reg = ToolRegistry()
reg.discover("tools")
assert reg.get("azure_tts") is not None
def test_capability_routing(self):
reg = ToolRegistry()
reg.discover("tools")
names = [t.name for t in reg.get_by_capability("tts")]
assert "azure_tts" in names
# ---- Status behavior ----
class TestStatus:
def test_unavailable_without_env(self, monkeypatch):
monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
assert AzureTTS().get_status() == ToolStatus.UNAVAILABLE
def test_available_with_key_and_region(self, azure_env):
assert AzureTTS().get_status() == ToolStatus.AVAILABLE
def test_available_with_key_and_endpoint(self, monkeypatch):
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
monkeypatch.setenv("AZURE_TTS_ENDPOINT", "https://custom.tts.example.com")
assert AzureTTS().get_status() == ToolStatus.AVAILABLE
def test_key_alone_is_not_enough(self, monkeypatch):
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
assert AzureTTS().get_status() == ToolStatus.UNAVAILABLE
# ---- Voice resolution + SSML construction (the risky logic) ----
class TestSSML:
def test_voice_alias_resolution(self):
t = AzureTTS()
assert t._resolve_voice({"voice": "andrew"}) == "en-US-AndrewMultilingualNeural"
assert t._resolve_voice({"voice": "JENNY"}) == "en-US-JennyNeural"
# full short names pass through untouched
assert t._resolve_voice({"voice": "de-DE-KatjaNeural"}) == "de-DE-KatjaNeural"
# default when omitted or blank
assert t._resolve_voice({}) == AzureTTS.DEFAULT_VOICE
assert t._resolve_voice({"voice": " "}) == AzureTTS.DEFAULT_VOICE
def test_ssml_prosody_and_voice(self):
t = AzureTTS()
ssml = t._build_ssml(
{"text": "Hello world", "rate": "-8%", "pitch": "+1st"},
"en-US-AndrewMultilingualNeural",
)
assert '<voice name="en-US-AndrewMultilingualNeural">' in ssml
assert '<prosody rate="-8%" pitch="+1st">Hello world</prosody>' in ssml
assert 'xml:lang="en-US"' in ssml
assert "<mstts:express-as" not in ssml # no style requested
def test_ssml_style_wrapping(self):
t = AzureTTS()
ssml = t._build_ssml(
{"text": "Hi", "style": "narration-professional"}, "en-US-JennyNeural"
)
assert '<mstts:express-as style="narration-professional">' in ssml
assert "</mstts:express-as>" in ssml
def test_ssml_escapes_xml(self):
t = AzureTTS()
ssml = t._build_ssml({"text": "Bread & <butter>"}, "en-US-GuyNeural")
assert "Bread &amp; &lt;butter&gt;" in ssml
assert "<butter>" not in ssml
def test_ssml_custom_locale(self):
t = AzureTTS()
ssml = t._build_ssml({"text": "Hallo", "locale": "de-DE"}, "de-DE-KatjaNeural")
assert 'xml:lang="de-DE"' in ssml
def test_ssml_quotes_in_attributes_remain_well_formed(self):
t = AzureTTS()
ssml = t._build_ssml(
{
"text": 'She said "hello" & waved',
"locale": 'en-US" data-bad="yes',
"rate": '0%" data-bad="yes',
"style": 'calm" data-bad="yes',
},
'en-US-GuyNeural" data-bad="yes',
)
root = ET.fromstring(ssml)
assert all("data-bad" not in element.attrib for element in root.iter())
def test_host_prefers_explicit_endpoint(self, monkeypatch):
monkeypatch.setenv("AZURE_TTS_ENDPOINT", "https://custom.tts.example.com/")
assert AzureTTS()._host() == "https://custom.tts.example.com"
monkeypatch.delenv("AZURE_TTS_ENDPOINT")
monkeypatch.setenv("AZURE_SPEECH_REGION", "westeurope")
assert AzureTTS()._host() == "https://westeurope.tts.speech.microsoft.com"
# ---- execute() guardrails + mocked success ----
class TestExecute:
def test_missing_credentials(self, monkeypatch):
monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
res = AzureTTS().execute({"text": "hello"})
assert not res.success
assert "not configured" in res.error.lower()
def test_success_path_mocked(self, azure_env, tmp_path, monkeypatch):
import requests
captured = {}
def fake_post(url, headers=None, data=None, timeout=None):
captured["url"] = url
captured["headers"] = headers
captured["body"] = data
return _FakeResponse()
monkeypatch.setattr(requests, "post", fake_post)
out = tmp_path / "narration.mp3"
res = AzureTTS().execute(
{"text": "Hello world", "voice": "andrew", "output_path": str(out)}
)
assert res.success
assert res.model == "azure-neural-tts:en-US-AndrewMultilingualNeural"
assert res.data["provider"] == "azure"
assert res.data["voice"] == "en-US-AndrewMultilingualNeural"
assert res.data["text_length"] == len("Hello world")
# cost is rounded to 4 decimals by estimate_cost
assert res.cost_usd == pytest.approx(round(11 * 16.0 / 1_000_000, 4))
# audio bytes written to the requested path
assert out.read_bytes() == FAKE_MP3
assert res.artifacts == [str(out)]
# correct endpoint, auth header, and output format used
assert captured["url"] == "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1"
assert captured["headers"]["Ocp-Apim-Subscription-Key"] == "fake-key"
assert captured["headers"]["X-Microsoft-OutputFormat"] == "audio-48khz-192kbitrate-mono-mp3"
assert b"Hello world" in captured["body"]
def test_selector_adapts_shared_controls_for_azure(self, azure_env, tmp_path, monkeypatch):
import requests
from tools.audio.tts_selector import TTSSelector
captured = {}
def fake_post(url, headers=None, data=None, timeout=None):
captured["body"] = data.decode("utf-8")
return _FakeResponse()
monkeypatch.setattr(requests, "post", fake_post)
monkeypatch.setattr(TTSSelector, "_providers", lambda self: [AzureTTS()])
result = TTSSelector().execute(
{
"text": "Selector narration",
"preferred_provider": "azure",
"voice_id": "jenny",
"speaking_rate": 1.1,
"pitch": 2,
"style": 0.8,
"output_format": "mp3_44100_128",
"output_path": str(tmp_path / "selector.mp3"),
}
)
assert result.success
assert result.data["selected_tool"] == "azure_tts"
assert result.data["voice"] == "en-US-JennyNeural"
assert 'rate="+10%"' in captured["body"]
assert 'pitch="+2st"' in captured["body"]
assert "express-as" not in captured["body"]
def test_wav_output_format(self, azure_env, tmp_path, monkeypatch):
import requests
captured = {}
def fake_post(url, headers=None, data=None, timeout=None):
captured["headers"] = headers
return _FakeResponse(content=b"RIFF....")
monkeypatch.setattr(requests, "post", fake_post)
out = tmp_path / "narration.wav"
res = AzureTTS().execute(
{"text": "Hi", "output_format": "wav", "output_path": str(out)}
)
assert res.success
assert captured["headers"]["X-Microsoft-OutputFormat"] == "riff-48khz-16bit-mono-pcm"
assert out.exists()
def test_http_error_surfaced(self, azure_env, tmp_path, monkeypatch):
import requests
monkeypatch.setattr(
requests, "post",
lambda *a, **k: _FakeResponse(content=b"", status_code=401, text="Unauthorized"),
)
res = AzureTTS().execute(
{"text": "hello", "output_path": str(tmp_path / "x.mp3")}
)
assert not res.success
assert "401" in res.error
def test_request_exception_surfaced(self, azure_env, tmp_path, monkeypatch):
import requests
def boom(*a, **k):
raise requests.exceptions.ConnectionError("no route to host")
monkeypatch.setattr(requests, "post", boom)
res = AzureTTS().execute(
{"text": "hello", "output_path": str(tmp_path / "x.mp3")}
)
assert not res.success
assert "no route to host" in res.error

287
tools/audio/azure_tts.py Normal file
View File

@@ -0,0 +1,287 @@
"""Azure AI Speech text-to-speech provider tool.
Neural TTS served by Azure AI Speech via the REST v1 endpoint. This is an
optional cloud TTS provider; when ``AZURE_SPEECH_KEY`` + ``AZURE_SPEECH_REGION``
are configured the agent may prefer it for high-quality narration, while the
local ``piper_tts`` tool remains the default offline path.
Shares the same Speech resource credentials as the ``azure_stt`` transcription
tool (one key/region unlocks both directions). Uses the synchronous
``/cognitiveservices/v1`` endpoint with an SSML body — no token exchange, Blob
storage, or job polling required.
Docs: https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech
"""
from __future__ import annotations
import os
import time
from pathlib import Path
from typing import Any
from xml.sax.saxutils import escape, quoteattr
from tools.base_tool import (
BaseTool,
Determinism,
ExecutionMode,
ResourceProfile,
RetryPolicy,
ToolResult,
ToolRuntime,
ToolStability,
ToolStatus,
ToolTier,
)
# Output format tokens keyed by container. Chosen for compositing quality.
_MP3_FORMAT = "audio-48khz-192kbitrate-mono-mp3"
_WAV_FORMAT = "riff-48khz-16bit-mono-pcm"
class AzureTTS(BaseTool):
name = "azure_tts"
version = "0.1.0"
tier = ToolTier.VOICE
capability = "tts"
provider = "azure"
stability = ToolStability.BETA
execution_mode = ExecutionMode.SYNC
# Azure neural TTS is effectively deterministic for a fixed voice + SSML.
determinism = Determinism.DETERMINISTIC
runtime = ToolRuntime.API
# Availability is decided by get_status() (env var check), mirroring the
# azure_stt and elevenlabs_tts provider tools — dependencies stays empty.
dependencies = []
install_instructions = (
"Set your Azure AI Speech credentials (same resource as azure_stt):\n"
" export AZURE_SPEECH_KEY=your_speech_resource_key\n"
" export AZURE_SPEECH_REGION=eastus # your Speech resource region\n"
"Create a Speech resource in the Azure portal "
"(https://portal.azure.com) — the key and region are on its "
"'Keys and Endpoint' page. Optionally set AZURE_TTS_ENDPOINT to a full "
"custom TTS host (e.g. https://<region>.tts.speech.microsoft.com)."
)
fallback = "piper_tts"
fallback_tools = ["elevenlabs_tts", "openai_tts", "piper_tts"]
agent_skills = ["azure-text-to-speech", "text-to-speech"]
capabilities = [
"text_to_speech",
"voice_selection",
"ssml_support",
"prosody_control",
]
supports = {
"voice_cloning": False,
"multilingual": True,
"offline": False,
"native_audio": True,
}
best_for = [
"high-quality neural narration on Azure credentials",
"calm, confident explainer / founder-register delivery",
"cloud TTS that shares one key with azure_stt",
]
not_good_for = [
"fully offline production (use piper_tts)",
"voice cloning (use elevenlabs_tts)",
]
# A small curated shortlist of expressive en-US neural voices. Any valid
# Azure voice short name may be passed via `voice`.
RECOMMENDED_VOICES = {
"andrew": "en-US-AndrewMultilingualNeural", # warm, confident, conversational (founder)
"brandon": "en-US-BrandonMultilingualNeural", # deeper, measured
"ava": "en-US-AvaMultilingualNeural", # confident, bright female
"guy": "en-US-GuyNeural", # authoritative
"jenny": "en-US-JennyNeural", # friendly, clear
}
DEFAULT_VOICE = "en-US-AndrewMultilingualNeural"
input_schema = {
"type": "object",
"required": ["text"],
"properties": {
"text": {"type": "string", "description": "Text to convert to speech"},
"voice": {
"type": "string",
"description": (
"Azure voice short name (e.g. 'en-US-AndrewMultilingualNeural') "
"or a shortlist alias: andrew, brandon, ava, guy, jenny. "
"Default: en-US-AndrewMultilingualNeural."
),
},
"rate": {
"type": "string",
"description": (
"SSML prosody rate, e.g. '-8%', '0%', '+5%', or 'slow'/'medium'. "
"Default '0%'."
),
"default": "0%",
},
"pitch": {
"type": "string",
"description": "SSML prosody pitch, e.g. '-2st', '0%', '+1st'. Default '0%'.",
"default": "0%",
},
"style": {
"type": "string",
"description": (
"Optional express-as style for voices that support it "
"(e.g. 'narration-professional', 'calm', 'newscast'). Omit for neutral."
),
},
"locale": {
"type": "string",
"default": "en-US",
"description": "BCP-47 locale for the SSML <speak> element.",
},
"output_path": {"type": "string"},
"output_format": {
"type": "string",
"enum": ["mp3", "wav"],
"default": "mp3",
"description": "Container: 48kHz 192kbit mp3 or 48kHz 16-bit PCM wav.",
},
},
}
output_schema = {
"type": "object",
"properties": {
"provider": {"type": "string"},
"voice": {"type": "string"},
"output": {"type": "string"},
"format": {"type": "string"},
"text_length": {"type": "integer"},
},
}
resource_profile = ResourceProfile(
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=50, network_required=True
)
retry_policy = RetryPolicy(
max_retries=2,
retryable_errors=["ConnectionError", "Timeout", "429", "503"],
)
idempotency_key_fields = ["text", "voice", "rate", "pitch", "style", "output_format"]
side_effects = ["writes audio file to output_path", "sends text to Azure AI Speech"]
user_visible_verification = ["Listen to generated audio for natural speech quality"]
# Azure neural TTS Standard tier bills roughly $16 per 1M characters.
COST_PER_CHAR = 16.0 / 1_000_000
def get_status(self) -> ToolStatus:
if os.environ.get("AZURE_SPEECH_KEY") and (
os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_TTS_ENDPOINT")
):
return ToolStatus.AVAILABLE
return ToolStatus.UNAVAILABLE
def estimate_cost(self, inputs: dict[str, Any]) -> float:
return round(len(inputs.get("text", "")) * self.COST_PER_CHAR, 4)
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
# Well under real-time for typical narration segments.
return 10.0
def _host(self) -> str:
endpoint = os.environ.get("AZURE_TTS_ENDPOINT")
if endpoint:
return endpoint.rstrip("/")
region = os.environ.get("AZURE_SPEECH_REGION", "").strip()
return f"https://{region}.tts.speech.microsoft.com"
def _resolve_voice(self, inputs: dict[str, Any]) -> str:
voice = (inputs.get("voice") or "").strip()
if not voice:
return self.DEFAULT_VOICE
return self.RECOMMENDED_VOICES.get(voice.lower(), voice)
def _build_ssml(self, inputs: dict[str, Any], voice: str) -> str:
locale = str(inputs.get("locale", "en-US"))
rate = str(inputs.get("rate", "0%"))
pitch = str(inputs.get("pitch", "0%"))
style = inputs.get("style")
text = escape(inputs["text"])
inner = f"<prosody rate={quoteattr(rate)} pitch={quoteattr(pitch)}>{text}</prosody>"
if style:
inner = f"<mstts:express-as style={quoteattr(str(style))}>{inner}</mstts:express-as>"
return (
f'<speak version="1.0" '
f'xmlns="http://www.w3.org/2001/10/synthesis" '
f'xmlns:mstts="https://www.w3.org/2001/mstts" '
f"xml:lang={quoteattr(locale)}>"
f"<voice name={quoteattr(voice)}>{inner}</voice></speak>"
)
def execute(self, inputs: dict[str, Any]) -> ToolResult:
api_key = os.environ.get("AZURE_SPEECH_KEY")
if not api_key or not (
os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_TTS_ENDPOINT")
):
return ToolResult(
success=False,
error="Azure Speech is not configured. " + self.install_instructions,
)
start = time.time()
try:
result = self._synthesize(inputs, api_key)
except Exception as exc:
return ToolResult(success=False, error=f"TTS generation failed: {exc}")
result.duration_seconds = round(time.time() - start, 2)
result.cost_usd = self.estimate_cost(inputs)
return result
def _synthesize(self, inputs: dict[str, Any], api_key: str) -> ToolResult:
import requests
voice = self._resolve_voice(inputs)
container = inputs.get("output_format", "mp3")
azure_format = _WAV_FORMAT if container == "wav" else _MP3_FORMAT
ext = "wav" if container == "wav" else "mp3"
ssml = self._build_ssml(inputs, voice)
url = f"{self._host()}/cognitiveservices/v1"
headers = {
"Ocp-Apim-Subscription-Key": api_key,
"Content-Type": "application/ssml+xml",
"X-Microsoft-OutputFormat": azure_format,
"User-Agent": "OpenMontage-azure-tts",
}
try:
response = requests.post(
url, headers=headers, data=ssml.encode("utf-8"), timeout=120
)
except requests.RequestException as exc:
return ToolResult(success=False, error=f"Azure TTS request failed: {exc}")
if response.status_code != 200:
detail = response.text[:500] if response.text else ""
return ToolResult(
success=False,
error=f"Azure TTS returned HTTP {response.status_code}: {detail}",
)
output_path = Path(inputs.get("output_path", f"tts_output.{ext}"))
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_bytes(response.content)
return ToolResult(
success=True,
data={
"provider": self.provider,
"voice": voice,
"text_length": len(inputs["text"]),
"output": str(output_path),
"format": azure_format,
},
artifacts=[str(output_path)],
model=f"azure-neural-tts:{voice}",
)

View File

@@ -210,7 +210,7 @@ class TTSSelector(BaseTool):
if tool is None:
return ToolResult(success=False, error="No TTS provider available.")
result = tool.execute(inputs)
result = tool.execute(self._adapt_inputs(tool, inputs))
if result.success:
result.data.setdefault("selected_tool", tool.name)
result.data["selected_provider"] = tool.provider
@@ -224,6 +224,37 @@ class TTSSelector(BaseTool):
]
return result
@staticmethod
def _adapt_inputs(tool: BaseTool, inputs: dict[str, Any]) -> dict[str, Any]:
"""Translate capability-level controls to provider-native inputs."""
adapted = dict(inputs)
if tool.name != "azure_tts":
return adapted
if inputs.get("voice_id") and not inputs.get("voice"):
adapted["voice"] = inputs["voice_id"]
speed = inputs.get("speaking_rate", inputs.get("speed"))
if speed is not None and "rate" not in inputs:
percent = round((float(speed) - 1.0) * 100)
adapted["rate"] = f"{percent:+d}%" if percent else "0%"
pitch = inputs.get("pitch")
if isinstance(pitch, (int, float)):
adapted["pitch"] = f"{pitch:+g}st" if pitch else "0%"
# The selector's numeric style is ElevenLabs-specific. Azure's style
# is a named express-as value such as "calm" or "newscast".
if not isinstance(inputs.get("style"), str):
adapted.pop("style", None)
output_format = str(inputs.get("output_format", ""))
if output_format.startswith("mp3"):
adapted["output_format"] = "mp3"
elif output_format.startswith(("wav", "riff", "pcm")):
adapted["output_format"] = "wav"
return adapted
def _select_best_tool(
self,
inputs: dict[str, Any],