mirror of
https://github.com/calesthio/OpenMontage.git
synced 2026-08-15 21:16:38 +08:00
Merge pull request #371 from amartya-dev/feat/azure-text-to-speech
feat(tts): add Azure AI Speech as an optional cloud text-to-speech provider
This commit is contained in:
110
.agents/skills/azure-text-to-speech/SKILL.md
Normal file
110
.agents/skills/azure-text-to-speech/SKILL.md
Normal file
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: azure-text-to-speech
|
||||
description: Generate neural narration audio using Azure AI Speech (REST text-to-speech). Use when synthesizing voiceovers or narration in OpenMontage. Optional cloud TTS provider — preferred when AZURE_SPEECH_KEY is configured; the local piper_tts remains the default offline path. Shares one Speech resource with azure_stt.
|
||||
license: MIT
|
||||
compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
|
||||
metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
|
||||
---
|
||||
|
||||
# Azure AI Speech — Text-to-Speech
|
||||
|
||||
Generate narration with **Azure neural TTS** — high-quality multilingual voices,
|
||||
SSML prosody control, and express-as styles, served synchronously by the REST
|
||||
`/cognitiveservices/v1` endpoint (no token exchange, Blob storage, or job
|
||||
polling). In OpenMontage this is exposed through the `azure_tts` tool
|
||||
(`capability=tts`, `provider=azure`). It is an **optional cloud TTS provider** —
|
||||
when `AZURE_SPEECH_KEY` is configured, prefer it for high-quality cloud
|
||||
narration. The local `piper_tts` remains the **default offline path** and the
|
||||
fallback when Azure is unavailable; `elevenlabs_tts` remains the choice for
|
||||
voice cloning.
|
||||
|
||||
> Docs: [REST text to speech](https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech) · [Voice gallery](https://speech.microsoft.com/portal/voicegallery)
|
||||
|
||||
## Setup
|
||||
|
||||
Same Speech resource as `azure_stt` — **one key/region unlocks both directions**
|
||||
(STT and TTS). Create a **Speech** resource in the
|
||||
[Azure portal](https://portal.azure.com); copy the key and region from its
|
||||
**Keys and Endpoint** page.
|
||||
|
||||
```bash
|
||||
export AZURE_SPEECH_KEY=your_speech_resource_key
|
||||
export AZURE_SPEECH_REGION=eastus # your resource's region
|
||||
# export AZURE_TTS_ENDPOINT=https://... # optional: full custom TTS host
|
||||
# (the TTS host is https://<region>.tts.speech.microsoft.com — a different
|
||||
# subdomain than the STT endpoint, hence the separate override var)
|
||||
```
|
||||
|
||||
`azure_tts` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
|
||||
`AZURE_SPEECH_REGION` or `AZURE_TTS_ENDPOINT` are set.
|
||||
|
||||
## Using it in a pipeline
|
||||
|
||||
Route through `tts_selector` as usual (it auto-discovers `azure_tts`), or call
|
||||
the provider tool directly when the user has approved Azure:
|
||||
|
||||
```python
|
||||
from tools.tool_registry import registry
|
||||
registry.discover()
|
||||
tts = registry._tools["azure_tts"]
|
||||
|
||||
result = tts.execute({
|
||||
"text": "Every design decision in this dashboard has a reason.",
|
||||
"voice": "andrew", # alias or full Azure short name
|
||||
"rate": "-4%", # slightly slower for narration
|
||||
# "style": "narration-professional", # for voices that support styles
|
||||
"output_path": "projects/my-video/assets/audio/seg_001.mp3",
|
||||
"output_format": "mp3", # or "wav" (48kHz PCM) for mixing
|
||||
})
|
||||
```
|
||||
|
||||
If `azure_tts` is unavailable (no key) or errors, fall back per its declared
|
||||
chain: `elevenlabs_tts` → `openai_tts` → `piper_tts`.
|
||||
|
||||
## Voice selection
|
||||
|
||||
Curated shortlist (aliases accepted by the `voice` param):
|
||||
|
||||
| Alias | Voice | Character |
|
||||
|-------|-------|-----------|
|
||||
| `andrew` | en-US-AndrewMultilingualNeural | warm, confident, conversational — the default; founder/explainer register |
|
||||
| `brandon` | en-US-BrandonMultilingualNeural | deeper, measured |
|
||||
| `ava` | en-US-AvaMultilingualNeural | confident, bright female |
|
||||
| `guy` | en-US-GuyNeural | authoritative |
|
||||
| `jenny` | en-US-JennyNeural | friendly, clear |
|
||||
|
||||
Any valid Azure voice short name may be passed verbatim (e.g.
|
||||
`de-DE-KatjaNeural`); the *Multilingual* voices handle non-English text well —
|
||||
set `locale` to match the text's language for correct SSML.
|
||||
|
||||
## Parameters that matter
|
||||
|
||||
- **`rate` / `pitch`** — SSML prosody. Narration usually reads best slightly
|
||||
slowed (`"-4%"` to `"-8%"`); leave pitch at `"0%"` unless correcting a voice.
|
||||
- **`style`** — express-as style for voices that support it
|
||||
(`narration-professional`, `calm`, `newscast`). Unsupported styles are
|
||||
silently ignored by Azure, so listen to a sample before batch runs.
|
||||
- **`output_format`** — `mp3` (48kHz/192kbit) for delivery, `wav` (48kHz PCM)
|
||||
when the segment feeds `audio_mixer` for further processing.
|
||||
- Determinism: a fixed voice + SSML re-renders effectively identical audio —
|
||||
safe to regenerate individual segments without re-recording the whole set.
|
||||
|
||||
## Cost
|
||||
|
||||
Azure neural TTS Standard tier bills roughly **$16 per 1M characters** (~$0.016
|
||||
per 1k chars; a 150-word narration segment ≈ $0.015). The tool reports
|
||||
per-call `cost_usd` for the cost tracker. See
|
||||
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
|
||||
|
||||
## Limits & tips
|
||||
|
||||
- One `execute` call = one narration segment. Generate per script section (the
|
||||
asset stage convention) rather than one giant paragraph — smaller segments
|
||||
align cleanly to scene timings and are cheap to regenerate.
|
||||
- The synchronous endpoint caps a request at 10 minutes of audio — far above
|
||||
any segment OpenMontage generates.
|
||||
- Text is XML-escaped automatically; do not pre-escape or wrap in SSML — pass
|
||||
plain text plus the `rate`/`pitch`/`style` params.
|
||||
- Verify quality: listen to the first generated segment before batch-running a
|
||||
full script (voice/style fit is a creative decision — surface it at the
|
||||
proposal stage per the Decision Communication Contract).
|
||||
110
.claude/skills/azure-text-to-speech/SKILL.md
Normal file
110
.claude/skills/azure-text-to-speech/SKILL.md
Normal file
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: azure-text-to-speech
|
||||
description: Generate neural narration audio using Azure AI Speech (REST text-to-speech). Use when synthesizing voiceovers or narration in OpenMontage. Optional cloud TTS provider — preferred when AZURE_SPEECH_KEY is configured; the local piper_tts remains the default offline path. Shares one Speech resource with azure_stt.
|
||||
license: MIT
|
||||
compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
|
||||
metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
|
||||
---
|
||||
|
||||
# Azure AI Speech — Text-to-Speech
|
||||
|
||||
Generate narration with **Azure neural TTS** — high-quality multilingual voices,
|
||||
SSML prosody control, and express-as styles, served synchronously by the REST
|
||||
`/cognitiveservices/v1` endpoint (no token exchange, Blob storage, or job
|
||||
polling). In OpenMontage this is exposed through the `azure_tts` tool
|
||||
(`capability=tts`, `provider=azure`). It is an **optional cloud TTS provider** —
|
||||
when `AZURE_SPEECH_KEY` is configured, prefer it for high-quality cloud
|
||||
narration. The local `piper_tts` remains the **default offline path** and the
|
||||
fallback when Azure is unavailable; `elevenlabs_tts` remains the choice for
|
||||
voice cloning.
|
||||
|
||||
> Docs: [REST text to speech](https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech) · [Voice gallery](https://speech.microsoft.com/portal/voicegallery)
|
||||
|
||||
## Setup
|
||||
|
||||
Same Speech resource as `azure_stt` — **one key/region unlocks both directions**
|
||||
(STT and TTS). Create a **Speech** resource in the
|
||||
[Azure portal](https://portal.azure.com); copy the key and region from its
|
||||
**Keys and Endpoint** page.
|
||||
|
||||
```bash
|
||||
export AZURE_SPEECH_KEY=your_speech_resource_key
|
||||
export AZURE_SPEECH_REGION=eastus # your resource's region
|
||||
# export AZURE_TTS_ENDPOINT=https://... # optional: full custom TTS host
|
||||
# (the TTS host is https://<region>.tts.speech.microsoft.com — a different
|
||||
# subdomain than the STT endpoint, hence the separate override var)
|
||||
```
|
||||
|
||||
`azure_tts` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
|
||||
`AZURE_SPEECH_REGION` or `AZURE_TTS_ENDPOINT` are set.
|
||||
|
||||
## Using it in a pipeline
|
||||
|
||||
Route through `tts_selector` as usual (it auto-discovers `azure_tts`), or call
|
||||
the provider tool directly when the user has approved Azure:
|
||||
|
||||
```python
|
||||
from tools.tool_registry import registry
|
||||
registry.discover()
|
||||
tts = registry._tools["azure_tts"]
|
||||
|
||||
result = tts.execute({
|
||||
"text": "Every design decision in this dashboard has a reason.",
|
||||
"voice": "andrew", # alias or full Azure short name
|
||||
"rate": "-4%", # slightly slower for narration
|
||||
# "style": "narration-professional", # for voices that support styles
|
||||
"output_path": "projects/my-video/assets/audio/seg_001.mp3",
|
||||
"output_format": "mp3", # or "wav" (48kHz PCM) for mixing
|
||||
})
|
||||
```
|
||||
|
||||
If `azure_tts` is unavailable (no key) or errors, fall back per its declared
|
||||
chain: `elevenlabs_tts` → `openai_tts` → `piper_tts`.
|
||||
|
||||
## Voice selection
|
||||
|
||||
Curated shortlist (aliases accepted by the `voice` param):
|
||||
|
||||
| Alias | Voice | Character |
|
||||
|-------|-------|-----------|
|
||||
| `andrew` | en-US-AndrewMultilingualNeural | warm, confident, conversational — the default; founder/explainer register |
|
||||
| `brandon` | en-US-BrandonMultilingualNeural | deeper, measured |
|
||||
| `ava` | en-US-AvaMultilingualNeural | confident, bright female |
|
||||
| `guy` | en-US-GuyNeural | authoritative |
|
||||
| `jenny` | en-US-JennyNeural | friendly, clear |
|
||||
|
||||
Any valid Azure voice short name may be passed verbatim (e.g.
|
||||
`de-DE-KatjaNeural`); the *Multilingual* voices handle non-English text well —
|
||||
set `locale` to match the text's language for correct SSML.
|
||||
|
||||
## Parameters that matter
|
||||
|
||||
- **`rate` / `pitch`** — SSML prosody. Narration usually reads best slightly
|
||||
slowed (`"-4%"` to `"-8%"`); leave pitch at `"0%"` unless correcting a voice.
|
||||
- **`style`** — express-as style for voices that support it
|
||||
(`narration-professional`, `calm`, `newscast`). Unsupported styles are
|
||||
silently ignored by Azure, so listen to a sample before batch runs.
|
||||
- **`output_format`** — `mp3` (48kHz/192kbit) for delivery, `wav` (48kHz PCM)
|
||||
when the segment feeds `audio_mixer` for further processing.
|
||||
- Determinism: a fixed voice + SSML re-renders effectively identical audio —
|
||||
safe to regenerate individual segments without re-recording the whole set.
|
||||
|
||||
## Cost
|
||||
|
||||
Azure neural TTS Standard tier bills roughly **$16 per 1M characters** (~$0.016
|
||||
per 1k chars; a 150-word narration segment ≈ $0.015). The tool reports
|
||||
per-call `cost_usd` for the cost tracker. See
|
||||
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
|
||||
|
||||
## Limits & tips
|
||||
|
||||
- One `execute` call = one narration segment. Generate per script section (the
|
||||
asset stage convention) rather than one giant paragraph — smaller segments
|
||||
align cleanly to scene timings and are cheap to regenerate.
|
||||
- The synchronous endpoint caps a request at 10 minutes of audio — far above
|
||||
any segment OpenMontage generates.
|
||||
- Text is XML-escaped automatically; do not pre-escape or wrap in SSML — pass
|
||||
plain text plus the `rate`/`pitch`/`style` params.
|
||||
- Verify quality: listen to the first generated segment before batch-running a
|
||||
full script (voice/style fit is a creative decision — surface it at the
|
||||
proposal stage per the Decision Communication Contract).
|
||||
11
.env.example
11
.env.example
@@ -112,14 +112,13 @@ UNSPLASH_ACCESS_KEY=
|
||||
# --- Analysis ---
|
||||
# HuggingFace token — enables speaker diarization in transcriber.
|
||||
HF_TOKEN=
|
||||
# Speech-to-text: optional Azure AI Speech (Fast Transcription). When set, the
|
||||
# agent prefers azure_stt for cloud STT; the local faster-whisper transcriber
|
||||
# remains the default offline path.
|
||||
# Azure AI Speech resource key ('Keys and Endpoint' page).
|
||||
# Speech: optional Azure AI Speech. One key/region unlocks both directions —
|
||||
# azure_stt (Fast Transcription cloud STT) and azure_tts (neural cloud TTS).
|
||||
# The local faster-whisper transcriber / piper_tts remain the default offline paths.
|
||||
AZURE_SPEECH_KEY=
|
||||
# Speech resource region, e.g. eastus.
|
||||
AZURE_SPEECH_REGION=
|
||||
# AZURE_SPEECH_ENDPOINT= # Optional: full custom endpoint URL (overrides region)
|
||||
# AZURE_SPEECH_ENDPOINT= # Optional: full custom STT endpoint URL (overrides region)
|
||||
# AZURE_TTS_ENDPOINT= # Optional: full custom TTS host (e.g. https://<region>.tts.speech.microsoft.com)
|
||||
|
||||
# --- Avatar (local installs) ---
|
||||
# WAV2LIP_PATH= # Path to cloned Wav2Lip repo (for lip sync)
|
||||
|
||||
@@ -505,7 +505,7 @@ Three selector tools abstract multi-provider capabilities. **Selectors auto-disc
|
||||
|
||||
| Selector | Routes to | How it discovers |
|
||||
|----------|-----------|-----------------|
|
||||
| `tts_selector` | All tools with `capability="tts"` (ElevenLabs, Google TTS, OpenAI, Piper) | `registry.get_by_capability("tts")` |
|
||||
| `tts_selector` | All tools with `capability="tts"` (ElevenLabs, Google TTS, OpenAI, Piper, Azure) | `registry.get_by_capability("tts")` |
|
||||
| `image_selector` | All tools with `capability="image_generation"` (FLUX, Google Imagen, GPT Image, Recraft, etc.) | `registry.get_by_capability("image_generation")` |
|
||||
| `video_selector` | All tools with `capability="video_generation"` | `registry.get_by_capability("video_generation")` |
|
||||
|
||||
@@ -682,7 +682,7 @@ The `.agents/skills/` directory is large. When you're not coming in through a to
|
||||
| **Character animation** | `character-rigging`, `svg-character-animation`, `pose-library-design`, `canvas-procedural-animation`, `character-animation-qa` |
|
||||
| **Image generation** | `bfl-api`, `flux-best-practices` |
|
||||
| **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` |
|
||||
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` |
|
||||
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `azure-text-to-speech` (optional cloud TTS — tool `azure_tts`, same Speech key as `azure_stt`), `setup-api-key` |
|
||||
| **Speech-to-text** | `speech-to-text` (whisper `transcriber` — default, offline), `azure-speech-to-text` (optional cloud STT — tool `azure_stt`, preferred when `AZURE_SPEECH_KEY` is set) |
|
||||
| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `agents` |
|
||||
| **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) |
|
||||
|
||||
@@ -45,7 +45,7 @@ OpenMontage/
|
||||
│ ├── tool_registry.py # Auto-discovery singleton registry
|
||||
│ ├── cost_tracker.py # Budget governance (estimate → reserve → reconcile)
|
||||
│ ├── analysis/ # Transcription, scene detection, frame sampling, video understanding
|
||||
│ ├── audio/ # TTS (ElevenLabs, OpenAI, Piper), music gen, mixing, enhancement
|
||||
│ ├── audio/ # TTS (ElevenLabs, OpenAI, Piper, Azure, Google), music gen, mixing, enhancement
|
||||
│ ├── avatar/ # Talking head animation, lip sync
|
||||
│ ├── enhancement/ # Upscale, bg removal, face enhance/restore, color grading
|
||||
│ ├── graphics/ # Image gen (FLUX, GPT Image, Recraft, local diffusion), stock, diagrams, code snippets, math animation
|
||||
@@ -150,9 +150,9 @@ Selectors route based on: user preference when explicitly set, then scored ranki
|
||||
|
||||
### Tool Inventory by Category
|
||||
|
||||
**Analysis (4):** transcriber (WhisperX), scene_detect, frame_sampler, video_understand (CLIP/BLIP-2)
|
||||
**Analysis (5):** transcriber (WhisperX), azure_stt, scene_detect, frame_sampler, video_understand (CLIP/BLIP-2)
|
||||
|
||||
**Audio (8):** elevenlabs_tts, google_tts, openai_tts, piper_tts, tts_selector, music_gen, audio_mixer, audio_enhance
|
||||
**Audio (9):** elevenlabs_tts, google_tts, openai_tts, piper_tts, azure_tts, tts_selector, music_gen, audio_mixer, audio_enhance
|
||||
|
||||
**Avatar (2):** talking_head (SadTalker/MuseTalk), lip_sync (Wav2Lip)
|
||||
|
||||
@@ -383,6 +383,7 @@ All config is validated via Pydantic models in `lib/config_model.py`.
|
||||
| Variable | Used By | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `ELEVENLABS_API_KEY` | elevenlabs_tts, music_gen | TTS, music, sound effects |
|
||||
| `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | azure_stt, azure_tts | Azure AI Speech cloud transcription + neural TTS (one resource, both directions) |
|
||||
| `OPENAI_API_KEY` | openai_tts, openai_image | TTS fallback, GPT Image 2 |
|
||||
| `XAI_API_KEY` | grok_image, grok_video | Grok image editing/generation, Grok video generation |
|
||||
| `FAL_KEY` | flux_image, kling_video, veo_video, minimax_video, recraft_image | fal.ai hosted models (FLUX, Veo, Kling, MiniMax, Recraft) |
|
||||
|
||||
@@ -46,8 +46,8 @@ DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin nar
|
||||
DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type
|
||||
DASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps)
|
||||
|
||||
# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default)
|
||||
AZURE_SPEECH_KEY= # Azure AI Speech — Fast Transcription (word-level timestamps)
|
||||
# AZURE AI SPEECH (optional cloud STT + TTS; one key unlocks both directions)
|
||||
AZURE_SPEECH_KEY= # Azure AI Speech — azure_stt (Fast Transcription) + azure_tts (neural narration)
|
||||
AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
|
||||
|
||||
# MULTI-MODEL GATEWAY (one key, 6+ tools)
|
||||
@@ -589,6 +589,64 @@ allowance). OpenMontage estimates cost from the transcribed audio duration. See
|
||||
|
||||
---
|
||||
|
||||
### Azure AI Speech — Text-to-Speech
|
||||
|
||||
> **Cloud neural narration.** Azure neural TTS delivers high-quality multilingual voices with SSML prosody control and express-as styles — same Speech resource as `azure_stt`, so one key/region unlocks both directions. Optional: the local `piper_tts` remains the default offline TTS path. When `AZURE_SPEECH_KEY` is set, the agent may prefer `azure_tts` for cloud narration.
|
||||
|
||||
**Tools unlocked:** `azure_tts`
|
||||
**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_TTS_ENDPOINT`)
|
||||
|
||||
#### Setup
|
||||
|
||||
Identical to the STT setup above — the same Speech resource key and region work
|
||||
for both. If you already configured `azure_stt`, `azure_tts` is available now.
|
||||
|
||||
```bash
|
||||
AZURE_SPEECH_KEY=your-speech-resource-key
|
||||
AZURE_SPEECH_REGION=eastus
|
||||
# AZURE_TTS_ENDPOINT=https://<region>.tts.speech.microsoft.com # optional, overrides region
|
||||
```
|
||||
|
||||
Note: the TTS host (`<region>.tts.speech.microsoft.com`) differs from the STT
|
||||
endpoint, so the optional override var is `AZURE_TTS_ENDPOINT`, not
|
||||
`AZURE_SPEECH_ENDPOINT`.
|
||||
|
||||
#### API Notes
|
||||
|
||||
OpenMontage uses the synchronous REST v1 endpoint with an SSML body — no token
|
||||
exchange, Blob storage, or job polling:
|
||||
|
||||
```text
|
||||
POST https://{region}.tts.speech.microsoft.com/cognitiveservices/v1
|
||||
Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY}
|
||||
Content-Type: application/ssml+xml
|
||||
X-Microsoft-OutputFormat: audio-48khz-192kbitrate-mono-mp3
|
||||
```
|
||||
|
||||
Voice shortlist aliases: `andrew` (default — warm, confident), `brandon`
|
||||
(deeper), `ava` (bright female), `guy` (authoritative), `jenny` (friendly). Any
|
||||
Azure voice short name is accepted verbatim. See the `azure-text-to-speech`
|
||||
skill for SSML `rate`/`pitch`/`style` guidance.
|
||||
|
||||
#### What It Is Best For
|
||||
|
||||
- High-quality neural narration on existing Azure credentials
|
||||
- Calm, confident explainer / founder-register delivery
|
||||
- Multilingual narration via the *Multilingual* voice family
|
||||
- Deterministic re-renders (fixed voice + SSML → identical audio)
|
||||
|
||||
Not for: fully offline production (use `piper_tts`) or voice cloning (use
|
||||
`elevenlabs_tts`).
|
||||
|
||||
#### Pricing
|
||||
|
||||
Azure neural TTS Standard (S0) bills roughly **$16 per 1M characters** (a free
|
||||
F0 tier includes a limited monthly allowance). A 150-word narration segment
|
||||
costs about $0.015. OpenMontage estimates cost from character count. See
|
||||
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
|
||||
|
||||
---
|
||||
|
||||
### Google — TTS + Imagen + Music + Video (Shared Key)
|
||||
|
||||
> **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. `google_imagen` supports both Imagen 4 and Gemini 2.5 Flash Image, including projects without Imagen catalog access. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips.
|
||||
@@ -1130,6 +1188,7 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
|
||||
| **Pexels** | `PEXELS_API_KEY` | `pexels_image`, `pexels_video` | Free |
|
||||
| **Pixabay** | `PIXABAY_API_KEY` | `pixabay_image`, `pixabay_video` | Free |
|
||||
| **Piper** | — (install only) | `piper_tts` | Free |
|
||||
| **Azure AI Speech** | `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | `azure_stt`, `azure_tts` | Free tier + paid |
|
||||
| **Google** | `GOOGLE_API_KEY` (or `GEMINI_API_KEY`) | `google_tts`, `google_imagen`, `google_music`, `gemini_omni_video`, `veo_video` | Free tier (TTS) + paid |
|
||||
| **ElevenLabs** | `ELEVENLABS_API_KEY` | `elevenlabs_tts`, `music_gen` | Free tier + paid |
|
||||
| **fal.ai** | `FAL_KEY` | `flux_image`, `recraft_image`, `kling_video`, `veo_video`, `minimax_video` | Pay-as-you-go |
|
||||
@@ -1156,7 +1215,7 @@ How many providers cover each capability:
|
||||
|-----------|----------------|-----------------|--------------|
|
||||
| **Image Generation** | FLUX, Kling Official, Grok, Google Imagen, GPT Image 2, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
|
||||
| **Video Generation** | Grok, Kling Official, Kling via fal.ai, Seedance via Volcengine Ark, Runway, Veo, Gemini Omni, Higgsfield, MiniMax, HeyGen, Tencent Hunyuan | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
|
||||
| **Text-to-Speech** | ElevenLabs, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier |
|
||||
| **Text-to-Speech** | Azure AI Speech, ElevenLabs, Google TTS, Kling Official, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier, Azure free tier |
|
||||
| **Music Generation** | ElevenLabs, Suno, Google Lyria | — | ElevenLabs free tier |
|
||||
| **Post-Production** | — | FFmpeg (compose, stitch, trim, mix, enhance, grade) | All free |
|
||||
| **Analysis** | — | WhisperX, Scene Detect, Frame Sampler, CLIP/BLIP-2 | All free |
|
||||
|
||||
@@ -89,6 +89,7 @@ Key capability families to look for in the output:
|
||||
| HyperFrames | `core/hyperframes.md` | HTML/CSS/GSAP composition runtime — kinetic typography, music-to-video, product promos, website capture. Vendored at v0.7.17 (2026-06-27). | `hyperframes` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type/narration), `hyperframes-media` (TTS/BGM/SFX/captions), `hyperframes-animation` (all motion), `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video` (beats-driven), `website-to-video`, `remotion-to-hyperframes` (migration), `gsap-core`, `gsap-timeline` |
|
||||
| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps — default STT (offline, free) | `speech-to-text` |
|
||||
| Azure STT | (tool: `azure_stt`) | Optional cloud speech-to-text, word-level timestamps — preferred when `AZURE_SPEECH_KEY` is set | `azure-speech-to-text` |
|
||||
| Azure TTS | (tool: `azure_tts`) | Optional cloud neural narration (SSML prosody, express-as styles) — same Speech key as `azure_stt` | `azure-text-to-speech` |
|
||||
| Subtitle Sync | `core/subtitle-sync.md` | Subtitle timing and alignment | `remotion-best-practices` |
|
||||
| Color Grading | `core/color-grading.md` | FFmpeg color profiles, LUT workflow, accessibility | `ffmpeg` |
|
||||
|
||||
|
||||
@@ -93,6 +93,7 @@ For each script section:
|
||||
- OpenAI: `instructions` only with `model: "gpt-4o-mini-tts"`; use `response_format` for output format
|
||||
- Google TTS: `input_type: "ssml"` when using `<break>` tags, plus `speaking_rate` in `0.25..2.0` and `pitch` in `-20..20`
|
||||
- ElevenLabs: `stability`, `similarity_boost`, `style`, `speed`, and `use_speaker_boost`
|
||||
- Azure: voice aliases (`andrew`, `brandon`, `ava`, `guy`, `jenny`) or any Azure short name, SSML `rate` (e.g. `"-4%"`), `pitch` (e.g. `"+1st"`), and `style` (e.g. `"narration-professional"`) — see the `azure-text-to-speech` skill
|
||||
7. Generate using `tts_selector` — it auto-routes to the best available TTS provider based on user preference and availability. Check the registry's `best_for` fields to understand each provider's strengths.
|
||||
8. Record the applied `voice_performance` metadata on each narration asset
|
||||
9. Verify the audio file exists and duration matches expected timing (±15%)
|
||||
|
||||
@@ -691,6 +691,7 @@ class TestCapabilityMetadata:
|
||||
assert "tts" in catalog
|
||||
providers = {item["provider"] for item in catalog["tts"] if item["provider"] != "selector"}
|
||||
assert providers == {
|
||||
"azure",
|
||||
"dashscope",
|
||||
"doubao",
|
||||
"elevenlabs",
|
||||
|
||||
295
tests/tools/test_azure_tts.py
Normal file
295
tests/tools/test_azure_tts.py
Normal file
@@ -0,0 +1,295 @@
|
||||
"""Focused tests for the Azure AI Speech neural TTS tool.
|
||||
|
||||
No live API calls: the network layer is monkeypatched. Covers the tool
|
||||
contract, registry discovery, status behavior, voice resolution, SSML
|
||||
construction, and execute() guardrails.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools.base_tool import BaseTool, ToolStatus, ToolTier, ToolRuntime
|
||||
from tools.tool_registry import ToolRegistry
|
||||
from tools.audio.azure_tts import AzureTTS
|
||||
|
||||
|
||||
FAKE_MP3 = b"\xff\xfb\x90\x00" + b"\x00" * 64
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
def __init__(self, content=FAKE_MP3, status_code=200, text=""):
|
||||
self.content = content
|
||||
self.status_code = status_code
|
||||
self.text = text
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def azure_env(monkeypatch):
|
||||
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
|
||||
monkeypatch.setenv("AZURE_SPEECH_REGION", "eastus")
|
||||
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
|
||||
|
||||
|
||||
# ---- Contract ----
|
||||
|
||||
class TestContract:
|
||||
def test_inherits_base_tool(self):
|
||||
assert issubclass(AzureTTS, BaseTool)
|
||||
|
||||
def test_identity(self):
|
||||
t = AzureTTS()
|
||||
assert t.name == "azure_tts"
|
||||
assert t.capability == "tts"
|
||||
assert t.provider == "azure"
|
||||
assert t.runtime == ToolRuntime.API
|
||||
assert t.tier == ToolTier.VOICE
|
||||
assert t.fallback == "piper_tts"
|
||||
assert "azure-text-to-speech" in t.agent_skills
|
||||
assert len(t.capabilities) > 0
|
||||
|
||||
def test_get_info_valid(self):
|
||||
info = AzureTTS().get_info()
|
||||
assert info["name"] == "azure_tts"
|
||||
assert info["capability"] == "tts"
|
||||
assert "text" in info["input_schema"]["properties"]
|
||||
|
||||
def test_estimate_cost_by_characters(self):
|
||||
t = AzureTTS()
|
||||
# Standard tier ≈ $16 per 1M characters.
|
||||
assert t.estimate_cost({"text": "x" * 1_000_000}) == pytest.approx(16.0)
|
||||
assert t.estimate_cost({}) == 0.0
|
||||
|
||||
|
||||
# ---- Registry discovery ----
|
||||
|
||||
class TestDiscovery:
|
||||
def test_discoverable(self):
|
||||
reg = ToolRegistry()
|
||||
reg.discover("tools")
|
||||
assert reg.get("azure_tts") is not None
|
||||
|
||||
def test_capability_routing(self):
|
||||
reg = ToolRegistry()
|
||||
reg.discover("tools")
|
||||
names = [t.name for t in reg.get_by_capability("tts")]
|
||||
assert "azure_tts" in names
|
||||
|
||||
|
||||
# ---- Status behavior ----
|
||||
|
||||
class TestStatus:
|
||||
def test_unavailable_without_env(self, monkeypatch):
|
||||
monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
|
||||
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
|
||||
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
|
||||
assert AzureTTS().get_status() == ToolStatus.UNAVAILABLE
|
||||
|
||||
def test_available_with_key_and_region(self, azure_env):
|
||||
assert AzureTTS().get_status() == ToolStatus.AVAILABLE
|
||||
|
||||
def test_available_with_key_and_endpoint(self, monkeypatch):
|
||||
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
|
||||
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
|
||||
monkeypatch.setenv("AZURE_TTS_ENDPOINT", "https://custom.tts.example.com")
|
||||
assert AzureTTS().get_status() == ToolStatus.AVAILABLE
|
||||
|
||||
def test_key_alone_is_not_enough(self, monkeypatch):
|
||||
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
|
||||
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
|
||||
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
|
||||
assert AzureTTS().get_status() == ToolStatus.UNAVAILABLE
|
||||
|
||||
|
||||
# ---- Voice resolution + SSML construction (the risky logic) ----
|
||||
|
||||
class TestSSML:
|
||||
def test_voice_alias_resolution(self):
|
||||
t = AzureTTS()
|
||||
assert t._resolve_voice({"voice": "andrew"}) == "en-US-AndrewMultilingualNeural"
|
||||
assert t._resolve_voice({"voice": "JENNY"}) == "en-US-JennyNeural"
|
||||
# full short names pass through untouched
|
||||
assert t._resolve_voice({"voice": "de-DE-KatjaNeural"}) == "de-DE-KatjaNeural"
|
||||
# default when omitted or blank
|
||||
assert t._resolve_voice({}) == AzureTTS.DEFAULT_VOICE
|
||||
assert t._resolve_voice({"voice": " "}) == AzureTTS.DEFAULT_VOICE
|
||||
|
||||
def test_ssml_prosody_and_voice(self):
|
||||
t = AzureTTS()
|
||||
ssml = t._build_ssml(
|
||||
{"text": "Hello world", "rate": "-8%", "pitch": "+1st"},
|
||||
"en-US-AndrewMultilingualNeural",
|
||||
)
|
||||
assert '<voice name="en-US-AndrewMultilingualNeural">' in ssml
|
||||
assert '<prosody rate="-8%" pitch="+1st">Hello world</prosody>' in ssml
|
||||
assert 'xml:lang="en-US"' in ssml
|
||||
assert "<mstts:express-as" not in ssml # no style requested
|
||||
|
||||
def test_ssml_style_wrapping(self):
|
||||
t = AzureTTS()
|
||||
ssml = t._build_ssml(
|
||||
{"text": "Hi", "style": "narration-professional"}, "en-US-JennyNeural"
|
||||
)
|
||||
assert '<mstts:express-as style="narration-professional">' in ssml
|
||||
assert "</mstts:express-as>" in ssml
|
||||
|
||||
def test_ssml_escapes_xml(self):
|
||||
t = AzureTTS()
|
||||
ssml = t._build_ssml({"text": "Bread & <butter>"}, "en-US-GuyNeural")
|
||||
assert "Bread & <butter>" in ssml
|
||||
assert "<butter>" not in ssml
|
||||
|
||||
def test_ssml_custom_locale(self):
|
||||
t = AzureTTS()
|
||||
ssml = t._build_ssml({"text": "Hallo", "locale": "de-DE"}, "de-DE-KatjaNeural")
|
||||
assert 'xml:lang="de-DE"' in ssml
|
||||
|
||||
def test_ssml_quotes_in_attributes_remain_well_formed(self):
|
||||
t = AzureTTS()
|
||||
ssml = t._build_ssml(
|
||||
{
|
||||
"text": 'She said "hello" & waved',
|
||||
"locale": 'en-US" data-bad="yes',
|
||||
"rate": '0%" data-bad="yes',
|
||||
"style": 'calm" data-bad="yes',
|
||||
},
|
||||
'en-US-GuyNeural" data-bad="yes',
|
||||
)
|
||||
root = ET.fromstring(ssml)
|
||||
assert all("data-bad" not in element.attrib for element in root.iter())
|
||||
|
||||
def test_host_prefers_explicit_endpoint(self, monkeypatch):
|
||||
monkeypatch.setenv("AZURE_TTS_ENDPOINT", "https://custom.tts.example.com/")
|
||||
assert AzureTTS()._host() == "https://custom.tts.example.com"
|
||||
monkeypatch.delenv("AZURE_TTS_ENDPOINT")
|
||||
monkeypatch.setenv("AZURE_SPEECH_REGION", "westeurope")
|
||||
assert AzureTTS()._host() == "https://westeurope.tts.speech.microsoft.com"
|
||||
|
||||
|
||||
# ---- execute() guardrails + mocked success ----
|
||||
|
||||
class TestExecute:
|
||||
def test_missing_credentials(self, monkeypatch):
|
||||
monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
|
||||
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
|
||||
monkeypatch.delenv("AZURE_TTS_ENDPOINT", raising=False)
|
||||
res = AzureTTS().execute({"text": "hello"})
|
||||
assert not res.success
|
||||
assert "not configured" in res.error.lower()
|
||||
|
||||
def test_success_path_mocked(self, azure_env, tmp_path, monkeypatch):
|
||||
import requests
|
||||
|
||||
captured = {}
|
||||
|
||||
def fake_post(url, headers=None, data=None, timeout=None):
|
||||
captured["url"] = url
|
||||
captured["headers"] = headers
|
||||
captured["body"] = data
|
||||
return _FakeResponse()
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
|
||||
out = tmp_path / "narration.mp3"
|
||||
res = AzureTTS().execute(
|
||||
{"text": "Hello world", "voice": "andrew", "output_path": str(out)}
|
||||
)
|
||||
|
||||
assert res.success
|
||||
assert res.model == "azure-neural-tts:en-US-AndrewMultilingualNeural"
|
||||
assert res.data["provider"] == "azure"
|
||||
assert res.data["voice"] == "en-US-AndrewMultilingualNeural"
|
||||
assert res.data["text_length"] == len("Hello world")
|
||||
# cost is rounded to 4 decimals by estimate_cost
|
||||
assert res.cost_usd == pytest.approx(round(11 * 16.0 / 1_000_000, 4))
|
||||
# audio bytes written to the requested path
|
||||
assert out.read_bytes() == FAKE_MP3
|
||||
assert res.artifacts == [str(out)]
|
||||
# correct endpoint, auth header, and output format used
|
||||
assert captured["url"] == "https://eastus.tts.speech.microsoft.com/cognitiveservices/v1"
|
||||
assert captured["headers"]["Ocp-Apim-Subscription-Key"] == "fake-key"
|
||||
assert captured["headers"]["X-Microsoft-OutputFormat"] == "audio-48khz-192kbitrate-mono-mp3"
|
||||
assert b"Hello world" in captured["body"]
|
||||
|
||||
def test_selector_adapts_shared_controls_for_azure(self, azure_env, tmp_path, monkeypatch):
|
||||
import requests
|
||||
from tools.audio.tts_selector import TTSSelector
|
||||
|
||||
captured = {}
|
||||
|
||||
def fake_post(url, headers=None, data=None, timeout=None):
|
||||
captured["body"] = data.decode("utf-8")
|
||||
return _FakeResponse()
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
monkeypatch.setattr(TTSSelector, "_providers", lambda self: [AzureTTS()])
|
||||
|
||||
result = TTSSelector().execute(
|
||||
{
|
||||
"text": "Selector narration",
|
||||
"preferred_provider": "azure",
|
||||
"voice_id": "jenny",
|
||||
"speaking_rate": 1.1,
|
||||
"pitch": 2,
|
||||
"style": 0.8,
|
||||
"output_format": "mp3_44100_128",
|
||||
"output_path": str(tmp_path / "selector.mp3"),
|
||||
}
|
||||
)
|
||||
|
||||
assert result.success
|
||||
assert result.data["selected_tool"] == "azure_tts"
|
||||
assert result.data["voice"] == "en-US-JennyNeural"
|
||||
assert 'rate="+10%"' in captured["body"]
|
||||
assert 'pitch="+2st"' in captured["body"]
|
||||
assert "express-as" not in captured["body"]
|
||||
|
||||
def test_wav_output_format(self, azure_env, tmp_path, monkeypatch):
|
||||
import requests
|
||||
|
||||
captured = {}
|
||||
|
||||
def fake_post(url, headers=None, data=None, timeout=None):
|
||||
captured["headers"] = headers
|
||||
return _FakeResponse(content=b"RIFF....")
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
|
||||
out = tmp_path / "narration.wav"
|
||||
res = AzureTTS().execute(
|
||||
{"text": "Hi", "output_format": "wav", "output_path": str(out)}
|
||||
)
|
||||
assert res.success
|
||||
assert captured["headers"]["X-Microsoft-OutputFormat"] == "riff-48khz-16bit-mono-pcm"
|
||||
assert out.exists()
|
||||
|
||||
def test_http_error_surfaced(self, azure_env, tmp_path, monkeypatch):
|
||||
import requests
|
||||
|
||||
monkeypatch.setattr(
|
||||
requests, "post",
|
||||
lambda *a, **k: _FakeResponse(content=b"", status_code=401, text="Unauthorized"),
|
||||
)
|
||||
res = AzureTTS().execute(
|
||||
{"text": "hello", "output_path": str(tmp_path / "x.mp3")}
|
||||
)
|
||||
assert not res.success
|
||||
assert "401" in res.error
|
||||
|
||||
def test_request_exception_surfaced(self, azure_env, tmp_path, monkeypatch):
|
||||
import requests
|
||||
|
||||
def boom(*a, **k):
|
||||
raise requests.exceptions.ConnectionError("no route to host")
|
||||
|
||||
monkeypatch.setattr(requests, "post", boom)
|
||||
res = AzureTTS().execute(
|
||||
{"text": "hello", "output_path": str(tmp_path / "x.mp3")}
|
||||
)
|
||||
assert not res.success
|
||||
assert "no route to host" in res.error
|
||||
287
tools/audio/azure_tts.py
Normal file
287
tools/audio/azure_tts.py
Normal file
@@ -0,0 +1,287 @@
|
||||
"""Azure AI Speech text-to-speech provider tool.
|
||||
|
||||
Neural TTS served by Azure AI Speech via the REST v1 endpoint. This is an
|
||||
optional cloud TTS provider; when ``AZURE_SPEECH_KEY`` + ``AZURE_SPEECH_REGION``
|
||||
are configured the agent may prefer it for high-quality narration, while the
|
||||
local ``piper_tts`` tool remains the default offline path.
|
||||
|
||||
Shares the same Speech resource credentials as the ``azure_stt`` transcription
|
||||
tool (one key/region unlocks both directions). Uses the synchronous
|
||||
``/cognitiveservices/v1`` endpoint with an SSML body — no token exchange, Blob
|
||||
storage, or job polling required.
|
||||
|
||||
Docs: https://learn.microsoft.com/azure/ai-services/speech-service/rest-text-to-speech
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from xml.sax.saxutils import escape, quoteattr
|
||||
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolStatus,
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
# Output format tokens keyed by container. Chosen for compositing quality.
|
||||
_MP3_FORMAT = "audio-48khz-192kbitrate-mono-mp3"
|
||||
_WAV_FORMAT = "riff-48khz-16bit-mono-pcm"
|
||||
|
||||
|
||||
class AzureTTS(BaseTool):
|
||||
name = "azure_tts"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.VOICE
|
||||
capability = "tts"
|
||||
provider = "azure"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
# Azure neural TTS is effectively deterministic for a fixed voice + SSML.
|
||||
determinism = Determinism.DETERMINISTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
# Availability is decided by get_status() (env var check), mirroring the
|
||||
# azure_stt and elevenlabs_tts provider tools — dependencies stays empty.
|
||||
dependencies = []
|
||||
install_instructions = (
|
||||
"Set your Azure AI Speech credentials (same resource as azure_stt):\n"
|
||||
" export AZURE_SPEECH_KEY=your_speech_resource_key\n"
|
||||
" export AZURE_SPEECH_REGION=eastus # your Speech resource region\n"
|
||||
"Create a Speech resource in the Azure portal "
|
||||
"(https://portal.azure.com) — the key and region are on its "
|
||||
"'Keys and Endpoint' page. Optionally set AZURE_TTS_ENDPOINT to a full "
|
||||
"custom TTS host (e.g. https://<region>.tts.speech.microsoft.com)."
|
||||
)
|
||||
fallback = "piper_tts"
|
||||
fallback_tools = ["elevenlabs_tts", "openai_tts", "piper_tts"]
|
||||
agent_skills = ["azure-text-to-speech", "text-to-speech"]
|
||||
|
||||
capabilities = [
|
||||
"text_to_speech",
|
||||
"voice_selection",
|
||||
"ssml_support",
|
||||
"prosody_control",
|
||||
]
|
||||
supports = {
|
||||
"voice_cloning": False,
|
||||
"multilingual": True,
|
||||
"offline": False,
|
||||
"native_audio": True,
|
||||
}
|
||||
best_for = [
|
||||
"high-quality neural narration on Azure credentials",
|
||||
"calm, confident explainer / founder-register delivery",
|
||||
"cloud TTS that shares one key with azure_stt",
|
||||
]
|
||||
not_good_for = [
|
||||
"fully offline production (use piper_tts)",
|
||||
"voice cloning (use elevenlabs_tts)",
|
||||
]
|
||||
|
||||
# A small curated shortlist of expressive en-US neural voices. Any valid
|
||||
# Azure voice short name may be passed via `voice`.
|
||||
RECOMMENDED_VOICES = {
|
||||
"andrew": "en-US-AndrewMultilingualNeural", # warm, confident, conversational (founder)
|
||||
"brandon": "en-US-BrandonMultilingualNeural", # deeper, measured
|
||||
"ava": "en-US-AvaMultilingualNeural", # confident, bright female
|
||||
"guy": "en-US-GuyNeural", # authoritative
|
||||
"jenny": "en-US-JennyNeural", # friendly, clear
|
||||
}
|
||||
DEFAULT_VOICE = "en-US-AndrewMultilingualNeural"
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["text"],
|
||||
"properties": {
|
||||
"text": {"type": "string", "description": "Text to convert to speech"},
|
||||
"voice": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Azure voice short name (e.g. 'en-US-AndrewMultilingualNeural') "
|
||||
"or a shortlist alias: andrew, brandon, ava, guy, jenny. "
|
||||
"Default: en-US-AndrewMultilingualNeural."
|
||||
),
|
||||
},
|
||||
"rate": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"SSML prosody rate, e.g. '-8%', '0%', '+5%', or 'slow'/'medium'. "
|
||||
"Default '0%'."
|
||||
),
|
||||
"default": "0%",
|
||||
},
|
||||
"pitch": {
|
||||
"type": "string",
|
||||
"description": "SSML prosody pitch, e.g. '-2st', '0%', '+1st'. Default '0%'.",
|
||||
"default": "0%",
|
||||
},
|
||||
"style": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Optional express-as style for voices that support it "
|
||||
"(e.g. 'narration-professional', 'calm', 'newscast'). Omit for neutral."
|
||||
),
|
||||
},
|
||||
"locale": {
|
||||
"type": "string",
|
||||
"default": "en-US",
|
||||
"description": "BCP-47 locale for the SSML <speak> element.",
|
||||
},
|
||||
"output_path": {"type": "string"},
|
||||
"output_format": {
|
||||
"type": "string",
|
||||
"enum": ["mp3", "wav"],
|
||||
"default": "mp3",
|
||||
"description": "Container: 48kHz 192kbit mp3 or 48kHz 16-bit PCM wav.",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
output_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"provider": {"type": "string"},
|
||||
"voice": {"type": "string"},
|
||||
"output": {"type": "string"},
|
||||
"format": {"type": "string"},
|
||||
"text_length": {"type": "integer"},
|
||||
},
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1, ram_mb=256, vram_mb=0, disk_mb=50, network_required=True
|
||||
)
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2,
|
||||
retryable_errors=["ConnectionError", "Timeout", "429", "503"],
|
||||
)
|
||||
idempotency_key_fields = ["text", "voice", "rate", "pitch", "style", "output_format"]
|
||||
side_effects = ["writes audio file to output_path", "sends text to Azure AI Speech"]
|
||||
user_visible_verification = ["Listen to generated audio for natural speech quality"]
|
||||
|
||||
# Azure neural TTS Standard tier bills roughly $16 per 1M characters.
|
||||
COST_PER_CHAR = 16.0 / 1_000_000
|
||||
|
||||
def get_status(self) -> ToolStatus:
|
||||
if os.environ.get("AZURE_SPEECH_KEY") and (
|
||||
os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_TTS_ENDPOINT")
|
||||
):
|
||||
return ToolStatus.AVAILABLE
|
||||
return ToolStatus.UNAVAILABLE
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
return round(len(inputs.get("text", "")) * self.COST_PER_CHAR, 4)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
# Well under real-time for typical narration segments.
|
||||
return 10.0
|
||||
|
||||
def _host(self) -> str:
|
||||
endpoint = os.environ.get("AZURE_TTS_ENDPOINT")
|
||||
if endpoint:
|
||||
return endpoint.rstrip("/")
|
||||
region = os.environ.get("AZURE_SPEECH_REGION", "").strip()
|
||||
return f"https://{region}.tts.speech.microsoft.com"
|
||||
|
||||
def _resolve_voice(self, inputs: dict[str, Any]) -> str:
|
||||
voice = (inputs.get("voice") or "").strip()
|
||||
if not voice:
|
||||
return self.DEFAULT_VOICE
|
||||
return self.RECOMMENDED_VOICES.get(voice.lower(), voice)
|
||||
|
||||
def _build_ssml(self, inputs: dict[str, Any], voice: str) -> str:
|
||||
locale = str(inputs.get("locale", "en-US"))
|
||||
rate = str(inputs.get("rate", "0%"))
|
||||
pitch = str(inputs.get("pitch", "0%"))
|
||||
style = inputs.get("style")
|
||||
text = escape(inputs["text"])
|
||||
|
||||
inner = f"<prosody rate={quoteattr(rate)} pitch={quoteattr(pitch)}>{text}</prosody>"
|
||||
if style:
|
||||
inner = f"<mstts:express-as style={quoteattr(str(style))}>{inner}</mstts:express-as>"
|
||||
return (
|
||||
f'<speak version="1.0" '
|
||||
f'xmlns="http://www.w3.org/2001/10/synthesis" '
|
||||
f'xmlns:mstts="https://www.w3.org/2001/mstts" '
|
||||
f"xml:lang={quoteattr(locale)}>"
|
||||
f"<voice name={quoteattr(voice)}>{inner}</voice></speak>"
|
||||
)
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
api_key = os.environ.get("AZURE_SPEECH_KEY")
|
||||
if not api_key or not (
|
||||
os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_TTS_ENDPOINT")
|
||||
):
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="Azure Speech is not configured. " + self.install_instructions,
|
||||
)
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
result = self._synthesize(inputs, api_key)
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, error=f"TTS generation failed: {exc}")
|
||||
|
||||
result.duration_seconds = round(time.time() - start, 2)
|
||||
result.cost_usd = self.estimate_cost(inputs)
|
||||
return result
|
||||
|
||||
def _synthesize(self, inputs: dict[str, Any], api_key: str) -> ToolResult:
|
||||
import requests
|
||||
|
||||
voice = self._resolve_voice(inputs)
|
||||
container = inputs.get("output_format", "mp3")
|
||||
azure_format = _WAV_FORMAT if container == "wav" else _MP3_FORMAT
|
||||
ext = "wav" if container == "wav" else "mp3"
|
||||
|
||||
ssml = self._build_ssml(inputs, voice)
|
||||
url = f"{self._host()}/cognitiveservices/v1"
|
||||
headers = {
|
||||
"Ocp-Apim-Subscription-Key": api_key,
|
||||
"Content-Type": "application/ssml+xml",
|
||||
"X-Microsoft-OutputFormat": azure_format,
|
||||
"User-Agent": "OpenMontage-azure-tts",
|
||||
}
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
url, headers=headers, data=ssml.encode("utf-8"), timeout=120
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
return ToolResult(success=False, error=f"Azure TTS request failed: {exc}")
|
||||
|
||||
if response.status_code != 200:
|
||||
detail = response.text[:500] if response.text else ""
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"Azure TTS returned HTTP {response.status_code}: {detail}",
|
||||
)
|
||||
|
||||
output_path = Path(inputs.get("output_path", f"tts_output.{ext}"))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_bytes(response.content)
|
||||
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data={
|
||||
"provider": self.provider,
|
||||
"voice": voice,
|
||||
"text_length": len(inputs["text"]),
|
||||
"output": str(output_path),
|
||||
"format": azure_format,
|
||||
},
|
||||
artifacts=[str(output_path)],
|
||||
model=f"azure-neural-tts:{voice}",
|
||||
)
|
||||
@@ -210,7 +210,7 @@ class TTSSelector(BaseTool):
|
||||
if tool is None:
|
||||
return ToolResult(success=False, error="No TTS provider available.")
|
||||
|
||||
result = tool.execute(inputs)
|
||||
result = tool.execute(self._adapt_inputs(tool, inputs))
|
||||
if result.success:
|
||||
result.data.setdefault("selected_tool", tool.name)
|
||||
result.data["selected_provider"] = tool.provider
|
||||
@@ -224,6 +224,37 @@ class TTSSelector(BaseTool):
|
||||
]
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _adapt_inputs(tool: BaseTool, inputs: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Translate capability-level controls to provider-native inputs."""
|
||||
adapted = dict(inputs)
|
||||
if tool.name != "azure_tts":
|
||||
return adapted
|
||||
|
||||
if inputs.get("voice_id") and not inputs.get("voice"):
|
||||
adapted["voice"] = inputs["voice_id"]
|
||||
|
||||
speed = inputs.get("speaking_rate", inputs.get("speed"))
|
||||
if speed is not None and "rate" not in inputs:
|
||||
percent = round((float(speed) - 1.0) * 100)
|
||||
adapted["rate"] = f"{percent:+d}%" if percent else "0%"
|
||||
|
||||
pitch = inputs.get("pitch")
|
||||
if isinstance(pitch, (int, float)):
|
||||
adapted["pitch"] = f"{pitch:+g}st" if pitch else "0%"
|
||||
|
||||
# The selector's numeric style is ElevenLabs-specific. Azure's style
|
||||
# is a named express-as value such as "calm" or "newscast".
|
||||
if not isinstance(inputs.get("style"), str):
|
||||
adapted.pop("style", None)
|
||||
|
||||
output_format = str(inputs.get("output_format", ""))
|
||||
if output_format.startswith("mp3"):
|
||||
adapted["output_format"] = "mp3"
|
||||
elif output_format.startswith(("wav", "riff", "pcm")):
|
||||
adapted["output_format"] = "wav"
|
||||
return adapted
|
||||
|
||||
def _select_best_tool(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
|
||||
Reference in New Issue
Block a user