mirror of
https://github.com/calesthio/OpenMontage.git
synced 2026-08-05 15:20:40 +08:00
feat(stt): add Azure AI Speech as an optional cloud speech-to-text provider
Add an Azure AI Speech transcription tool. It is opt-in: when AZURE_SPEECH_KEY is configured the agent may prefer it for cloud STT, while the local faster-whisper `transcriber` stays the default offline path. Shared pipeline manifests are intentionally left unchanged, so no default provider selection is altered for existing users. - tools/analysis/azure_stt.py: new `azure_stt` tool (capability=analysis, provider=azure) calling the Fast Transcription REST API. The local file is uploaded via multipart and transcribed synchronously with word-level timestamps and optional diarization — no Blob storage or async polling. Output schema mirrors `transcriber` exactly, so it is a drop-in for `subtitle_gen` and other transcript consumers. Follows the existing provider-tool conventions (env-var status check, `_transcribe` helper, cost_usd/model on the result, fallback="transcriber"). - Auto-discovered by the registry; no registry or selector changes. - tests/tools/test_azure_stt.py: contract, discovery, status, response mapping, execute guardrails, and a mocked-network success path (no live API calls). - .agents/skills + .claude/skills: azure-speech-to-text Layer-3 skill. - docs/PROVIDERS.md: Azure AI Speech setup, API notes, and pricing. - .env.example, skills/INDEX.md, AGENT_GUIDE.md: document the optional cloud STT path alongside the default whisper transcriber.
This commit is contained in:
amartya-dev
115
.agents/skills/azure-speech-to-text/SKILL.md
Normal file
115
.agents/skills/azure-speech-to-text/SKILL.md
Normal file
@@ -0,0 +1,115 @@
|
||||
---
|
||||
name: azure-speech-to-text
|
||||
description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path.
|
||||
license: MIT
|
||||
compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
|
||||
metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
|
||||
---
|
||||
|
||||
# Azure AI Speech — Speech-to-Text
|
||||
|
||||
Transcribe audio to text with **Azure Fast Transcription** — synchronous,
|
||||
word-level timestamps, speaker diarization, and multi-language identification.
|
||||
In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`,
|
||||
`provider=azure`). It is an **optional cloud STT provider** — when
|
||||
`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local
|
||||
`transcriber` tool (faster-whisper) remains the **default offline path** and the
|
||||
fallback when Azure is unavailable.
|
||||
|
||||
> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview)
|
||||
|
||||
## Why Fast Transcription (not Batch)
|
||||
|
||||
Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because
|
||||
the pipeline transcribes **local audio files**:
|
||||
|
||||
| Surface | Input | Latency | Needs |
|
||||
|---------|-------|---------|-------|
|
||||
| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region |
|
||||
| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing |
|
||||
| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package |
|
||||
|
||||
Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just
|
||||
`requests` and the two env vars.
|
||||
|
||||
## Setup
|
||||
|
||||
Create a **Speech** resource in the [Azure portal](https://portal.azure.com);
|
||||
copy the key and region from its **Keys and Endpoint** page.
|
||||
|
||||
```bash
|
||||
export AZURE_SPEECH_KEY=your_speech_resource_key
|
||||
export AZURE_SPEECH_REGION=eastus # your resource's region
|
||||
# export AZURE_SPEECH_ENDPOINT=https://... # optional: overrides region
|
||||
```
|
||||
|
||||
`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
|
||||
`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set.
|
||||
|
||||
## Using it in a pipeline
|
||||
|
||||
Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output
|
||||
matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen`
|
||||
and any stage that consumes a transcript.
|
||||
|
||||
```python
|
||||
from tools.tool_registry import registry
|
||||
registry.discover()
|
||||
stt = registry._tools["azure_stt"]
|
||||
|
||||
result = stt.execute({
|
||||
"input_path": "projects/my-video/assets/audio/narration.mp3",
|
||||
# "language": "en", # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID
|
||||
# "diarize": True, # speaker labels, no HuggingFace token needed
|
||||
# "max_speakers": 4,
|
||||
"output_dir": "projects/my-video/artifacts",
|
||||
})
|
||||
if result.success:
|
||||
segs = result.data["segments"] # [{id,start,end,text,words:[...]}]
|
||||
words = result.data["word_timestamps"] # flat [{word,start,end,probability}]
|
||||
```
|
||||
|
||||
If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber`
|
||||
(local whisper) — its `execute` signature and output are identical.
|
||||
|
||||
## Parameters that matter
|
||||
|
||||
- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it
|
||||
when you know the language; it is faster and more accurate than auto-ID.
|
||||
- **`candidate_locales`** — when `language` is omitted, Azure runs language
|
||||
identification across this shortlist. Narrow it to the languages you actually
|
||||
expect; a huge list slows detection and invites misclassification.
|
||||
- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews,
|
||||
podcasts). Set `max_speakers` to the real upper bound.
|
||||
- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`.
|
||||
|
||||
## Response shape (mapped to the transcriber schema)
|
||||
|
||||
The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is
|
||||
converted to seconds and the OpenMontage transcript schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"segments": [
|
||||
{"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world",
|
||||
"speaker": 1,
|
||||
"words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]}
|
||||
],
|
||||
"word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}],
|
||||
"language": "en-US",
|
||||
"duration_seconds": 2.4,
|
||||
"provider": "azure"
|
||||
}
|
||||
```
|
||||
|
||||
Note: Fast Transcription has no *per-word* confidence, so each word carries the
|
||||
**phrase** confidence in `probability`.
|
||||
|
||||
## Limits & tips
|
||||
|
||||
- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk
|
||||
jobs, use Azure Batch Transcription instead.
|
||||
- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if
|
||||
you only need speech — smaller upload, same result.
|
||||
- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check
|
||||
the first and last cues against the source audio.
|
||||
115
.claude/skills/azure-speech-to-text/SKILL.md
Normal file
115
.claude/skills/azure-speech-to-text/SKILL.md
Normal file
@@ -0,0 +1,115 @@
|
||||
---
|
||||
name: azure-speech-to-text
|
||||
description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path.
|
||||
license: MIT
|
||||
compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
|
||||
metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
|
||||
---
|
||||
|
||||
# Azure AI Speech — Speech-to-Text
|
||||
|
||||
Transcribe audio to text with **Azure Fast Transcription** — synchronous,
|
||||
word-level timestamps, speaker diarization, and multi-language identification.
|
||||
In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`,
|
||||
`provider=azure`). It is an **optional cloud STT provider** — when
|
||||
`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local
|
||||
`transcriber` tool (faster-whisper) remains the **default offline path** and the
|
||||
fallback when Azure is unavailable.
|
||||
|
||||
> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview)
|
||||
|
||||
## Why Fast Transcription (not Batch)
|
||||
|
||||
Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because
|
||||
the pipeline transcribes **local audio files**:
|
||||
|
||||
| Surface | Input | Latency | Needs |
|
||||
|---------|-------|---------|-------|
|
||||
| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region |
|
||||
| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing |
|
||||
| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package |
|
||||
|
||||
Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just
|
||||
`requests` and the two env vars.
|
||||
|
||||
## Setup
|
||||
|
||||
Create a **Speech** resource in the [Azure portal](https://portal.azure.com);
|
||||
copy the key and region from its **Keys and Endpoint** page.
|
||||
|
||||
```bash
|
||||
export AZURE_SPEECH_KEY=your_speech_resource_key
|
||||
export AZURE_SPEECH_REGION=eastus # your resource's region
|
||||
# export AZURE_SPEECH_ENDPOINT=https://... # optional: overrides region
|
||||
```
|
||||
|
||||
`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
|
||||
`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set.
|
||||
|
||||
## Using it in a pipeline
|
||||
|
||||
Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output
|
||||
matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen`
|
||||
and any stage that consumes a transcript.
|
||||
|
||||
```python
|
||||
from tools.tool_registry import registry
|
||||
registry.discover()
|
||||
stt = registry._tools["azure_stt"]
|
||||
|
||||
result = stt.execute({
|
||||
"input_path": "projects/my-video/assets/audio/narration.mp3",
|
||||
# "language": "en", # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID
|
||||
# "diarize": True, # speaker labels, no HuggingFace token needed
|
||||
# "max_speakers": 4,
|
||||
"output_dir": "projects/my-video/artifacts",
|
||||
})
|
||||
if result.success:
|
||||
segs = result.data["segments"] # [{id,start,end,text,words:[...]}]
|
||||
words = result.data["word_timestamps"] # flat [{word,start,end,probability}]
|
||||
```
|
||||
|
||||
If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber`
|
||||
(local whisper) — its `execute` signature and output are identical.
|
||||
|
||||
## Parameters that matter
|
||||
|
||||
- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it
|
||||
when you know the language; it is faster and more accurate than auto-ID.
|
||||
- **`candidate_locales`** — when `language` is omitted, Azure runs language
|
||||
identification across this shortlist. Narrow it to the languages you actually
|
||||
expect; a huge list slows detection and invites misclassification.
|
||||
- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews,
|
||||
podcasts). Set `max_speakers` to the real upper bound.
|
||||
- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`.
|
||||
|
||||
## Response shape (mapped to the transcriber schema)
|
||||
|
||||
The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is
|
||||
converted to seconds and the OpenMontage transcript schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"segments": [
|
||||
{"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world",
|
||||
"speaker": 1,
|
||||
"words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]}
|
||||
],
|
||||
"word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}],
|
||||
"language": "en-US",
|
||||
"duration_seconds": 2.4,
|
||||
"provider": "azure"
|
||||
}
|
||||
```
|
||||
|
||||
Note: Fast Transcription has no *per-word* confidence, so each word carries the
|
||||
**phrase** confidence in `probability`.
|
||||
|
||||
## Limits & tips
|
||||
|
||||
- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk
|
||||
jobs, use Azure Batch Transcription instead.
|
||||
- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if
|
||||
you only need speech — smaller upload, same result.
|
||||
- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check
|
||||
the first and last cues against the source audio.
|
||||
@@ -56,6 +56,12 @@ UNSPLASH_ACCESS_KEY= # Unsplash stock images (free developer key)
|
||||
|
||||
# --- Analysis ---
|
||||
HF_TOKEN= # HuggingFace token — enables speaker diarization in transcriber
|
||||
# Speech-to-text: optional Azure AI Speech (Fast Transcription). When set, the
|
||||
# agent prefers azure_stt for cloud STT; the local faster-whisper transcriber
|
||||
# remains the default offline path.
|
||||
AZURE_SPEECH_KEY= # Azure AI Speech resource key ('Keys and Endpoint' page)
|
||||
AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
|
||||
# AZURE_SPEECH_ENDPOINT= # Optional: full custom endpoint URL (overrides region)
|
||||
|
||||
# --- Avatar (local installs) ---
|
||||
# WAV2LIP_PATH= # Path to cloned Wav2Lip repo (for lip sync)
|
||||
|
||||
@@ -678,7 +678,8 @@ The `.agents/skills/` directory is large. When you're not coming in through a to
|
||||
| **Image generation** | `bfl-api`, `flux-best-practices` |
|
||||
| **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` |
|
||||
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` |
|
||||
| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `speech-to-text`, `agents` |
|
||||
| **Speech-to-text** | `speech-to-text` (whisper `transcriber` — default, offline), `azure-speech-to-text` (optional cloud STT — tool `azure_stt`, preferred when `AZURE_SPEECH_KEY` is set) |
|
||||
| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `agents` |
|
||||
| **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) |
|
||||
| **Visualization** | `beautiful-mermaid`, `d3-viz`, `manim-composer`, `manimce-best-practices`, `manimgl-best-practices` |
|
||||
| **Media editing** | `video-edit`, `video-download`, `video-understand`, `video-toolkit`, `visual-style` |
|
||||
|
||||
@@ -43,6 +43,10 @@ DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin nar
|
||||
DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type
|
||||
DASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps)
|
||||
|
||||
# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default)
|
||||
AZURE_SPEECH_KEY= # Azure AI Speech — Fast Transcription (word-level timestamps)
|
||||
AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
|
||||
|
||||
# MULTI-MODEL GATEWAY (one key, 6+ tools)
|
||||
FAL_KEY= # FLUX, Recraft, Kling, Veo, MiniMax video
|
||||
|
||||
@@ -245,6 +249,54 @@ Doubao Speech 2.0 is billed by character package or usage in Volcengine. OpenMon
|
||||
|
||||
---
|
||||
|
||||
### Azure AI Speech — Speech-to-Text
|
||||
|
||||
> **Cloud transcription.** Azure AI Speech Fast Transcription turns local audio into text with word-level timestamps, speaker diarization, and multi-language identification — no GPU required. Optional: the local faster-whisper `transcriber` remains the default offline STT path. When `AZURE_SPEECH_KEY` is set, the agent prefers `azure_stt` for cloud transcription.
|
||||
|
||||
**Tools unlocked:** `azure_stt`
|
||||
**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_SPEECH_ENDPOINT`)
|
||||
|
||||
#### Setup
|
||||
|
||||
1. In the [Azure portal](https://portal.azure.com), create a **Speech** resource (Azure AI services → Speech service).
|
||||
2. Open the resource's **Keys and Endpoint** page.
|
||||
3. Copy **KEY 1** and the **Location/Region** (e.g. `eastus`).
|
||||
4. Add to `.env`:
|
||||
```bash
|
||||
AZURE_SPEECH_KEY=your-speech-resource-key
|
||||
AZURE_SPEECH_REGION=eastus
|
||||
# AZURE_SPEECH_ENDPOINT=https://<custom>... # optional, overrides region
|
||||
```
|
||||
|
||||
#### API Notes
|
||||
|
||||
OpenMontage uses the **Fast Transcription** REST endpoint, which accepts a local
|
||||
audio file directly (multipart upload) and returns a synchronous result — no
|
||||
Azure Blob storage, SAS URLs, or async job polling:
|
||||
|
||||
```text
|
||||
POST https://{region}.api.cognitive.microsoft.com/speechtotext/transcriptions:transcribe?api-version=2024-11-15
|
||||
Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY}
|
||||
```
|
||||
|
||||
For files longer than ~2 hours or bulk jobs, use Azure Batch Transcription instead (not wired into OpenMontage).
|
||||
|
||||
#### What It Is Best For
|
||||
|
||||
- Cloud transcription with word-level timestamps and no local GPU
|
||||
- Multi-language auto-detection across a candidate locale set
|
||||
- Speaker diarization without a HuggingFace token
|
||||
- Subtitle timing metadata that flows straight into `subtitle_gen`
|
||||
|
||||
#### Pricing
|
||||
|
||||
Azure AI Speech Standard (S0) bills speech-to-text by audio-hour (roughly
|
||||
$1.00/audio-hour at time of writing; a free F0 tier includes a limited monthly
|
||||
allowance). OpenMontage estimates cost from the transcribed audio duration. See
|
||||
[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
|
||||
|
||||
---
|
||||
|
||||
### Google — TTS + Imagen + Music + Video (Shared Key)
|
||||
|
||||
> **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. Imagen 4 generates high-quality images. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips.
|
||||
|
||||
@@ -81,7 +81,8 @@ Key capability families to look for in the output:
|
||||
| FFmpeg | `core/ffmpeg.md` | Video encoding, filtering, composition | `ffmpeg`, `video-toolkit` |
|
||||
| Remotion | `core/remotion.md` | React-based composition, Phase 3+ | `remotion-best-practices`, `remotion` |
|
||||
| HyperFrames | `core/hyperframes.md` | HTML/CSS/GSAP composition runtime — kinetic typography, music-to-video, product promos, website capture. Vendored at v0.7.17 (2026-06-27). | `hyperframes` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type/narration), `hyperframes-media` (TTS/BGM/SFX/captions), `hyperframes-animation` (all motion), `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video` (beats-driven), `website-to-video`, `remotion-to-hyperframes` (migration), `gsap-core`, `gsap-timeline` |
|
||||
| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps | `speech-to-text` |
|
||||
| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps — default STT (offline, free) | `speech-to-text` |
|
||||
| Azure STT | (tool: `azure_stt`) | Optional cloud speech-to-text, word-level timestamps — preferred when `AZURE_SPEECH_KEY` is set | `azure-speech-to-text` |
|
||||
| Subtitle Sync | `core/subtitle-sync.md` | Subtitle timing and alignment | `remotion-best-practices` |
|
||||
| Color Grading | `core/color-grading.md` | FFmpeg color profiles, LUT workflow, accessibility | `ffmpeg` |
|
||||
|
||||
@@ -306,7 +307,7 @@ Claude Code accesses them via symlinks in `.claude/skills/`.
|
||||
|----------|-----------------|--------|
|
||||
| **Video Composition** | `remotion-best-practices`, `remotion`, `hyperframes` (router), `hyperframes-core`, `hyperframes-creative`, `hyperframes-media`, `hyperframes-animation`, `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video`, `remotion-to-hyperframes`, `website-to-video` | `remotion-dev/skills`, `digitalsamba/claude-code-video-toolkit`, `heygen-com/hyperframes` (vendored v0.7.17, see `.agents/skills/hyperframes/PROVENANCE.md`) |
|
||||
| **Video Processing** | `ffmpeg`, `video-toolkit` | `digitalsamba/claude-code-video-toolkit` |
|
||||
| **TTS & Audio** | `text-to-speech`, `speech-to-text`, `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` |
|
||||
| **TTS & Audio** | `text-to-speech`, `speech-to-text` (whisper, default STT), `azure-speech-to-text` (optional cloud STT), `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` |
|
||||
| **Image Generation** | `flux-best-practices`, `bfl-api`, `grok-media` | `black-forest-labs/skills`, local OpenMontage skill |
|
||||
| **Math Animation** | `manimce-best-practices`, `manimgl-best-practices`, `manim-composer` | `adithya-s-k/manim_skill` |
|
||||
| **3D Graphics** | `threejs-animation`, `threejs-fundamentals`, `threejs-geometry`, `threejs-interaction`, `threejs-lighting`, `threejs-loaders`, `threejs-materials`, `threejs-postprocessing`, `threejs-shaders`, `threejs-textures` | `cloudai-x/threejs-skills` |
|
||||
|
||||
242
tests/tools/test_azure_stt.py
Normal file
242
tests/tools/test_azure_stt.py
Normal file
@@ -0,0 +1,242 @@
|
||||
"""Focused tests for the Azure AI Speech (Fast Transcription) STT tool.
|
||||
|
||||
No live API calls: the network layer is monkeypatched. Covers the tool
|
||||
contract, registry discovery, status behavior, the response→transcript
|
||||
mapping, execute() guardrails, and downstream subtitle compatibility.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from tools.base_tool import BaseTool, ToolStatus, ToolTier, ToolRuntime
|
||||
from tools.tool_registry import ToolRegistry
|
||||
from tools.analysis.azure_stt import AzureSpeechToText
|
||||
from tools.subtitle.subtitle_gen import SubtitleGen
|
||||
|
||||
|
||||
# A representative Fast Transcription response payload.
|
||||
SAMPLE_PAYLOAD = {
|
||||
"durationMilliseconds": 2400,
|
||||
"combinedPhrases": [{"text": "Hello world this is a test"}],
|
||||
"phrases": [
|
||||
{
|
||||
"offsetMilliseconds": 0,
|
||||
"durationMilliseconds": 1200,
|
||||
"text": "Hello world",
|
||||
"locale": "en-US",
|
||||
"confidence": 0.97,
|
||||
"speaker": 1,
|
||||
"words": [
|
||||
{"text": "Hello", "offsetMilliseconds": 0, "durationMilliseconds": 500},
|
||||
{"text": "world", "offsetMilliseconds": 500, "durationMilliseconds": 700},
|
||||
],
|
||||
},
|
||||
{
|
||||
"offsetMilliseconds": 1200,
|
||||
"durationMilliseconds": 1200,
|
||||
"text": "this is a test",
|
||||
"locale": "en-US",
|
||||
"confidence": 0.95,
|
||||
"words": [
|
||||
{"text": "this", "offsetMilliseconds": 1200, "durationMilliseconds": 300},
|
||||
{"text": "is", "offsetMilliseconds": 1500, "durationMilliseconds": 200},
|
||||
{"text": "a", "offsetMilliseconds": 1700, "durationMilliseconds": 200},
|
||||
{"text": "test", "offsetMilliseconds": 1900, "durationMilliseconds": 500},
|
||||
],
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
def __init__(self, payload, status_code=200, text="ok"):
|
||||
self._payload = payload
|
||||
self.status_code = status_code
|
||||
self.text = text
|
||||
|
||||
def json(self):
|
||||
return self._payload
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def azure_env(monkeypatch):
|
||||
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
|
||||
monkeypatch.setenv("AZURE_SPEECH_REGION", "eastus")
|
||||
|
||||
|
||||
# ---- Contract ----
|
||||
|
||||
class TestContract:
|
||||
def test_inherits_base_tool(self):
|
||||
assert issubclass(AzureSpeechToText, BaseTool)
|
||||
|
||||
def test_identity(self):
|
||||
t = AzureSpeechToText()
|
||||
assert t.name == "azure_stt"
|
||||
assert t.capability == "analysis"
|
||||
assert t.provider == "azure"
|
||||
assert t.runtime == ToolRuntime.API
|
||||
assert t.tier == ToolTier.CORE
|
||||
assert t.fallback == "transcriber"
|
||||
assert "azure-speech-to-text" in t.agent_skills
|
||||
assert len(t.capabilities) > 0
|
||||
|
||||
def test_get_info_valid(self):
|
||||
info = AzureSpeechToText().get_info()
|
||||
assert info["name"] == "azure_stt"
|
||||
assert info["capability"] == "analysis"
|
||||
assert "input_path" in info["input_schema"]["properties"]
|
||||
|
||||
def test_estimate_cost_by_duration(self):
|
||||
t = AzureSpeechToText()
|
||||
assert t.estimate_cost({"duration_seconds": 3600}) == pytest.approx(1.0)
|
||||
assert t.estimate_cost({}) == 0.0
|
||||
|
||||
|
||||
# ---- Registry discovery ----
|
||||
|
||||
class TestDiscovery:
|
||||
def test_discoverable(self):
|
||||
reg = ToolRegistry()
|
||||
reg.discover("tools")
|
||||
assert reg.get("azure_stt") is not None
|
||||
|
||||
def test_capability_routing(self):
|
||||
reg = ToolRegistry()
|
||||
reg.discover("tools")
|
||||
names = [t.name for t in reg.get_by_capability("analysis")]
|
||||
assert "azure_stt" in names
|
||||
|
||||
|
||||
# ---- Status behavior ----
|
||||
|
||||
class TestStatus:
|
||||
def test_unavailable_without_env(self, monkeypatch):
|
||||
monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
|
||||
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
|
||||
monkeypatch.delenv("AZURE_SPEECH_ENDPOINT", raising=False)
|
||||
assert AzureSpeechToText().get_status() == ToolStatus.UNAVAILABLE
|
||||
|
||||
def test_available_with_key_and_region(self, azure_env):
|
||||
assert AzureSpeechToText().get_status() == ToolStatus.AVAILABLE
|
||||
|
||||
def test_available_with_key_and_endpoint(self, monkeypatch):
|
||||
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
|
||||
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
|
||||
monkeypatch.setenv("AZURE_SPEECH_ENDPOINT", "https://custom.example.com")
|
||||
assert AzureSpeechToText().get_status() == ToolStatus.AVAILABLE
|
||||
|
||||
def test_key_alone_is_not_enough(self, monkeypatch):
|
||||
monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
|
||||
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
|
||||
monkeypatch.delenv("AZURE_SPEECH_ENDPOINT", raising=False)
|
||||
assert AzureSpeechToText().get_status() == ToolStatus.UNAVAILABLE
|
||||
|
||||
|
||||
# ---- Response → transcript mapping (the risky logic) ----
|
||||
|
||||
class TestParsePayload:
|
||||
def test_maps_segments_words_and_timestamps(self):
|
||||
data = AzureSpeechToText._parse_payload(SAMPLE_PAYLOAD, ["en-US"])
|
||||
assert data["language"] == "en-US"
|
||||
assert data["duration_seconds"] == 2.4
|
||||
assert len(data["segments"]) == 2
|
||||
|
||||
seg0 = data["segments"][0]
|
||||
assert seg0["text"] == "Hello world"
|
||||
assert seg0["start"] == 0.0 and seg0["end"] == 1.2
|
||||
assert seg0["speaker"] == 1
|
||||
# ms → seconds conversion on words
|
||||
assert seg0["words"][0] == {
|
||||
"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.97,
|
||||
}
|
||||
# flat word stream aggregates every phrase
|
||||
assert len(data["word_timestamps"]) == 6
|
||||
|
||||
def test_locale_resolution(self):
|
||||
r = AzureSpeechToText._resolve_locales
|
||||
assert r({"language": "en"}) == ["en-US"] # ISO → default locale
|
||||
assert r({"language": "en-GB"}) == ["en-GB"] # explicit locale kept
|
||||
assert r({"candidate_locales": ["fr-FR"]}) == ["fr-FR"]
|
||||
assert len(r({})) > 1 # falls back to a shortlist
|
||||
|
||||
def test_duration_falls_back_to_last_segment(self):
|
||||
payload = {k: v for k, v in SAMPLE_PAYLOAD.items() if k != "durationMilliseconds"}
|
||||
data = AzureSpeechToText._parse_payload(payload, ["en-US"])
|
||||
assert data["duration_seconds"] == pytest.approx(2.4)
|
||||
|
||||
def test_output_feeds_subtitle_gen(self, tmp_path):
|
||||
"""Parsed segments must be a drop-in for the subtitle generator."""
|
||||
data = AzureSpeechToText._parse_payload(SAMPLE_PAYLOAD, ["en-US"])
|
||||
out = tmp_path / "captions.srt"
|
||||
res = SubtitleGen().execute(
|
||||
{"segments": data["segments"], "format": "srt", "output_path": str(out)}
|
||||
)
|
||||
assert res.success
|
||||
assert out.exists()
|
||||
assert "Hello world" in out.read_text()
|
||||
|
||||
|
||||
# ---- execute() guardrails + mocked success ----
|
||||
|
||||
class TestExecute:
|
||||
def test_missing_file(self, azure_env, tmp_path):
|
||||
res = AzureSpeechToText().execute({"input_path": str(tmp_path / "nope.wav")})
|
||||
assert not res.success
|
||||
assert "not found" in res.error.lower()
|
||||
|
||||
def test_missing_credentials(self, monkeypatch, tmp_path):
|
||||
monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
|
||||
monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
|
||||
audio = tmp_path / "a.wav"
|
||||
audio.write_bytes(b"RIFF....")
|
||||
res = AzureSpeechToText().execute({"input_path": str(audio)})
|
||||
assert not res.success
|
||||
assert "not configured" in res.error.lower()
|
||||
|
||||
def test_success_path_mocked(self, azure_env, tmp_path, monkeypatch):
|
||||
import requests
|
||||
|
||||
captured = {}
|
||||
|
||||
def fake_post(url, headers=None, files=None, timeout=None):
|
||||
captured["url"] = url
|
||||
captured["headers"] = headers
|
||||
return _FakeResponse(SAMPLE_PAYLOAD)
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
|
||||
audio = tmp_path / "a.wav"
|
||||
audio.write_bytes(b"RIFF....")
|
||||
res = AzureSpeechToText().execute(
|
||||
{"input_path": str(audio), "language": "en", "output_dir": str(tmp_path)}
|
||||
)
|
||||
|
||||
assert res.success
|
||||
assert res.model == "azure-fast-transcription"
|
||||
assert res.cost_usd == pytest.approx(0.0007, abs=1e-4) # 2.4s at $1/hr
|
||||
assert res.data["provider"] == "azure"
|
||||
assert len(res.data["segments"]) == 2
|
||||
# transcript JSON artifact was written
|
||||
assert res.artifacts and Path(res.artifacts[0]).exists()
|
||||
# correct endpoint + auth header used
|
||||
assert "transcriptions:transcribe" in captured["url"]
|
||||
assert captured["headers"]["Ocp-Apim-Subscription-Key"] == "fake-key"
|
||||
|
||||
def test_http_error_surfaced(self, azure_env, tmp_path, monkeypatch):
|
||||
import requests
|
||||
|
||||
monkeypatch.setattr(
|
||||
requests, "post",
|
||||
lambda *a, **k: _FakeResponse(None, status_code=401, text="Unauthorized"),
|
||||
)
|
||||
audio = tmp_path / "a.wav"
|
||||
audio.write_bytes(b"RIFF....")
|
||||
res = AzureSpeechToText().execute({"input_path": str(audio), "output_dir": str(tmp_path)})
|
||||
assert not res.success
|
||||
assert "401" in res.error
|
||||
365
tools/analysis/azure_stt.py
Normal file
365
tools/analysis/azure_stt.py
Normal file
@@ -0,0 +1,365 @@
|
||||
"""Azure AI Speech transcription tool (Fast Transcription REST API).
|
||||
|
||||
Speech-to-text with word-level timestamps and optional speaker diarization,
|
||||
served by Azure AI Speech. This is an optional cloud STT provider; when
|
||||
`AZURE_SPEECH_KEY` is configured the agent may prefer it for cloud
|
||||
transcription, while the local faster-whisper `transcriber` tool remains the
|
||||
default offline path.
|
||||
|
||||
Uses the **Fast Transcription** REST endpoint, which accepts a local audio
|
||||
file directly (multipart upload) and returns a synchronous JSON result — no
|
||||
Azure Blob storage, SAS URLs, or async job polling required. Output is shaped
|
||||
to match `transcriber` exactly (segments + word_timestamps) so it is a
|
||||
drop-in for `subtitle_gen` and every pipeline that consumes a transcript.
|
||||
|
||||
Docs: https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from tools.base_tool import (
|
||||
BaseTool,
|
||||
Determinism,
|
||||
ExecutionMode,
|
||||
ResourceProfile,
|
||||
ResumeSupport,
|
||||
RetryPolicy,
|
||||
ToolResult,
|
||||
ToolRuntime,
|
||||
ToolStability,
|
||||
ToolStatus,
|
||||
ToolTier,
|
||||
)
|
||||
|
||||
# Fast Transcription API version (GA).
|
||||
_API_VERSION = "2024-11-15"
|
||||
|
||||
# Candidate locales used for automatic language identification when the caller
|
||||
# does not pin a language. Azure Fast Transcription accepts several locales and
|
||||
# picks the best match per phrase.
|
||||
_DEFAULT_CANDIDATE_LOCALES = [
|
||||
"en-US",
|
||||
"es-ES",
|
||||
"fr-FR",
|
||||
"de-DE",
|
||||
"it-IT",
|
||||
"pt-BR",
|
||||
"hi-IN",
|
||||
"ja-JP",
|
||||
"zh-CN",
|
||||
]
|
||||
|
||||
# Minimal ISO 639-1 -> default BCP-47 locale map so callers can keep passing the
|
||||
# short codes the whisper transcriber accepts (e.g. "en", "es").
|
||||
_ISO_TO_LOCALE = {
|
||||
"en": "en-US",
|
||||
"es": "es-ES",
|
||||
"fr": "fr-FR",
|
||||
"de": "de-DE",
|
||||
"it": "it-IT",
|
||||
"pt": "pt-BR",
|
||||
"hi": "hi-IN",
|
||||
"ja": "ja-JP",
|
||||
"ko": "ko-KR",
|
||||
"zh": "zh-CN",
|
||||
"ar": "ar-EG",
|
||||
"ru": "ru-RU",
|
||||
"nl": "nl-NL",
|
||||
"tr": "tr-TR",
|
||||
"pl": "pl-PL",
|
||||
"id": "id-ID",
|
||||
}
|
||||
|
||||
|
||||
class AzureSpeechToText(BaseTool):
|
||||
name = "azure_stt"
|
||||
version = "0.1.0"
|
||||
tier = ToolTier.CORE
|
||||
capability = "analysis"
|
||||
provider = "azure"
|
||||
stability = ToolStability.BETA
|
||||
execution_mode = ExecutionMode.SYNC
|
||||
determinism = Determinism.DETERMINISTIC
|
||||
runtime = ToolRuntime.API
|
||||
|
||||
# Availability is decided by get_status() (env var check), mirroring the
|
||||
# ElevenLabs provider tool — dependencies stays empty.
|
||||
dependencies = []
|
||||
install_instructions = (
|
||||
"Set your Azure AI Speech credentials:\n"
|
||||
" export AZURE_SPEECH_KEY=your_speech_resource_key\n"
|
||||
" export AZURE_SPEECH_REGION=eastus # your Speech resource region\n"
|
||||
"Create a Speech resource in the Azure portal "
|
||||
"(https://portal.azure.com) — the key and region are on its "
|
||||
"'Keys and Endpoint' page. Optionally set AZURE_SPEECH_ENDPOINT to a "
|
||||
"full custom endpoint URL instead of the region."
|
||||
)
|
||||
agent_skills = ["azure-speech-to-text"]
|
||||
|
||||
capabilities = [
|
||||
"transcribe",
|
||||
"word_timestamps",
|
||||
"diarization",
|
||||
"language_detection",
|
||||
]
|
||||
|
||||
best_for = [
|
||||
"Cloud transcription with word-level timestamps and no GPU",
|
||||
"Multi-language auto-detection across a candidate locale set",
|
||||
"Speaker diarization without a HuggingFace token",
|
||||
]
|
||||
not_good_for = [
|
||||
"Fully offline runs (use the whisper `transcriber` fallback)",
|
||||
"Single files longer than ~2 hours (use Azure Batch transcription)",
|
||||
]
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"required": ["input_path"],
|
||||
"properties": {
|
||||
"input_path": {"type": "string", "description": "Path to audio or video file"},
|
||||
"language": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"ISO 639-1 code (e.g. 'en') or BCP-47 locale (e.g. 'en-US'). "
|
||||
"Omit for automatic language identification across common locales."
|
||||
),
|
||||
},
|
||||
"candidate_locales": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": (
|
||||
"BCP-47 locales to consider for language ID when `language` is "
|
||||
"not set. Defaults to a common-language shortlist."
|
||||
),
|
||||
},
|
||||
"diarize": {"type": "boolean", "default": False},
|
||||
"max_speakers": {"type": "integer", "default": 4},
|
||||
"profanity_filter": {
|
||||
"type": "string",
|
||||
"enum": ["None", "Masked", "Removed", "Tags"],
|
||||
"default": "Masked",
|
||||
},
|
||||
"output_dir": {"type": "string", "description": "Directory for output files"},
|
||||
},
|
||||
}
|
||||
|
||||
output_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"segments": {"type": "array"},
|
||||
"word_timestamps": {"type": "array"},
|
||||
"language": {"type": "string"},
|
||||
"duration_seconds": {"type": "number"},
|
||||
},
|
||||
}
|
||||
|
||||
resource_profile = ResourceProfile(
|
||||
cpu_cores=1,
|
||||
ram_mb=256,
|
||||
vram_mb=0,
|
||||
disk_mb=50,
|
||||
network_required=True,
|
||||
)
|
||||
|
||||
retry_policy = RetryPolicy(
|
||||
max_retries=2,
|
||||
retryable_errors=["ConnectionError", "Timeout", "429", "503"],
|
||||
)
|
||||
resume_support = ResumeSupport.FROM_START
|
||||
idempotency_key_fields = ["input_path", "language", "diarize"]
|
||||
side_effects = ["writes transcript JSON to output_dir", "sends audio to Azure AI Speech"]
|
||||
fallback = "transcriber"
|
||||
fallback_tools = ["transcriber"]
|
||||
user_visible_verification = [
|
||||
"Check transcript text against source audio",
|
||||
"Verify word timestamps align with speech",
|
||||
]
|
||||
|
||||
def get_status(self) -> ToolStatus:
|
||||
if os.environ.get("AZURE_SPEECH_KEY") and (
|
||||
os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_SPEECH_ENDPOINT")
|
||||
):
|
||||
return ToolStatus.AVAILABLE
|
||||
return ToolStatus.UNAVAILABLE
|
||||
|
||||
# Azure AI Speech Fast Transcription bills per audio-hour (Standard tier is
|
||||
# roughly $1.00/audio-hour at time of writing).
|
||||
COST_PER_AUDIO_HOUR = 1.0
|
||||
|
||||
def estimate_cost(self, inputs: dict[str, Any]) -> float:
|
||||
# Duration is unknown until the audio is transcribed; return the actual
|
||||
# cost by passing duration_seconds, else 0.0 as a pre-call estimate.
|
||||
seconds = inputs.get("duration_seconds", 0) or 0
|
||||
return round((seconds / 3600.0) * self.COST_PER_AUDIO_HOUR, 4)
|
||||
|
||||
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
|
||||
# Fast Transcription is well under real-time; conservative default.
|
||||
return 20.0
|
||||
|
||||
def _endpoint(self) -> str:
|
||||
endpoint = os.environ.get("AZURE_SPEECH_ENDPOINT")
|
||||
if endpoint:
|
||||
return endpoint.rstrip("/")
|
||||
region = os.environ.get("AZURE_SPEECH_REGION", "").strip()
|
||||
return f"https://{region}.api.cognitive.microsoft.com"
|
||||
|
||||
@staticmethod
|
||||
def _resolve_locales(inputs: dict[str, Any]) -> list[str]:
|
||||
language = inputs.get("language")
|
||||
if language:
|
||||
if "-" in language: # already a BCP-47 locale
|
||||
return [language]
|
||||
return [_ISO_TO_LOCALE.get(language.lower(), f"{language.lower()}-US")]
|
||||
candidates = inputs.get("candidate_locales")
|
||||
return list(candidates) if candidates else list(_DEFAULT_CANDIDATE_LOCALES)
|
||||
|
||||
def execute(self, inputs: dict[str, Any]) -> ToolResult:
|
||||
input_path = Path(inputs["input_path"])
|
||||
if not input_path.exists():
|
||||
return ToolResult(success=False, error=f"Input file not found: {input_path}")
|
||||
|
||||
api_key = os.environ.get("AZURE_SPEECH_KEY")
|
||||
if not api_key or not (
|
||||
os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_SPEECH_ENDPOINT")
|
||||
):
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="Azure Speech is not configured. " + self.install_instructions,
|
||||
)
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
result = self._transcribe(inputs, api_key, input_path)
|
||||
except Exception as exc:
|
||||
return ToolResult(success=False, error=f"Transcription failed: {exc}")
|
||||
|
||||
result.duration_seconds = round(time.time() - start, 2)
|
||||
result.model = "azure-fast-transcription"
|
||||
result.cost_usd = self.estimate_cost(
|
||||
{"duration_seconds": result.data.get("duration_seconds", 0)}
|
||||
)
|
||||
return result
|
||||
|
||||
def _transcribe(
|
||||
self, inputs: dict[str, Any], api_key: str, input_path: Path
|
||||
) -> ToolResult:
|
||||
import requests
|
||||
|
||||
output_dir = Path(inputs.get("output_dir", input_path.parent))
|
||||
diarize = inputs.get("diarize", False)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
locales = self._resolve_locales(inputs)
|
||||
definition: dict[str, Any] = {
|
||||
"locales": locales,
|
||||
"profanityFilterMode": inputs.get("profanity_filter", "Masked"),
|
||||
}
|
||||
if diarize:
|
||||
definition["diarization"] = {
|
||||
"maxSpeakers": inputs.get("max_speakers", 4),
|
||||
"enabled": True,
|
||||
}
|
||||
|
||||
url = f"{self._endpoint()}/speechtotext/transcriptions:transcribe?api-version={_API_VERSION}"
|
||||
headers = {"Ocp-Apim-Subscription-Key": api_key}
|
||||
|
||||
try:
|
||||
with open(input_path, "rb") as audio_file:
|
||||
response = requests.post(
|
||||
url,
|
||||
headers=headers,
|
||||
files={
|
||||
"audio": (input_path.name, audio_file, "application/octet-stream"),
|
||||
"definition": (None, json.dumps(definition), "application/json"),
|
||||
},
|
||||
timeout=600,
|
||||
)
|
||||
except requests.RequestException as exc: # network/timeout
|
||||
return ToolResult(success=False, error=f"Azure Speech request failed: {exc}")
|
||||
|
||||
if response.status_code != 200:
|
||||
detail = response.text[:500] if response.text else ""
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error=f"Azure Speech returned HTTP {response.status_code}: {detail}",
|
||||
)
|
||||
|
||||
try:
|
||||
payload = response.json()
|
||||
except ValueError:
|
||||
return ToolResult(
|
||||
success=False,
|
||||
error="Azure Speech returned a non-JSON response.",
|
||||
)
|
||||
|
||||
result_data = self._parse_payload(payload, locales)
|
||||
result_data["provider"] = "azure"
|
||||
output_path = output_dir / f"{input_path.stem}_transcript.json"
|
||||
output_path.write_text(json.dumps(result_data, indent=2), encoding="utf-8")
|
||||
|
||||
return ToolResult(
|
||||
success=True,
|
||||
data=result_data,
|
||||
artifacts=[str(output_path)],
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _parse_payload(payload: dict[str, Any], requested_locales: list[str]) -> dict[str, Any]:
|
||||
"""Map the Fast Transcription response onto the transcriber output schema."""
|
||||
segments: list[dict[str, Any]] = []
|
||||
word_timestamps: list[dict[str, Any]] = []
|
||||
detected_locale: str | None = None
|
||||
|
||||
for idx, phrase in enumerate(payload.get("phrases", [])):
|
||||
offset_ms = phrase.get("offsetMilliseconds", 0)
|
||||
dur_ms = phrase.get("durationMilliseconds", 0)
|
||||
confidence = phrase.get("confidence")
|
||||
if detected_locale is None and phrase.get("locale"):
|
||||
detected_locale = phrase["locale"]
|
||||
|
||||
seg: dict[str, Any] = {
|
||||
"id": idx,
|
||||
"start": round(offset_ms / 1000.0, 3),
|
||||
"end": round((offset_ms + dur_ms) / 1000.0, 3),
|
||||
"text": (phrase.get("text") or "").strip(),
|
||||
}
|
||||
if "speaker" in phrase:
|
||||
seg["speaker"] = phrase["speaker"]
|
||||
|
||||
words = phrase.get("words") or []
|
||||
if words:
|
||||
seg_words = []
|
||||
for w in words:
|
||||
w_offset = w.get("offsetMilliseconds", 0)
|
||||
w_dur = w.get("durationMilliseconds", 0)
|
||||
word_entry = {
|
||||
"word": w.get("text", ""),
|
||||
"start": round(w_offset / 1000.0, 3),
|
||||
"end": round((w_offset + w_dur) / 1000.0, 3),
|
||||
# Fast Transcription has no per-word confidence; carry the
|
||||
# phrase confidence so downstream schemas stay populated.
|
||||
"probability": round(confidence, 3) if confidence is not None else None,
|
||||
}
|
||||
seg_words.append(word_entry)
|
||||
word_timestamps.append(word_entry)
|
||||
seg["words"] = seg_words
|
||||
|
||||
segments.append(seg)
|
||||
|
||||
duration_ms = payload.get("durationMilliseconds")
|
||||
if duration_ms is None and segments:
|
||||
duration_ms = max(int(s["end"] * 1000) for s in segments)
|
||||
|
||||
return {
|
||||
"segments": segments,
|
||||
"word_timestamps": word_timestamps,
|
||||
"language": detected_locale or (requested_locales[0] if requested_locales else None),
|
||||
"duration_seconds": round((duration_ms or 0) / 1000.0, 3),
|
||||
}
|
||||
Reference in New Issue
Block a user