mirror of
https://github.com/calesthio/OpenMontage.git
synced 2026-08-05 15:20:40 +08:00
video-compose: catch TTS punctuation leaks in final review
Adds a transcript_comparison check to VideoCompose._run_final_review that word-diffs the whisper/whisperx transcript against script.txt and fails loudly when the TTS engine literally voiced punctuation tokens (dot, dots, ellipsis, comma, dash, hyphen, period). Chirp3-HD did this to ellipses in a production run and it slipped past review; now it cannot. Also corrects the "tiny background video" gotcha in skills/core/hyperframes.md. After six renders of blaming HyperFrames CSS, the real root cause was 640x360 Pexels sources combined with a fit-and-pad pre-transform — HyperFrames was rendering the letterboxed input faithfully. Gotcha now walks through the ffprobe diagnostic and the scale-to-cover fix, and keeps the wrapper-div pattern for the right reasons (aspect mismatch handling, not framework bug workaround). Four regression tests cover the new check: punctuation-leak detection, clean-audio false-positive guard, graceful skip when inputs missing, and always-present transcript_comparison section.
This commit is contained in:
calesthio
@@ -288,54 +288,76 @@ Same pattern as Remotion.
|
||||
Things the upstream docs don't warn about but will cost you a 60-minute
|
||||
render to discover. Fix at author time, not at render time.
|
||||
|
||||
### Video elements need explicit size, in BOTH HTML attrs and `!important` CSS
|
||||
### Full-frame background video: source-resolution trap (NOT a framework bug)
|
||||
|
||||
HyperFrames runtime applies inline `style="width:...px; height:...px"` to
|
||||
every `<video>` element based on the video's **intrinsic** dimensions.
|
||||
That inline style beats any class-selector CSS you write, so even a
|
||||
`video.bg-video { width: 100%; height: 100% }` rule gets silently ignored
|
||||
and the video renders at whatever size the decoded stream reports —
|
||||
usually a small centered box inside an otherwise-black clip.
|
||||
Symptom: background video renders as a small centered box with black
|
||||
around it, even though you've told HyperFrames the clip is 1920×1080.
|
||||
Six renders of debugging revealed this is **almost always a source
|
||||
quality issue, not a HyperFrames framework bug.** Fix the input, not
|
||||
the CSS.
|
||||
|
||||
The bug is invisible in some scenarios:
|
||||
- Low `filter: brightness(0.25-0.35)` makes the small video box look
|
||||
mostly black.
|
||||
- Fullscreen typography layered on top hides the problem.
|
||||
- `object-fit: cover` on a class selector does NOT fix it because the
|
||||
inline width/height wins first.
|
||||
Root cause (observed on Pexels + Pixabay stock): many free stock
|
||||
clips ship at 640×360 or 960×540 even when you ask for the "large"
|
||||
size tier. If your pre-transform does
|
||||
`scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:...`
|
||||
you'll produce a 1920×1080 file whose visible content is a centered
|
||||
640×360 rectangle with black padding. HyperFrames plays this clip
|
||||
faithfully — the small-video-in-black-frame you see is a correctly
|
||||
rendered letterboxed input.
|
||||
|
||||
It becomes obvious the moment you go footage-forward (brightness > 0.5,
|
||||
lower-third typography) — the small centered video box appears against
|
||||
a black frame that was supposed to be full-bleed b-roll.
|
||||
Diagnostic first: before blaming CSS, run
|
||||
`ffprobe -v error -select_streams v -show_entries stream=width,height`
|
||||
on the source clip and on your pre-transformed clip. If the pre-
|
||||
transform already shows a 640×360 active region inside 1920×1080
|
||||
padding, the problem is upstream.
|
||||
|
||||
**Correct pattern** for a full-frame background video:
|
||||
Fix the pre-transform to scale-to-COVER and crop, not scale-to-FIT
|
||||
and pad:
|
||||
|
||||
```
|
||||
-vf "scale=1920:1080:force_original_aspect_ratio=increase,\
|
||||
crop=1920:1080,unsharp=3:3:0.6"
|
||||
```
|
||||
|
||||
This upscales the small clip to at least cover 1920×1080, crops the
|
||||
overflow, and sharpens (unsharp) to offset perceived softness. The
|
||||
output is full-bleed and HyperFrames will render it edge-to-edge.
|
||||
|
||||
The canonical wrapper-div pattern (`patterns.md`) is still the right
|
||||
HTML shape for backgrounds:
|
||||
|
||||
```html
|
||||
<video
|
||||
class="clip bg-video"
|
||||
data-start="0" data-duration="8" data-track-index="0"
|
||||
src="assets/hero.mp4"
|
||||
muted playsinline preload="metadata"
|
||||
width="1920" height="1080"
|
||||
></video>
|
||||
<div style="position:absolute;top:0;left:0;width:1920px;height:1080px;overflow:hidden;">
|
||||
<video
|
||||
data-start="0" data-duration="60" data-track-index="0"
|
||||
src="..." muted playsinline
|
||||
style="width:100%;height:100%;object-fit:cover;"
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
```css
|
||||
video.bg-video {
|
||||
position: absolute !important; inset: 0 !important;
|
||||
width: 100% !important; height: 100% !important;
|
||||
object-fit: cover !important;
|
||||
/* filter optional */
|
||||
}
|
||||
```
|
||||
Use it because `overflow: hidden` on the wrapper crops gracefully when
|
||||
the video's aspect differs slightly from 16:9, and because
|
||||
`object-fit: cover` on the inner `<video>` handles the (rare) case
|
||||
where source aspect isn't 16:9. It is **not** a workaround for a
|
||||
framework layout bug — the framework will size `<video class="clip">`
|
||||
correctly too if the source file has content filling the full frame.
|
||||
|
||||
Both the explicit `width="1920" height="1080"` HTML attrs AND the
|
||||
`!important` CSS are required. Either one alone isn't enough.
|
||||
Apply visual treatments (`filter`, `border-radius`, etc.) on the
|
||||
wrapper or on a scoped selector like `.bg-slot video { filter: ... }`.
|
||||
|
||||
Alternative: wrap the `<video>` in a `<div class="clip">` and put
|
||||
`data-start`/`data-duration` on the wrapper. Let the video fill its div
|
||||
via `position: absolute; inset: 0` without the framework's inline-style
|
||||
interference. Slightly more DOM but more robust.
|
||||
Invisible-vs-obvious failure mode: this trap hides when you stack
|
||||
dim filters (`brightness(0.25–0.35)`) or full-bleed typography on top —
|
||||
the letterbox reads as "moody darkness." It becomes obvious the moment
|
||||
you go footage-forward (brightness > 0.5, lower-third typography). If
|
||||
you're planning footage-forward from the start, probe your source
|
||||
resolutions in the asset stage.
|
||||
|
||||
When you truly can't source HD clips, switch pipelines: use FFmpeg
|
||||
hybrid-compositing (scale-to-cover + crop + unsharp in an external
|
||||
b-roll reel, then overlay HyperFrames typography via chromakey). That's
|
||||
the pattern `projects/quantum-willow-multiverse/` settled on after six
|
||||
HyperFrames renders with 640×360 Pexels sources.
|
||||
|
||||
### Always preview-scrub footage-forward scenes before render
|
||||
|
||||
|
||||
@@ -627,6 +627,171 @@ def test_decision_log_accepts_render_runtime_selection_with_both_options():
|
||||
jsonschema.validate(log, schema)
|
||||
|
||||
|
||||
def test_transcript_comparison_catches_literal_punctuation_leak(tmp_path):
|
||||
"""Regression: Chirp3-HD (and some other TTS engines) read literal `...`
|
||||
as the word 'dot' in audio output. This failure is invisible to
|
||||
volume-based audio spotchecks but ships audio that literally says
|
||||
'dot dot dot' twelve times. The transcript_comparison check catches
|
||||
this automatically before the video is marked pass."""
|
||||
import json
|
||||
|
||||
# Real-world example: user's script had `...` everywhere for dramatic
|
||||
# pause, Chirp read them all as "dot", transcript contains "dot dot dot"
|
||||
# phrases the script never had.
|
||||
script_text = (
|
||||
"A computer just did in five minutes what would take every machine on Earth, "
|
||||
"running since the Big Bang, ten septillion years to finish. "
|
||||
"We may have gotten help from parallel universes."
|
||||
)
|
||||
transcript_data = {
|
||||
"word_timestamps": [
|
||||
{"word": "A", "start": 0.0, "end": 0.1},
|
||||
{"word": "computer", "start": 0.1, "end": 0.5},
|
||||
{"word": "just", "start": 0.5, "end": 0.7},
|
||||
{"word": "did", "start": 0.7, "end": 0.9},
|
||||
{"word": "in", "start": 0.9, "end": 1.0},
|
||||
{"word": "five", "start": 1.0, "end": 1.3},
|
||||
{"word": "minutes", "start": 1.3, "end": 1.7},
|
||||
{"word": "dot", "start": 1.7, "end": 1.9}, # leak!
|
||||
{"word": "dot", "start": 1.9, "end": 2.1}, # leak!
|
||||
{"word": "dot", "start": 2.1, "end": 2.3}, # leak!
|
||||
{"word": "what", "start": 2.5, "end": 2.8},
|
||||
{"word": "would", "start": 2.8, "end": 3.0},
|
||||
{"word": "take", "start": 3.0, "end": 3.3},
|
||||
{"word": "every", "start": 3.3, "end": 3.6},
|
||||
{"word": "machine", "start": 3.6, "end": 4.0},
|
||||
{"word": "on", "start": 4.0, "end": 4.2},
|
||||
{"word": "Earth", "start": 4.2, "end": 4.6},
|
||||
{"word": "running", "start": 4.7, "end": 5.1},
|
||||
{"word": "since", "start": 5.1, "end": 5.4},
|
||||
{"word": "the", "start": 5.4, "end": 5.5},
|
||||
{"word": "Big", "start": 5.5, "end": 5.8},
|
||||
{"word": "Bang", "start": 5.8, "end": 6.2},
|
||||
{"word": "ten", "start": 6.2, "end": 6.5},
|
||||
{"word": "septillion", "start": 6.5, "end": 7.3},
|
||||
{"word": "years", "start": 7.3, "end": 7.7},
|
||||
{"word": "to", "start": 7.7, "end": 7.9},
|
||||
{"word": "finish", "start": 7.9, "end": 8.3},
|
||||
{"word": "dot", "start": 8.3, "end": 8.5}, # another leak
|
||||
{"word": "We", "start": 9.0, "end": 9.2},
|
||||
{"word": "may", "start": 9.2, "end": 9.4},
|
||||
{"word": "have", "start": 9.4, "end": 9.6},
|
||||
{"word": "gotten", "start": 9.6, "end": 9.9},
|
||||
{"word": "help", "start": 9.9, "end": 10.3},
|
||||
{"word": "from", "start": 10.3, "end": 10.5},
|
||||
{"word": "parallel", "start": 10.5, "end": 11.0},
|
||||
{"word": "universes", "start": 11.0, "end": 11.7},
|
||||
]
|
||||
}
|
||||
transcript_path = tmp_path / "transcript.json"
|
||||
transcript_path.write_text(json.dumps(transcript_data), encoding="utf-8")
|
||||
|
||||
result = VideoCompose._compare_transcript_to_script(transcript_path, script_text)
|
||||
|
||||
# Must catch the punctuation leak
|
||||
assert result["spurious_punctuation_words"], (
|
||||
"transcript_comparison failed to detect the 'dot' leak from literal ... punctuation."
|
||||
)
|
||||
leak_counts = {
|
||||
entry["word"]: entry["count"]
|
||||
for entry in result["spurious_punctuation_words"]
|
||||
}
|
||||
assert leak_counts.get("dot") == 4, f"Expected 4 'dot' leaks, got {leak_counts}"
|
||||
|
||||
# Must produce a CRITICAL-severity issue message
|
||||
issue_text = " ".join(result["issues"]).lower()
|
||||
assert "tts punctuation leak" in issue_text
|
||||
assert "not in the script" in issue_text
|
||||
|
||||
# Must NOT mark the transcript as matching
|
||||
assert result["transcript_matches_script"] is False
|
||||
|
||||
|
||||
def test_transcript_comparison_passes_clean_audio(tmp_path):
|
||||
"""Clean audio with no punctuation leaks must NOT trigger a false
|
||||
positive."""
|
||||
import json
|
||||
|
||||
script_text = "The quick brown fox jumps over the lazy dog."
|
||||
transcript_data = {
|
||||
"word_timestamps": [
|
||||
{"word": "The", "start": 0.0, "end": 0.1},
|
||||
{"word": "quick", "start": 0.1, "end": 0.4},
|
||||
{"word": "brown", "start": 0.4, "end": 0.7},
|
||||
{"word": "fox", "start": 0.7, "end": 1.0},
|
||||
{"word": "jumps", "start": 1.0, "end": 1.3},
|
||||
{"word": "over", "start": 1.3, "end": 1.6},
|
||||
{"word": "the", "start": 1.6, "end": 1.7},
|
||||
{"word": "lazy", "start": 1.7, "end": 2.0},
|
||||
{"word": "dog", "start": 2.0, "end": 2.3},
|
||||
]
|
||||
}
|
||||
transcript_path = tmp_path / "transcript.json"
|
||||
transcript_path.write_text(json.dumps(transcript_data), encoding="utf-8")
|
||||
|
||||
result = VideoCompose._compare_transcript_to_script(transcript_path, script_text)
|
||||
assert result["spurious_punctuation_words"] == []
|
||||
assert result["transcript_matches_script"] is True
|
||||
assert result["word_accuracy"] >= 0.9
|
||||
# issues may still have informational content but no CRITICAL TTS leak
|
||||
assert not any("tts punctuation leak" in i.lower() for i in result["issues"])
|
||||
|
||||
|
||||
def test_transcript_comparison_graceful_when_inputs_missing(tmp_path):
|
||||
"""When transcript or script is unavailable, the check should NOT
|
||||
crash — it should record the skip in issues so the silence is visible."""
|
||||
# No transcript
|
||||
result = VideoCompose._compare_transcript_to_script(None, "some script text")
|
||||
assert any("not provided" in i for i in result["issues"])
|
||||
|
||||
# No script
|
||||
dummy = tmp_path / "t.json"
|
||||
dummy.write_text('{"word_timestamps": []}', encoding="utf-8")
|
||||
result = VideoCompose._compare_transcript_to_script(dummy, "")
|
||||
assert any("not provided" in i for i in result["issues"])
|
||||
|
||||
# Transcript file missing
|
||||
result = VideoCompose._compare_transcript_to_script(tmp_path / "nonexistent.json", "script")
|
||||
assert any("not provided" in i for i in result["issues"])
|
||||
|
||||
|
||||
def test_run_final_review_includes_transcript_comparison_section(tmp_path):
|
||||
"""Regression: the `transcript_comparison` section must ALWAYS appear in
|
||||
the final_review output — even when the caller doesn't provide a
|
||||
transcript. A missing section = silent governance failure."""
|
||||
import subprocess
|
||||
|
||||
# Build a minimal MP4 so _run_final_review can probe it.
|
||||
mp4 = tmp_path / "out.mp4"
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-y",
|
||||
"-f", "lavfi", "-i", "color=c=#000000:s=320x240:d=2",
|
||||
"-f", "lavfi", "-i", "sine=frequency=440:duration=2",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p",
|
||||
"-c:a", "aac", "-shortest", str(mp4),
|
||||
],
|
||||
capture_output=True, check=True, timeout=30,
|
||||
)
|
||||
|
||||
review = VideoCompose()._run_final_review(
|
||||
mp4,
|
||||
edit_decisions={
|
||||
"version": "1.0",
|
||||
"renderer_family": "animation-first",
|
||||
"render_runtime": "hyperframes",
|
||||
"cuts": [{"id": "c1", "source": "x", "in_seconds": 0, "out_seconds": 2}],
|
||||
},
|
||||
)
|
||||
assert "transcript_comparison" in review["checks"], (
|
||||
"final_review must always include a transcript_comparison section. "
|
||||
"When the caller doesn't provide a transcript, the section should "
|
||||
"still appear with a 'skipped' issue entry — not be omitted."
|
||||
)
|
||||
tc = review["checks"]["transcript_comparison"]
|
||||
assert any("not provided" in i for i in tc["issues"])
|
||||
|
||||
|
||||
def test_hyperframes_root_composition_has_data_start_and_duration(tmp_path):
|
||||
"""Regression: the generated root composition was missing data-start
|
||||
and data-duration, violating the HyperFrames contract (SKILL.md table)."""
|
||||
|
||||
@@ -105,6 +105,32 @@ class VideoCompose(BaseTool):
|
||||
"edit_decisions.metadata.proposal_render_runtime."
|
||||
),
|
||||
},
|
||||
"narration_transcript_path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Path to a word-level transcript JSON (from `transcriber` "
|
||||
"tool output). Optional but STRONGLY recommended: when "
|
||||
"combined with script_path/script_text, final_review "
|
||||
"runs transcript_comparison and catches TTS failures "
|
||||
"like 'Chirp3-HD reads ... as the word dot'. Without "
|
||||
"it, content-level audio bugs ship silently."
|
||||
),
|
||||
},
|
||||
"script_path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Path to the source narration script (plain text). "
|
||||
"Used by transcript_comparison to diff against the "
|
||||
"transcribed audio. Provide this OR script_text."
|
||||
),
|
||||
},
|
||||
"script_text": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Inline source narration script. Used by "
|
||||
"transcript_comparison when a file path is unavailable."
|
||||
),
|
||||
},
|
||||
"subtitle_path": {"type": "string"},
|
||||
"subtitle_style": {
|
||||
"type": "object",
|
||||
@@ -1043,7 +1069,13 @@ class VideoCompose(BaseTool):
|
||||
# --- Post-render: mandatory final self-review ---
|
||||
if render_result.success and output_path.exists():
|
||||
final_review = self._run_final_review(
|
||||
output_path, edit_decisions, inputs.get("proposal_packet")
|
||||
output_path,
|
||||
edit_decisions,
|
||||
inputs.get("proposal_packet"),
|
||||
narration_transcript_path=inputs.get("narration_transcript_path"),
|
||||
script_text=inputs.get("script_text") or self._read_text_file(
|
||||
inputs.get("script_path")
|
||||
),
|
||||
)
|
||||
|
||||
# Attach final_review to the ToolResult data so the compose-director
|
||||
@@ -1160,7 +1192,13 @@ class VideoCompose(BaseTool):
|
||||
# Post-render: mandatory final self-review (identical contract to the Remotion path).
|
||||
if output_path.exists():
|
||||
final_review = self._run_final_review(
|
||||
output_path, edit_decisions, inputs.get("proposal_packet")
|
||||
output_path,
|
||||
edit_decisions,
|
||||
inputs.get("proposal_packet"),
|
||||
narration_transcript_path=inputs.get("narration_transcript_path"),
|
||||
script_text=inputs.get("script_text") or self._read_text_file(
|
||||
inputs.get("script_path")
|
||||
),
|
||||
)
|
||||
if render_result.data is None:
|
||||
render_result.data = {}
|
||||
@@ -1214,7 +1252,13 @@ class VideoCompose(BaseTool):
|
||||
|
||||
if render_result.success and output_path.exists():
|
||||
final_review = self._run_final_review(
|
||||
output_path, edit_decisions, inputs.get("proposal_packet")
|
||||
output_path,
|
||||
edit_decisions,
|
||||
inputs.get("proposal_packet"),
|
||||
narration_transcript_path=inputs.get("narration_transcript_path"),
|
||||
script_text=inputs.get("script_text") or self._read_text_file(
|
||||
inputs.get("script_path")
|
||||
),
|
||||
)
|
||||
if render_result.data is None:
|
||||
render_result.data = {}
|
||||
@@ -1353,11 +1397,156 @@ class VideoCompose(BaseTool):
|
||||
# Final self-review — mandatory post-render inspection
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
# Punctuation/SSML-leak words that should NEVER appear in rendered audio.
|
||||
# When a TTS engine reads a literal "..." as the word "dot", or a "—" as
|
||||
# "hyphen", those leak into the transcript. Catching these in the final
|
||||
# review is the difference between catching a bad voice render in-tool
|
||||
# vs. shipping a video that says "dot dot dot" twelve times. CRITICAL.
|
||||
_TTS_PUNCTUATION_LEAK_WORDS = {
|
||||
"dot", "dots", "ellipsis", "period", "periods",
|
||||
"comma", "commas", "semicolon", "colon",
|
||||
"dash", "hyphen", "emdash", "endash",
|
||||
"parenthesis", "bracket", "brace",
|
||||
"asterisk", "slash", "backslash",
|
||||
"exclamation", "question mark",
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _read_text_file(path: str | Path | None) -> str | None:
|
||||
"""Read a small text file if given a path; None-safe and exception-safe."""
|
||||
if not path:
|
||||
return None
|
||||
try:
|
||||
return Path(path).read_text(encoding="utf-8")
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
def _tokenize(cls, text: str) -> list[str]:
|
||||
"""Split text into comparable word tokens (lowercased, punctuation
|
||||
stripped, numeric-word-aware). Empty tokens dropped."""
|
||||
import re
|
||||
|
||||
# Preserve hyphenated words as single tokens ("many-worlds" -> "many-worlds").
|
||||
# Drop everything except letters, digits, hyphens, apostrophes.
|
||||
cleaned = re.sub(r"[^A-Za-z0-9\-' ]+", " ", text.lower())
|
||||
return [t for t in cleaned.split() if t and t != "-"]
|
||||
|
||||
@classmethod
|
||||
def _compare_transcript_to_script(
|
||||
cls,
|
||||
transcript_path: Path,
|
||||
script_text: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Compare a word-level transcript against the source script.
|
||||
|
||||
Purpose: catch TTS failures that look fine on audio-volume/duration
|
||||
checks but produce garbage content. The canonical example is
|
||||
Chirp3-HD reading ellipses ("...") literally as the word "dot" — our
|
||||
volume check says "narration present, not clipped" and the video
|
||||
ships. This check diffs the actual transcribed audio against what
|
||||
was supposed to be said, and flags:
|
||||
|
||||
- Spurious punctuation-leak words ("dot", "comma", "hyphen", etc.)
|
||||
that appear in audio but not script → CRITICAL
|
||||
- Overall word-accuracy ratio against script → SUGGESTION if < 0.9
|
||||
|
||||
Returns the transcript_comparison section of final_review, or a
|
||||
placeholder with an issue describing why the check couldn't run
|
||||
(missing transcript, missing script) so the review never goes
|
||||
silently quiet on this contract.
|
||||
"""
|
||||
result: dict[str, Any] = {
|
||||
"transcript_matches_script": False,
|
||||
"word_accuracy": None,
|
||||
"script_word_count": 0,
|
||||
"transcript_word_count": 0,
|
||||
"spurious_punctuation_words": [],
|
||||
"issues": [],
|
||||
}
|
||||
|
||||
if not transcript_path or not Path(transcript_path).is_file():
|
||||
result["issues"].append(
|
||||
"transcript_comparison skipped: narration_transcript not provided"
|
||||
)
|
||||
return result
|
||||
if not script_text:
|
||||
result["issues"].append(
|
||||
"transcript_comparison skipped: script_text not provided"
|
||||
)
|
||||
return result
|
||||
|
||||
try:
|
||||
transcript_data = json.loads(Path(transcript_path).read_text(encoding="utf-8"))
|
||||
except Exception as e:
|
||||
result["issues"].append(f"transcript_comparison could not parse transcript: {e}")
|
||||
return result
|
||||
|
||||
transcript_words = [
|
||||
w.get("word", "").strip() for w in transcript_data.get("word_timestamps", [])
|
||||
]
|
||||
transcript_tokens = cls._tokenize(" ".join(transcript_words))
|
||||
script_tokens = cls._tokenize(script_text)
|
||||
|
||||
result["script_word_count"] = len(script_tokens)
|
||||
result["transcript_word_count"] = len(transcript_tokens)
|
||||
|
||||
if not script_tokens or not transcript_tokens:
|
||||
result["issues"].append(
|
||||
f"transcript_comparison: empty token set "
|
||||
f"(script={len(script_tokens)}, transcript={len(transcript_tokens)})"
|
||||
)
|
||||
return result
|
||||
|
||||
# --- Punctuation-leak detection (TTS reading literal punctuation) ---
|
||||
script_set = set(script_tokens)
|
||||
leak_occurrences: dict[str, int] = {}
|
||||
for token in transcript_tokens:
|
||||
if token in cls._TTS_PUNCTUATION_LEAK_WORDS and token not in script_set:
|
||||
leak_occurrences[token] = leak_occurrences.get(token, 0) + 1
|
||||
|
||||
if leak_occurrences:
|
||||
formatted = ", ".join(
|
||||
f"{w!r}×{n}" for w, n in sorted(leak_occurrences.items(), key=lambda x: -x[1])
|
||||
)
|
||||
result["spurious_punctuation_words"] = [
|
||||
{"word": w, "count": n} for w, n in leak_occurrences.items()
|
||||
]
|
||||
result["issues"].append(
|
||||
f"TTS punctuation leak: transcript contains {formatted} — "
|
||||
f"these words are NOT in the script, which means the voice "
|
||||
f"engine is reading literal punctuation aloud. Rewrite the "
|
||||
f"script to eliminate the corresponding characters (ellipses, "
|
||||
f"em-dashes, etc.) and regenerate narration."
|
||||
)
|
||||
|
||||
# --- Word accuracy via set overlap (cheap & ordering-insensitive) ---
|
||||
# We don't penalize small word-order differences or minor TTS
|
||||
# hallucinations; we just want to know "did 90%+ of the script's
|
||||
# content make it into the audio." Using set overlap on the script
|
||||
# side is robust to transcription noise.
|
||||
matched = sum(1 for t in script_tokens if t in set(transcript_tokens))
|
||||
accuracy = matched / max(1, len(script_tokens))
|
||||
result["word_accuracy"] = round(accuracy, 3)
|
||||
result["transcript_matches_script"] = accuracy >= 0.9 and not leak_occurrences
|
||||
|
||||
if accuracy < 0.9:
|
||||
result["issues"].append(
|
||||
f"Low transcript-to-script match: only {accuracy:.0%} of script "
|
||||
f"words appear in the transcribed audio ({matched}/"
|
||||
f"{len(script_tokens)}). Narration may be truncated, mispronounced, "
|
||||
f"or the wrong script was used."
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def _run_final_review(
|
||||
self,
|
||||
output_path: Path,
|
||||
edit_decisions: dict[str, Any] | None = None,
|
||||
proposal_packet: dict[str, Any] | None = None,
|
||||
narration_transcript_path: str | Path | None = None,
|
||||
script_text: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Run post-render self-review and produce a final_review artifact.
|
||||
|
||||
@@ -1706,12 +1895,24 @@ class VideoCompose(BaseTool):
|
||||
|
||||
issues.extend(subtitle_check.get("issues", []))
|
||||
|
||||
# --- 6. Determine overall status ---
|
||||
# --- 6. Transcript-vs-script comparison ---
|
||||
# Catches content-level TTS failures (the classic "Chirp reads `...`
|
||||
# as the word 'dot'" trap) that volume-based audio checks miss.
|
||||
# Only runs when caller provides both the transcript and script; when
|
||||
# skipped, issues list records that so the silence is visible.
|
||||
transcript_comparison = self._compare_transcript_to_script(
|
||||
Path(narration_transcript_path) if narration_transcript_path else None,
|
||||
script_text,
|
||||
)
|
||||
issues.extend(transcript_comparison.get("issues", []))
|
||||
|
||||
# --- 7. Determine overall status ---
|
||||
critical_issues = [
|
||||
i for i in issues
|
||||
if any(kw in i.lower() for kw in [
|
||||
"silent downgrade", "delivery promise violation",
|
||||
"effectively silent", "ffprobe failed", "suspiciously short",
|
||||
"tts punctuation leak", # reading literal punctuation aloud
|
||||
])
|
||||
]
|
||||
|
||||
@@ -1739,6 +1940,7 @@ class VideoCompose(BaseTool):
|
||||
"audio_spotcheck": audio_spotcheck,
|
||||
"promise_preservation": promise_preservation,
|
||||
"subtitle_check": subtitle_check,
|
||||
"transcript_comparison": transcript_comparison,
|
||||
},
|
||||
"issues_found": issues,
|
||||
"recommended_action": recommended_action,
|
||||
|
||||
Reference in New Issue
Block a user