zlei6/OpenMontage
1
0
Fork 0
mirror of https://github.com/calesthio/OpenMontage.git synced 2026-08-05 15:20:40 +08:00
Code Issues Packages Projects Releases Wiki Activity

Remove internal capability audit document

Browse Source
This commit is contained in:
calesthio
2026-03-29 09:45:40 -07:00
parent 3edf87da85
commit 8d1920cf24
1 changed files with 0 additions and 402 deletions
Download Patch File Download Diff File Expand all files Collapse all files

402
docs/CAPABILITY_AUDIT.md
View File

@@ -1,402 +0,0 @@
# OpenMontage Capability Audit
> 2026-03-28 | Code-verified. Every finding traced to specific file + line.
**Core question:** When an agent discovers its capabilities via the registry and reads
the skills/manifests, does it get an accurate picture of what it can and can't do?
**Verdict:** No. The agent receives a mostly-accurate tool inventory but is actively
misled at the skill-to-tool boundary. Skills promise inputs tools don't accept, schemas
reject structures skills tell the agent to produce, and the discovery mechanism omits
the one thing the agent needs most: what inputs each tool takes.
---
## 1. Discovery: What the Agent Learns at Boot
The agent runs:
```python
from tools.tool_registry import registry
registry.discover()
envelope = registry.support_envelope()
```
**What it gets per tool** (via `BaseTool.get_info()`):
- Identity: name, version, tier, capability, provider, stability
- Status: available/unavailable/degraded, dependencies list
- Runtime: execution_mode, determinism, runtime type, resource_profile
- Decision hints: best_for, not_good_for, provider_matrix, fallback_tools
- Skill links: agent_skills, related_skills
- Side effects, resume_support, user_visible_verification
**What it does NOT get:**
| Missing Field | Why It Matters |
|---|---|
| `input_schema` | Agent can't know what parameters a tool accepts without reading source code |
| `output_schema` | Agent can't know what a tool returns |
| `artifact_schema` | Agent can't know what files a tool produces |
| `estimate_cost()` results | Agent can't compare costs without calling each tool individually |
**This is the root cause of most downstream issues.** The agent knows *what tools exist* but not *what they accept or produce*. It must rely on skills for that information — and skills get it wrong in several places.
### Fix
Add to `BaseTool.get_info()`:
```python
"input_schema": self.input_schema,
"output_schema": self.output_schema,
"artifact_schema": self.artifact_schema,
```
This single change would let the agent validate skill instructions against actual tool contracts at runtime.
---
## 2. Skills Tell the Agent to Pass Inputs Tools Don't Accept
These are cases where a stage-director skill instructs the agent to call a tool with
specific parameters, but the tool's `input_schema` and `execute()` don't read them.
The agent follows the skill, the extra fields are silently ignored, and the output
doesn't reflect the agent's intent.
### 2a. tts_selector: voice style, speaker directions, pronunciation
**Skill says** (`skills/pipelines/explainer/asset-director.md` lines 54-62):
> Apply speaker directions from the script (pace, emphasis, emotion).
> Apply the playbook's `audio.voice_style`.
> Include a pronunciation map in the TTS request for technical terms.
**Tool accepts** (`tools/audio/tts_selector.py` input_schema):
`text`, `preferred_provider`, `allowed_providers`, `output_path`
No fields for voice_style, speaker_directions, pace, emphasis, emotion, or pronunciation.
None of the concrete TTS tools (elevenlabs, openai, piper) accept these either — except
elevenlabs_tts which accepts `stability`, `similarity_boost`, `style` (numeric 0-1 floats),
not the semantic concepts the skill describes.
**Agent impact:** Agent is told to express creative direction through tool parameters that
don't exist. The narration will be generated with default voice settings regardless of what
the script specifies.
### 2b. image_selector: negative_prompt, consistency_anchors
**Skill says** (`skills/pipelines/explainer/asset-director.md` lines 68-73):
> Build the prompt: playbook.image_prompt_prefix + scene description + style cues.
> Add negative prompt from playbook.
> Include consistency anchors (same palette, same style across all images).
**Tool accepts** (`tools/graphics/image_selector.py` input_schema):
`prompt`, `preferred_provider`, `allowed_providers`, `output_path`
No `negative_prompt` or `consistency_anchors` field. The selector passes `prompt` through
to whichever provider it selects. Some downstream providers (flux_image) DO accept
`negative_prompt`, but the selector doesn't forward it.
**Agent impact:** Agent must flatten everything into a single `prompt` string. Negative
prompts and consistency anchors are lost unless the agent bypasses the selector and calls
flux_image directly — which contradicts the skill's guidance.
### 2c. video_compose "render" operation: options, output_profile
**Skill says** (`skills/pipelines/explainer/compose-director.md` lines 66-80):
```json
{
"operation": "render",
"edit_decisions": {...},
"asset_manifest": {...},
"output_profile": "youtube_landscape",
"options": {
"subtitle_burn": true,
"audio_normalize": true,
"two_pass_encode": true
}
}
```
**Tool accepts** (`tools/video/video_compose.py` input_schema):
`operation`, `input_path`, `output_path`, `edit_decisions`, `subtitle_path`,
`subtitle_style`, `overlays`, `audio_path`, `profile`, `codec`, `crf`, `preset`
- `asset_manifest`: NOT in input_schema but IS read by `_render()` (line 271). Works but undocumented.
- `output_profile`: Not accepted. The field is `profile` and only used by `_encode()`, not `_render()`.
- `options`: Not accepted at all. `subtitle_burn`, `audio_normalize`, `two_pass_encode` are silently dropped.
**Agent impact:** Agent follows the skill, passes these fields, they vanish. Output won't
match target platform. No subtitle burn, no normalization, no two-pass encode.
### 2d. audio_mixer: skill implies single call, tool requires multiple
**Skill says** (`skills/pipelines/explainer/compose-director.md` lines 87-95):
> Call the audio_mixer tool to:
> 1. Layer narration segments in order
> 2. Mix background music at playbook volume
> 3. Apply ducking (music dips during narration)
> 4. Normalize overall audio levels
**Tool accepts** (`tools/audio/audio_mixer.py`):
Three discrete operations: `mix`, `duck`, `extract`. Each is a separate `execute()` call.
There is no single-call "mix everything together" mode.
**Agent impact:** Agent attempts a single call with all requirements. The tool handles one
operation per call. Agent must orchestrate: mix narration tracks first, then duck with
music, then the compose step muxes audio into video. The skill doesn't explain this
multi-step choreography.
---
## 3. Skills Tell the Agent to Produce Artifacts Schemas Reject
These are cases where a skill shows the agent an example JSON structure for an artifact,
but the corresponding JSON Schema has `additionalProperties: false` and rejects the
skill's fields.
### 3a. edit_decisions: transforms and transitions
**Skill says** (`skills/pipelines/explainer/edit-director.md` lines 38-53):
```json
{
"id": "cut-1",
"source": "img-scene-1",
"in_seconds": 0,
"out_seconds": 10,
"layer": "primary",
"transform": {
"scale": 1.0,
"position": "center",
"animation": "ken-burns-slow-zoom"
},
"transition_in": "fade",
"transition_out": "dissolve",
"transition_duration": 0.4
}
```
**Schema allows** (`schemas/artifacts/edit_decisions.schema.json` lines 12-24):
```json
{
"required": ["id", "source", "in_seconds", "out_seconds"],
"properties": {
"id": {}, "source": {}, "in_seconds": {}, "out_seconds": {},
"speed": {}, "reason": {}
},
"additionalProperties": false
}
```
**Rejected fields:** `layer`, `transform` (and all sub-fields), `transition_in`,
`transition_out`, `transition_duration`.
**Agent impact:** Agent builds the edit_decisions artifact per skill instructions.
Checkpoint validation rejects it. Agent is stuck between following the skill and
passing schema validation. The schema also means `video_compose._compose()` will
never receive transition/transform data even if it could handle it — which it can't
(it only does hard cuts via concat).
### 3b. edit_decisions: subtitle styling
**Skill says** (edit-director.md lines 62-78):
```json
"subtitles": {
"enabled": true,
"style": "word-by-word",
"font": "Inter",
"font_size": 48,
"color": "#FFFFFF",
"background": "#00000088",
"position": "bottom-center",
"max_words_per_line": 8
}
```
**Schema allows** (edit_decisions.schema.json lines 51-58):
```json
"subtitles": {
"properties": {
"enabled": { "type": "boolean" },
"style": { "type": "string" },
"source": { "type": "string" }
}
}
```
**Rejected fields:** `font`, `font_size`, `color`, `background`, `position`,
`max_words_per_line`.
**Agent impact:** All subtitle styling information is lost at schema validation.
The compose stage gets `enabled: true` and `style: "word-by-word"` but no font,
color, or positioning data. Subtitle styling must come from video_compose's
`subtitle_style` input instead — but the edit-director skill doesn't mention this path.
### 3c. edit_decisions: narration and SFX audio configuration
**Skill says** (edit-director.md lines 84-110):
```json
"audio": {
"narration": {
"segments": [
{ "asset_id": "narration-s1", "start_seconds": 0 }
]
},
"music": {
"asset_id": "...",
"ducking": {
"enabled": true,
"threshold_db": -3,
"reduction_db": -8,
"attack_ms": 200,
"release_ms": 500
}
},
"sfx": []
}
```
**Schema allows** (edit_decisions.schema.json lines 59-67):
`music` with `asset_id`, `volume`, `ducking` (boolean only), `fade_in_seconds`, `fade_out_seconds`.
**Rejected:** Entire `audio.narration` section, entire `audio.sfx` section, structured
`ducking` object (schema only accepts boolean). Narration segment timing and SFX
have no home in the validated artifact.
---
## 4. Tools Lie About What They Can Do
### 4a. video_compose._render() claims image-to-video conversion
**Docstring** (`tools/video/video_compose.py` line 267):
> "It orchestrates: image-to-video conversion, concatenation, audio mixing,
> subtitle burn-in, and final encoding to target profile."
**Actual code** (lines 280-302): Resolves asset IDs from asset_manifest to file paths,
then calls `_compose()`. No image-to-video conversion. `_compose()` uses `ffmpeg -ss/-to`
on each source, which fails on still images (PNG/JPG have no temporal dimension to seek into).
**Agent impact:** Agent trusts that the "render" operation handles images. It doesn't.
Explainer pipelines where most cuts are generated images will produce broken output.
### 4b. subtitle_gen declares highlight_style but ignores it
**Input schema** (`tools/subtitle/subtitle_gen.py`): includes `highlight_style` parameter.
**Execute method:** Never reads `highlight_style`. It's a phantom parameter.
**Agent impact:** Low — agent might pass highlight_style expecting highlighted captions
and get plain ones.
### 4c. video_selector declares operation parameter but ignores it
**Input schema** (`tools/video/video_selector.py`): includes `operation` field.
**Execute method:** Reads it but never passes it to the selected provider tool.
**Agent impact:** Low — operation context is lost during delegation but most video
providers only do one thing anyway.
---
## 5. Phantom Tools in Pipeline Manifests
`pipeline_defs/framework-smoke.yaml` references three tools that don't exist:
| Tool Name | Stage | Field |
|---|---|---|
| `idea_explorer_llm` | research | preferred_tools |
| `script_writer_llm` | script | preferred_tools |
| `script_writer_template` | script | fallback_tools |
`registry.discover()` will not find these. If the agent tries to resolve them, it gets
nothing. These appear to be placeholders from an earlier design where LLM capabilities
were modeled as tools.
**Agent impact:** framework-smoke pipeline is broken. Other 10 pipelines are clean —
all their tool references resolve to real tools.
---
## 6. Unused Tools the Agent Doesn't Know to Use
Five tools exist in `tools/` but no pipeline manifest references them:
| Tool | What It Does | Why It's Missing |
|---|---|---|
| `bg_remove` | Background removal (rembg/U2Net) | Useful for compositing, overlays. No skill mentions it. |
| `face_restore` | Face restoration (CodeFormer/GFPGAN) | Useful for low-res faces in talking-head. No skill mentions it. |
| `upscale` | Image/video upscaling (Real-ESRGAN) | Useful for enhancement pass. No skill mentions it. |
| `video_understand` | Vision-language analysis (CLIP/BLIP-2) | Could be used for quality review. No skill mentions it. |
| `image_gen` | Legacy multi-provider image gen | Deprecated, replaced by image_selector. Expected. |
The agent discovers these via `support_envelope()` but has no skill guidance on when or
how to use them. They're invisible capabilities — the agent CAN use them but WON'T
unless it independently decides to.
---
## 7. The Playbook Gap: Declared Knowledge the Agent Can't Apply
Style playbooks (`styles/*.yaml`) define detailed visual contracts:
`image_prompt_prefix`, `negative_prompt`, `consistency_anchors`, `typography`,
`motion.transition_duration_seconds`, `audio.music_mood`, `quality_rules`.
Skills extensively reference these: "apply the playbook's voice_style", "use playbook's
image_prompt_prefix", "set music volume per playbook". **30+ skill references across
all pipelines.**
But:
- Zero tools import `playbook_loader` or read playbook data
- The selector tools (tts_selector, image_selector) have no playbook-aware parameters
- The agent must: load the YAML, extract values, manually inject them into tool calls
This works IF the agent knows to do it. But the tools give no signal that they expect
playbook-derived values. The playbook system is a skill-layer convention that the tool
layer is unaware of. This is architecturally fine for an agent-first system — but the
skills need to be explicit about HOW the agent should bridge this (e.g., "read
`playbook.asset_generation.image_prompt_prefix` and prepend it to the `prompt` field
when calling image_selector"). Currently skills say WHAT to apply but not HOW to
translate playbook values into tool parameters.
---
## 8. Missing Skill: Talking-Head Executive Producer
`pipeline_defs/talking-head.yaml` declares `orchestration.skill: pipelines/talking-head/executive-producer` but the file `skills/pipelines/talking-head/executive-producer.md` does not exist. All other 9 pipelines with orchestration have their EP skill.
**Agent impact:** Agent following the talking-head pipeline reads the manifest, tries to
load the EP skill, gets a file-not-found. Must improvise orchestration without guidance.
---
## Summary: What the Agent Gets Right vs. Wrong
### Accurate (agent can trust these)
- Tool inventory: 44/47 tool references in manifests resolve correctly (3 phantoms in framework-smoke only)
- Tool status: `get_status()` accurately reports availability based on real dependency checks
- Fallback chains: All `fallback_tools` references point to tools that exist
- Tool self-description: 10/12 tools audited have accurate input_schema vs execute() behavior
- Pipeline stage ordering and artifact flow: checkpoint system matches manifests
- Cost estimates: Tools that implement `estimate_cost()` return realistic numbers
### Inaccurate (agent will be misled by these)
| Issue | Count | Root Cause |
|---|---|---|
| Skills pass inputs tools don't accept | 4 cases | Skills written to aspirational API, not actual tool schema |
| Skills produce artifacts schemas reject | 3 cases | Schemas stricter than skills expect (additionalProperties: false) |
| Tool overclaims capability | 1 case | video_compose._render() docstring lies about image handling |
| Discovery omits input/output schemas | All tools | get_info() doesn't include schemas |
| Phantom tools in manifests | 3 tools | framework-smoke has placeholder references |
| Missing EP skill | 1 pipeline | talking-head pipeline incomplete |
### Recommended Fix Priority
| # | Fix | Effort | Effect |
|---|---|---|---|
| 1 | Add input_schema/output_schema to `get_info()` | 3 lines | Agent can self-validate against skills |
| 2 | Align edit_decisions.schema.json with edit-director skills | Medium | Unblocks transitions, subtitle styling, narration config |
| 3 | Add image-to-video loop in video_compose._compose() | Small | Unblocks all image-based pipelines |
| 4 | Add negative_prompt passthrough to image_selector | Small | Enables playbook integration for image gen |
| 5 | Add voice_id/style passthrough to tts_selector | Small | Enables creative direction for narration |
| 6 | Fix compose-director to match video_compose's actual interface | Small | Agent stops passing phantom options |
| 7 | Fix compose-director to document multi-call audio_mixer flow | Small | Agent knows to call mix then duck |
| 8 | Remove phantom tools from framework-smoke.yaml | Trivial | Clean manifest |
| 9 | Create talking-head executive-producer.md | Medium | Completes pipeline |
| 10 | Document playbook-to-tool bridging pattern in skills | Medium | Agent knows HOW to apply playbook values |