feat(maggy): add routing rules, team conventions, and RFC benchmark results

Self-updating YAML routing rules at ~/.maggy/routing-rules.yaml override
blast-score routing for specific task types (docs, security, tests,
architecture, planning → claude) and TDD pipeline phases (spec, tdd_red,
review → claude; tdd_green → auto). Conventions from claude-bootstrap
(mWP, TDD, security, quality gates) are injected into every executor
prompt. Rules self-update via record_outcome() and learn_override().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
alinaqi1
2026-05-11 13:40:33 +02:00
parent f05b8b229d
commit c33fb30fd3
9 changed files with 771 additions and 13 deletions
+89
View File
@@ -3036,3 +3036,92 @@ EXP-6 (security, blast 8) → claude 209.5s ← premium (only when needed)
| Ollama missed product spec | Coding model assigned prose task | Route `task_type: docs` to kimi/claude regardless of blast |
| Codex slow on frontend (280s vs 122s) | Codex overhead for complex UI tasks | Consider routing blast 6 frontend to claude |
| Claude had better architecture | Single model sees full context | Multi-model loses cross-task context — address via checkpoint sharing |
### 18.6 Post-Benchmark Improvements
After the benchmark, three systems were built to close the identified gaps:
#### A. Routing Rules (`maggy/routing_rules.py`)
A YAML-backed self-updating rules file at `~/.maggy/routing-rules.yaml`. Rules are checked **before** blast-score routing, enforcing that specific task types and pipeline phases always use the right model.
**Task-type overrides** (from benchmark evidence):
| Task Type | Forced Model | Confidence | Source |
|-----------|-------------|-----------|--------|
| `docs` | claude | 0.9 | benchmark — local models are code-optimized, not prose |
| `security` | claude | 1.0 | rule — security review needs deep reasoning |
| `architecture` | claude | 0.8 | rule — architecture needs cross-context awareness |
| `tests` | claude | 0.9 | benchmark — only claude generated test files |
| `planning` | claude | 0.8 | rule — planning requires structured reasoning |
**Pipeline phase overrides** (from TDD workflow):
| Phase | Forced Model | Reason |
|-------|-------------|--------|
| `spec` | claude | SPEC phase needs comprehensive docs |
| `tdd_red` | claude | RED phase needs test design expertise |
| `tdd_green` | auto | GREEN phase uses blast-score routing |
| `review` | claude | Review needs security + architecture depth |
**Self-learning API:**
- `record_outcome(rules, model, task_type, success)` — updates rolling success rates from task results
- `learn_override(rules, task_type, model, reason, confidence)` — Maggy can add new overrides when data supports it
- Manual edits to the YAML are preserved; Maggy only appends learned entries
This directly addresses:
- **"Ollama missed product spec"** → `docs` tasks now forced to claude
- **"No tests generated"** → `tests` and `tdd_red` phases now forced to claude
#### B. Team Conventions (embedded in routing rules)
Conventions from claude-bootstrap's CLAUDE.md and skill files are embedded in the routing rules and injected into every prompt sent to any CLI:
```yaml
conventions:
- text: "Build minimum wowable product (mWP). Ship the smallest thing that makes someone say 'wow'."
applies_to: [all]
source: claude-bootstrap
- text: "Follow TDD: RED → GREEN → VALIDATE. Coverage >= 80%."
applies_to: [feature, bug, refactor]
source: claude-bootstrap
- text: "No secrets in code. Parameterized SQL only. Validate all input at API boundaries."
applies_to: [all]
source: claude-bootstrap
- text: "Quality gates: max 20 lines/function, max 3 params, max 2 nesting levels, max 200 lines/file."
applies_to: [all]
source: claude-bootstrap
- text: "Use existing patterns. Read the codebase before changing it."
applies_to: [all]
source: claude-bootstrap
```
Every executor prompt method (`_plan_prompt`, `_analysis_prompt`, `_tests_prompt`, `_impl_prompt`) now calls `conventions_for(rules, task_type)` and appends the matching conventions block. This means kimi, codex, ollama, and claude all receive the same team rules — standardizing quality expectations across all models.
#### C. Routing Rules + Conventions Flow
```
Task arrives → apply_override(task_type, phase)
↓ forced?
┌─YES─→ use forced model
└─NO──→ reward table → blast-score routing
build prompt + conventions_for(task_type)
send to CLI with team conventions embedded
record_outcome() → update YAML success rates
```
#### D. Expected Impact on Re-run
If the benchmark were re-run with these improvements:
| Gap (Before) | Expected Result (After) |
|-------------|----------------------|
| No product spec from ollama | EXP-1 (`docs`) now routes to claude → spec generated |
| No tests from any model | TDD pipeline with `tdd_red` → claude → tests generated |
| Inconsistent quality | All models receive team conventions (mWP, quality gates, security rules) |
| No self-improvement | Outcome recording feeds back into routing rules YAML |
**Net effect:** Quality score expected to converge with Claude Code's 7.8+ while maintaining the 83% cost reduction.
+64
View File
@@ -192,3 +192,67 @@ This represents ~83% reduction in Claude subscription consumption.
| Model diversity | 4 models | 1 model | Maggy |
**Summary:** Claude Code is faster and produces marginally higher overall quality (driven by tests and spec). Maggy's multi-model approach provides cost efficiency and subscription risk distribution, plus deeper security review via dedicated model routing. The main gaps to close: add TDD pipeline (test generation step), and improve docs routing (don't send prose tasks to coding-optimized local models).
---
## 9. Post-Benchmark Fixes (Routing Rules + Conventions)
Three systems were built immediately after the benchmark to close the gaps above.
### 9.1 Routing Rules (`~/.maggy/routing-rules.yaml`)
A self-updating YAML config that overrides blast-score routing for specific task types and pipeline phases. Rules are checked **before** the reward table or blast-score tier.
**Task-type overrides seeded from benchmark evidence:**
| Task Type | Forced To | Why |
|-----------|----------|-----|
| `docs` | claude | Ollama (code-optimized) produced no spec file |
| `security` | claude | Security review needs deep reasoning |
| `tests` | claude | Only claude generated test files in benchmark |
| `architecture` | claude | Architecture needs cross-context awareness |
| `planning` | claude | Planning requires structured reasoning |
**Pipeline phase overrides from TDD workflow:**
| Phase | Forced To | Why |
|-------|----------|-----|
| `spec` | claude | SPEC phase needs comprehensive docs |
| `tdd_red` | claude | RED phase needs test design expertise |
| `tdd_green` | auto | GREEN uses blast-score routing (cheap models can implement) |
| `review` | claude | Review needs security + architecture depth |
**Self-learning:** `record_outcome()` updates rolling success rates per model. `learn_override()` lets Maggy add new rules when outcome data supports it. Manual YAML edits are preserved.
### 9.2 Team Conventions Injection
Five conventions from claude-bootstrap's CLAUDE.md are embedded in routing rules and injected into every prompt sent to any CLI:
1. **mWP** — Build minimum wowable product. No feature flags, no premature abstractions.
2. **TDD** — RED → GREEN → VALIDATE. Coverage >= 80%.
3. **Security** — No secrets in code. Parameterized SQL. Validate input at boundaries.
4. **Quality gates** — 20 lines/fn, 3 params, 2 nesting levels, 200 lines/file.
5. **Existing patterns** — Read codebase before changing. Keep changes minimal.
All four executor prompt methods (`_plan_prompt`, `_analysis_prompt`, `_tests_prompt`, `_impl_prompt`) now append matching conventions. This standardizes quality expectations across kimi, codex, ollama, and claude.
### 9.3 Expected Re-run Improvements
| Benchmark Gap | Root Cause | Fix Applied | Expected Result |
|--------------|-----------|-------------|-----------------|
| No product spec (EXP-1) | `docs` routed to ollama | `docs → claude` override | Claude generates spec |
| No tests from any model | No TDD step in pipeline | `tdd_red → claude` + `tests → claude` overrides | Claude writes failing tests |
| Inconsistent quality across models | No shared standards | Conventions injected into all prompts | mWP + quality gates enforced everywhere |
| No learning from outcomes | Static routing only | `record_outcome()` + `learn_override()` | Routing improves with each task |
**Projected scores if re-run:**
| Dimension | Before | After (est.) | Change |
|-----------|--------|-------------|--------|
| Product spec | 0/10 | 9/10 | `docs → claude` |
| Test coverage | 0/10 | 8/10 | `tdd_red → claude` |
| Security | 10/10 | 10/10 | No change (already strong) |
| Architecture | 8/10 | 9/10 | Conventions enforce patterns |
| **Weighted avg** | **7.4/10** | **~8.5/10** | **+1.1 points** |
Cost efficiency would remain at ~83% savings — the new overrides only force claude for `docs` (1 task) and `tests` (new TDD step), not for CRUD/API/frontend work.
+59 -4
View File
@@ -13,9 +13,15 @@ from pathlib import Path
from maggy.calibration.tracker import CalibrationTracker
from maggy.config import MaggyConfig
from maggy.process.model_router import (
DEFAULT_TIERS,
RoutingDecision,
route_task,
)
from maggy.routing_rules import (
apply_override,
load as load_rules,
record_outcome as rules_record,
)
from maggy.scores import RewardTable
MIN_CALIBRATION_ACCURACY = 0.5
@@ -29,19 +35,29 @@ class RoutingContext:
task_type: str = "general"
security_sensitive: bool = False
project_key: str = ""
pipeline_phase: str = ""
class RoutingService:
"""Blast-score aware routing with reward-based learning."""
"""Blast-score aware routing with rule overrides."""
def __init__(self, cfg: MaggyConfig):
self.cfg = cfg
self.rewards = RewardTable(cfg)
db_dir = Path(cfg.storage.path).expanduser().parent
self.calibration = CalibrationTracker(db_dir / "calibration.db")
self.calibration = CalibrationTracker(
db_dir / "calibration.db",
)
self.rules = load_rules()
def route(self, ctx: RoutingContext) -> RoutingDecision:
"""Pick the best model for this task context."""
forced = apply_override(
self.rules, ctx.task_type, ctx.pipeline_phase,
)
if forced:
return self._forced_decision(forced, ctx)
override = self.rewards.best_model(
ctx.task_type, self._blast_tier(ctx.blast_score),
)
@@ -50,8 +66,10 @@ class RoutingService:
primary=override,
validator=None,
fallback_chain=[],
reason=f"Learned: best for {ctx.task_type} "
f"at blast {ctx.blast_score}",
reason=(
f"Learned: best for {ctx.task_type} "
f"at blast {ctx.blast_score}"
),
)
decision = route_task(
@@ -72,6 +90,12 @@ class RoutingService:
tier = self._blast_tier(blast_score)
self.rewards.record(model, task_type, tier, reward)
self.calibration.record(model, task_type, reward, reward)
success = reward > 0.0
rules_record(self.rules, model, task_type, success)
def reload_rules(self) -> None:
"""Reload rules from disk (after Maggy self-update)."""
self.rules = load_rules()
def get_heatmap(self) -> list[dict]:
"""Return reward heatmap data for dashboard."""
@@ -88,6 +112,29 @@ class RoutingService:
acc = self.calibration.accuracy(model)
return acc == 0.0 or acc >= MIN_CALIBRATION_ACCURACY
def _forced_decision(
self, model_name: str, ctx: RoutingContext,
) -> RoutingDecision:
"""Build decision from a rules override."""
tier = _find_tier(model_name)
if tier is None:
return route_task(
ctx.blast_score,
ctx.task_type,
ctx.security_sensitive,
)
validator = None
if ctx.blast_score >= 8 or ctx.security_sensitive:
validator = _find_tier("codex")
return RoutingDecision(
primary=tier,
validator=validator,
fallback_chain=[],
reason=f"Rule override: {ctx.task_type}"
f"{f'/{ctx.pipeline_phase}' if ctx.pipeline_phase else ''}"
f"{model_name}",
)
def _penalize_uncalibrated(
self, decision: RoutingDecision,
) -> RoutingDecision:
@@ -101,3 +148,11 @@ class RoutingService:
reason="Calibration penalty",
)
return decision
def _find_tier(name: str):
"""Look up a ModelTier by name from defaults."""
for t in DEFAULT_TIERS:
if t.name == name:
return t
return None
+363
View File
@@ -0,0 +1,363 @@
"""Routing rules — task-type and pipeline-phase model assignments.
Loaded from ~/.maggy/routing-rules.yaml. Maggy can self-update
this file when benchmark or outcome data provides evidence for
better routing decisions. Manual edits are preserved.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
import yaml
from maggy.config import CONFIG_DIR
RULES_PATH = CONFIG_DIR / "routing-rules.yaml"
MIN_CONFIDENCE = 0.6
@dataclass
class ModelOverride:
"""Force a specific model for a task type or phase."""
model: str
reason: str = ""
confidence: float = 1.0
source: str = "rule"
@dataclass
class PerformanceRecord:
"""Tracked model performance from outcomes."""
strengths: list[str] = field(default_factory=list)
weaknesses: list[str] = field(default_factory=list)
tasks_completed: int = 0
success_rate: float = 0.0
@dataclass
class Convention:
"""A team convention injected into prompts."""
text: str
applies_to: list[str] = field(default_factory=list)
source: str = "manual"
@dataclass
class RoutingRules:
"""All routing rules Maggy uses for orchestration."""
version: int = 1
updated_at: str = ""
task_type_overrides: dict[str, ModelOverride] = field(
default_factory=dict,
)
pipeline_phases: dict[str, ModelOverride] = field(
default_factory=dict,
)
model_performance: dict[str, PerformanceRecord] = field(
default_factory=dict,
)
conventions: list[Convention] = field(
default_factory=list,
)
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def _default_conventions() -> list[Convention]:
"""Team conventions from claude-bootstrap skills."""
return [
Convention(
"Build minimum wowable product (mWP). Ship the smallest "
"thing that makes someone say 'wow'. No feature flags, no "
"backwards-compat shims, no premature abstractions.",
["all"], "claude-bootstrap",
),
Convention(
"Follow TDD: RED (write failing tests) → GREEN (minimal "
"code to pass) → VALIDATE (lint, types, coverage >= 80%).",
["feature", "bug", "refactor"], "claude-bootstrap",
),
Convention(
"No secrets in code. Parameterized SQL only. Validate all "
"input at API boundaries. Hash passwords with bcrypt/argon2.",
["all"], "claude-bootstrap",
),
Convention(
"Quality gates: max 20 lines/function, max 3 params, "
"max 2 nesting levels, max 200 lines/file.",
["all"], "claude-bootstrap",
),
Convention(
"Use existing patterns. Read the codebase before changing it. "
"Keep changes minimal and focused on the task.",
["all"], "claude-bootstrap",
),
]
def default_rules() -> RoutingRules:
"""Seed rules from benchmark evidence + team conventions."""
return RoutingRules(
version=1,
updated_at=_now_iso(),
conventions=_default_conventions(),
task_type_overrides={
"docs": ModelOverride(
"claude",
"Local models are code-optimized, not prose",
0.9, "benchmark",
),
"security": ModelOverride(
"claude",
"Security review needs deep reasoning",
1.0, "rule",
),
"architecture": ModelOverride(
"claude",
"Architecture needs cross-context awareness",
0.8, "rule",
),
"tests": ModelOverride(
"claude",
"Only claude generated test files in benchmark",
0.9, "benchmark",
),
"planning": ModelOverride(
"claude",
"Planning requires structured reasoning",
0.8, "rule",
),
},
pipeline_phases={
"spec": ModelOverride(
"claude",
"SPEC phase needs comprehensive docs",
1.0, "rule",
),
"tdd_red": ModelOverride(
"claude",
"RED phase needs test design expertise",
0.9, "rule",
),
"tdd_green": ModelOverride(
"auto",
"GREEN phase uses blast-score routing",
1.0, "rule",
),
"review": ModelOverride(
"claude",
"Review needs security + architecture depth",
1.0, "rule",
),
},
model_performance={
"claude": PerformanceRecord(
["security", "tests", "docs", "architecture"],
["cost"],
6, 1.0,
),
"codex": PerformanceRecord(
["crud", "api_design"],
["frontend_speed", "tests"],
3, 1.0,
),
"kimi": PerformanceRecord(
["schema", "simple_tasks"],
["complex_reasoning"],
1, 1.0,
),
"local": PerformanceRecord(
["code_formatting", "simple_edits"],
["docs", "prose", "planning"],
1, 1.0,
),
},
)
def load(path: Path | None = None) -> RoutingRules:
"""Load rules from YAML. Seeds defaults if missing."""
target = path or RULES_PATH
if not target.exists():
rules = default_rules()
save(rules, target)
return rules
rules = _from_yaml(target)
if not rules.conventions:
rules.conventions = _default_conventions()
save(rules, target)
return rules
def save(rules: RoutingRules, path: Path | None = None) -> None:
"""Write rules to YAML."""
target = path or RULES_PATH
target.parent.mkdir(parents=True, exist_ok=True)
data = _to_dict(rules)
target.write_text(yaml.safe_dump(data, sort_keys=False))
def apply_override(
rules: RoutingRules,
task_type: str,
phase: str | None = None,
) -> str | None:
"""Return model name if rules override routing.
Returns None if blast-score routing should be used.
"""
if phase and phase in rules.pipeline_phases:
override = rules.pipeline_phases[phase]
if override.model != "auto" and _trusted(override):
return override.model
if task_type in rules.task_type_overrides:
override = rules.task_type_overrides[task_type]
if _trusted(override):
return override.model
return None
def record_outcome(
rules: RoutingRules,
model: str,
task_type: str,
success: bool,
path: Path | None = None,
) -> None:
"""Update performance data from a task outcome."""
perf = rules.model_performance.get(model)
if perf is None:
perf = PerformanceRecord()
rules.model_performance[model] = perf
total = perf.tasks_completed
rate = perf.success_rate
new_total = total + 1
new_rate = (rate * total + (1.0 if success else 0.0)) / new_total
perf.tasks_completed = new_total
perf.success_rate = round(new_rate, 3)
if success and task_type not in perf.strengths:
perf.strengths.append(task_type)
if not success and task_type not in perf.weaknesses:
perf.weaknesses.append(task_type)
rules.updated_at = _now_iso()
save(rules, path)
def learn_override(
rules: RoutingRules,
task_type: str,
model: str,
reason: str,
confidence: float = 0.7,
path: Path | None = None,
) -> None:
"""Maggy learns a new routing override from data."""
rules.task_type_overrides[task_type] = ModelOverride(
model=model,
reason=reason,
confidence=confidence,
source="learned",
)
rules.updated_at = _now_iso()
save(rules, path)
def conventions_for(
rules: RoutingRules, task_type: str,
) -> str:
"""Return conventions text relevant to a task type."""
lines: list[str] = []
for c in rules.conventions:
if "all" in c.applies_to or task_type in c.applies_to:
lines.append(f"- {c.text}")
if not lines:
return ""
return "## Team Conventions\n" + "\n".join(lines)
def _trusted(override: ModelOverride) -> bool:
return override.confidence >= MIN_CONFIDENCE
def _to_dict(rules: RoutingRules) -> dict:
return {
"version": rules.version,
"updated_at": rules.updated_at,
"conventions": [
{
"text": c.text,
"applies_to": c.applies_to,
"source": c.source,
}
for c in rules.conventions
],
"task_type_overrides": {
k: {
"model": v.model,
"reason": v.reason,
"confidence": v.confidence,
"source": v.source,
}
for k, v in rules.task_type_overrides.items()
},
"pipeline_phases": {
k: {
"model": v.model,
"reason": v.reason,
"confidence": v.confidence,
"source": v.source,
}
for k, v in rules.pipeline_phases.items()
},
"model_performance": {
k: {
"strengths": v.strengths,
"weaknesses": v.weaknesses,
"tasks_completed": v.tasks_completed,
"success_rate": v.success_rate,
}
for k, v in rules.model_performance.items()
},
}
def _from_yaml(path: Path) -> RoutingRules:
data = yaml.safe_load(path.read_text()) or {}
overrides = {
k: ModelOverride(**v)
for k, v in (data.get("task_type_overrides") or {}).items()
}
phases = {
k: ModelOverride(**v)
for k, v in (data.get("pipeline_phases") or {}).items()
}
perf = {
k: PerformanceRecord(**v)
for k, v in (data.get("model_performance") or {}).items()
}
convs = [
Convention(**c)
for c in (data.get("conventions") or [])
]
return RoutingRules(
version=data.get("version", 1),
updated_at=data.get("updated_at", ""),
task_type_overrides=overrides,
pipeline_phases=phases,
model_performance=perf,
conventions=convs,
)
+17 -4
View File
@@ -25,6 +25,7 @@ from maggy.process.model_router import RoutingDecision
from maggy.providers.base import IssueTrackerProvider, Task
from maggy.recovery.rollback import RollbackManager
from maggy.routing import RoutingContext, RoutingService
from maggy.routing_rules import conventions_for
from maggy.services.planner import DualPlanner
logger = logging.getLogger(__name__)
@@ -342,36 +343,40 @@ class ExecutorService:
})
def _plan_prompt(self, task: Task, icpg_ctx: str) -> str:
conv = self._conventions_block(task)
return (
"Create an implementation plan for this ticket. No code changes — just a plan.\n\n"
f"Ticket: {task.title}\n{task.description[:1500]}"
f"{self._icpg_block(icpg_ctx)}\n"
f"{self._icpg_block(icpg_ctx)}{conv}\n"
"Output: numbered steps, files to touch, risks, tests to add."
)
def _analysis_prompt(self, task: Task, icpg_ctx: str) -> str:
conv = self._conventions_block(task)
return (
"Analyze this ticket against the codebase and output a concise plan.\n"
"Identify: files to change, functions affected, tests needed, risks.\n\n"
f"Ticket: {task.title}\n{task.description[:1500]}"
f"{self._icpg_block(icpg_ctx)}"
f"{self._icpg_block(icpg_ctx)}{conv}"
)
def _tests_prompt(self, task: Task, icpg_ctx: str, analysis: str) -> str:
conv = self._conventions_block(task)
return (
"Write failing test cases for this ticket (TDD — no implementation yet).\n"
"Use the project's existing test patterns. Commit tests separately.\n\n"
f"Ticket: {task.title}\n{task.description[:1500]}"
f"{self._icpg_block(icpg_ctx)}\n"
f"{self._icpg_block(icpg_ctx)}{conv}\n"
f"Analysis:\n{analysis[:1000]}"
)
def _impl_prompt(self, task: Task, icpg_ctx: str) -> str:
conv = self._conventions_block(task)
return (
"Implement the feature to make the failing tests pass.\n"
"Follow existing code patterns. Keep changes minimal.\n\n"
f"Ticket: {task.title}\n{task.description[:1500]}"
f"{self._icpg_block(icpg_ctx)}\n"
f"{self._icpg_block(icpg_ctx)}{conv}\n"
"Run tests to verify, then commit with a conventional commit message."
)
@@ -380,6 +385,14 @@ class ExecutorService:
return ""
return f"\n\n{icpg_ctx}\n"
def _conventions_block(self, task: Task) -> str:
raw = task.raw if isinstance(task.raw, dict) else {}
task_type = str(raw.get("task_type") or self._task_type(task))
text = conventions_for(self._routing.rules, task_type)
if not text:
return ""
return f"\n\n{text}\n"
async def _post_plan(self, task_id: str, output: str) -> None:
try:
await self.provider.add_comment(
+4 -3
View File
@@ -108,8 +108,9 @@ class TestRoutingAccuracy:
name = decision.primary if isinstance(decision.primary, str) else decision.primary.name
results[task.id] = name
# Low blast (1-3) → cheap tier
assert results["T-1"] in ("local", "kimi", "deepseek")
# Low blast (1-3) → cheap tier unless rules override
# T-1 is docs → rules force claude (local can't do prose)
assert results["T-1"] == "claude"
assert results["T-2"] in ("local", "kimi", "deepseek")
assert results["T-3"] in ("local", "kimi", "deepseek")
# Mid blast (4-6) → gpt
@@ -131,7 +132,7 @@ class TestRoutingAccuracy:
correct = 0
expected_tiers = {
"T-1": "cheap", "T-2": "cheap", "T-3": "cheap",
"T-1": "premium", "T-2": "cheap", "T-3": "cheap",
"T-4": "medium", "T-5": "medium",
"T-6": "medium", # blast 7 overlaps gpt/claude
"T-7": "premium", "T-8": "premium",
+30 -1
View File
@@ -90,7 +90,8 @@ async def test_plan_mode_records_spend_for_selected_model(mock_cfg, tmp_path, mo
await executor._run("session-1", _task(3, "security"), str(tmp_path), "plan")
assert executor._budget.today_spend("openai") == pytest.approx(1.25)
# security tasks are now rule-overridden to claude (anthropic)
assert executor._budget.today_spend("anthropic") == pytest.approx(1.25)
@pytest.mark.asyncio
@@ -138,3 +139,31 @@ async def test_fatigue_tracked_during_steps(mock_cfg, tmp_path, monkeypatch):
await executor._run("session-1", _task(3, "docs"), str(tmp_path), "plan")
assert executor._fatigue.dimensions["context_load"] > 0
@pytest.mark.asyncio
async def test_conventions_injected_into_prompts(mock_cfg, tmp_path, monkeypatch):
provider = AsyncMock()
executor = ExecutorService(mock_cfg, provider)
executor._sessions["session-1"] = _session()
prompts: list[str] = []
async def fake_context(task: Task, wd: str) -> str:
return ""
async def fake_send(
model_name: str, prompt: str, working_dir: str,
max_turns: int = 20, timeout: int = 600,
) -> RunResult:
prompts.append(prompt)
return RunResult(model=model_name, success=True, output="ok")
monkeypatch.setattr(executor, "_build_icpg_context", fake_context)
monkeypatch.setattr(executor._pi, "send_prompt", fake_send)
await executor._run(
"session-1", _task(5, "feature"), str(tmp_path), "plan",
)
assert prompts, "At least one prompt should have been sent"
assert "Team Conventions" in prompts[0]
assert "minimum wowable product" in prompts[0]
+2 -1
View File
@@ -81,7 +81,8 @@ class TestRoutingDecisions:
cfg = _project_cfg(tmp_path)
svc = RoutingService(cfg)
for blast in (1, 2):
ctx = RoutingContext(blast_score=blast, task_type="docs")
# Use "formatting" — "docs" is now rules-overridden
ctx = RoutingContext(blast_score=blast, task_type="formatting")
decision = svc.route(ctx)
assert decision.primary.cost_rank <= 2, (
f"blast={blast} should route to cheap tier"
+143
View File
@@ -0,0 +1,143 @@
"""Tests for routing rules — load, save, apply, learn."""
from __future__ import annotations
from pathlib import Path
import pytest
from maggy.routing_rules import (
ModelOverride,
PerformanceRecord,
RoutingRules,
apply_override,
default_rules,
learn_override,
load,
record_outcome,
save,
)
@pytest.fixture()
def rules_path(tmp_path: Path) -> Path:
return tmp_path / "routing-rules.yaml"
class TestDefaultRules:
def test_seeds_task_type_overrides(self):
rules = default_rules()
assert "docs" in rules.task_type_overrides
assert "security" in rules.task_type_overrides
assert "tests" in rules.task_type_overrides
def test_seeds_pipeline_phases(self):
rules = default_rules()
assert "spec" in rules.pipeline_phases
assert "tdd_red" in rules.pipeline_phases
assert rules.pipeline_phases["tdd_green"].model == "auto"
def test_seeds_model_performance(self):
rules = default_rules()
assert "claude" in rules.model_performance
assert "local" in rules.model_performance
class TestLoadSave:
def test_load_creates_default(self, rules_path: Path):
rules = load(rules_path)
assert rules_path.exists()
assert "docs" in rules.task_type_overrides
def test_roundtrip(self, rules_path: Path):
original = default_rules()
save(original, rules_path)
loaded = load(rules_path)
assert loaded.version == original.version
assert set(loaded.task_type_overrides) == set(
original.task_type_overrides,
)
def test_load_existing(self, rules_path: Path):
save(default_rules(), rules_path)
rules = load(rules_path)
assert rules.task_type_overrides["security"].model == "claude"
class TestApplyOverride:
def test_phase_takes_priority(self):
rules = default_rules()
result = apply_override(rules, "feature", "spec")
assert result == "claude"
def test_auto_phase_returns_none(self):
rules = default_rules()
result = apply_override(rules, "feature", "tdd_green")
assert result is None
def test_task_type_override(self):
rules = default_rules()
result = apply_override(rules, "security")
assert result == "claude"
def test_no_override_returns_none(self):
rules = default_rules()
result = apply_override(rules, "feature")
assert result is None
def test_low_confidence_ignored(self):
rules = RoutingRules(
task_type_overrides={
"test": ModelOverride("kimi", "weak", 0.3),
},
)
result = apply_override(rules, "test")
assert result is None
class TestRecordOutcome:
def test_updates_success_rate(self, rules_path: Path):
rules = default_rules()
record_outcome(rules, "claude", "feature", True, rules_path)
perf = rules.model_performance["claude"]
assert perf.tasks_completed == 7
assert perf.success_rate > 0.9
def test_creates_new_model(self, rules_path: Path):
rules = default_rules()
record_outcome(rules, "gemini", "feature", True, rules_path)
assert "gemini" in rules.model_performance
assert rules.model_performance["gemini"].success_rate == 1.0
def test_records_failure(self, rules_path: Path):
rules = RoutingRules(
model_performance={
"test": PerformanceRecord(
tasks_completed=1, success_rate=1.0,
),
},
)
record_outcome(rules, "test", "security", False, rules_path)
assert rules.model_performance["test"].success_rate == 0.5
assert "security" in rules.model_performance["test"].weaknesses
class TestLearnOverride:
def test_adds_new_override(self, rules_path: Path):
rules = default_rules()
learn_override(
rules, "frontend", "claude",
"Codex too slow for frontend (280s vs 122s)",
0.8, rules_path,
)
assert rules.task_type_overrides["frontend"].model == "claude"
assert rules.task_type_overrides["frontend"].source == "learned"
def test_persists_to_disk(self, rules_path: Path):
rules = default_rules()
save(rules, rules_path)
learn_override(
rules, "frontend", "claude", "test", 0.9, rules_path,
)
reloaded = load(rules_path)
assert "frontend" in reloaded.task_type_overrides