mirror of
https://github.com/alinaqi/maggy.git
synced 2026-09-14 13:55:36 +08:00
feat(maggy): add routing rules, team conventions, and RFC benchmark results
Self-updating YAML routing rules at ~/.maggy/routing-rules.yaml override blast-score routing for specific task types (docs, security, tests, architecture, planning → claude) and TDD pipeline phases (spec, tdd_red, review → claude; tdd_green → auto). Conventions from claude-bootstrap (mWP, TDD, security, quality gates) are injected into every executor prompt. Rules self-update via record_outcome() and learn_override(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -3036,3 +3036,92 @@ EXP-6 (security, blast 8) → claude 209.5s ← premium (only when needed)
|
||||
| Ollama missed product spec | Coding model assigned prose task | Route `task_type: docs` to kimi/claude regardless of blast |
|
||||
| Codex slow on frontend (280s vs 122s) | Codex overhead for complex UI tasks | Consider routing blast 6 frontend to claude |
|
||||
| Claude had better architecture | Single model sees full context | Multi-model loses cross-task context — address via checkpoint sharing |
|
||||
|
||||
### 18.6 Post-Benchmark Improvements
|
||||
|
||||
After the benchmark, three systems were built to close the identified gaps:
|
||||
|
||||
#### A. Routing Rules (`maggy/routing_rules.py`)
|
||||
|
||||
A YAML-backed self-updating rules file at `~/.maggy/routing-rules.yaml`. Rules are checked **before** blast-score routing, enforcing that specific task types and pipeline phases always use the right model.
|
||||
|
||||
**Task-type overrides** (from benchmark evidence):
|
||||
|
||||
| Task Type | Forced Model | Confidence | Source |
|
||||
|-----------|-------------|-----------|--------|
|
||||
| `docs` | claude | 0.9 | benchmark — local models are code-optimized, not prose |
|
||||
| `security` | claude | 1.0 | rule — security review needs deep reasoning |
|
||||
| `architecture` | claude | 0.8 | rule — architecture needs cross-context awareness |
|
||||
| `tests` | claude | 0.9 | benchmark — only claude generated test files |
|
||||
| `planning` | claude | 0.8 | rule — planning requires structured reasoning |
|
||||
|
||||
**Pipeline phase overrides** (from TDD workflow):
|
||||
|
||||
| Phase | Forced Model | Reason |
|
||||
|-------|-------------|--------|
|
||||
| `spec` | claude | SPEC phase needs comprehensive docs |
|
||||
| `tdd_red` | claude | RED phase needs test design expertise |
|
||||
| `tdd_green` | auto | GREEN phase uses blast-score routing |
|
||||
| `review` | claude | Review needs security + architecture depth |
|
||||
|
||||
**Self-learning API:**
|
||||
- `record_outcome(rules, model, task_type, success)` — updates rolling success rates from task results
|
||||
- `learn_override(rules, task_type, model, reason, confidence)` — Maggy can add new overrides when data supports it
|
||||
- Manual edits to the YAML are preserved; Maggy only appends learned entries
|
||||
|
||||
This directly addresses:
|
||||
- **"Ollama missed product spec"** → `docs` tasks now forced to claude
|
||||
- **"No tests generated"** → `tests` and `tdd_red` phases now forced to claude
|
||||
|
||||
#### B. Team Conventions (embedded in routing rules)
|
||||
|
||||
Conventions from claude-bootstrap's CLAUDE.md and skill files are embedded in the routing rules and injected into every prompt sent to any CLI:
|
||||
|
||||
```yaml
|
||||
conventions:
|
||||
- text: "Build minimum wowable product (mWP). Ship the smallest thing that makes someone say 'wow'."
|
||||
applies_to: [all]
|
||||
source: claude-bootstrap
|
||||
- text: "Follow TDD: RED → GREEN → VALIDATE. Coverage >= 80%."
|
||||
applies_to: [feature, bug, refactor]
|
||||
source: claude-bootstrap
|
||||
- text: "No secrets in code. Parameterized SQL only. Validate all input at API boundaries."
|
||||
applies_to: [all]
|
||||
source: claude-bootstrap
|
||||
- text: "Quality gates: max 20 lines/function, max 3 params, max 2 nesting levels, max 200 lines/file."
|
||||
applies_to: [all]
|
||||
source: claude-bootstrap
|
||||
- text: "Use existing patterns. Read the codebase before changing it."
|
||||
applies_to: [all]
|
||||
source: claude-bootstrap
|
||||
```
|
||||
|
||||
Every executor prompt method (`_plan_prompt`, `_analysis_prompt`, `_tests_prompt`, `_impl_prompt`) now calls `conventions_for(rules, task_type)` and appends the matching conventions block. This means kimi, codex, ollama, and claude all receive the same team rules — standardizing quality expectations across all models.
|
||||
|
||||
#### C. Routing Rules + Conventions Flow
|
||||
|
||||
```
|
||||
Task arrives → apply_override(task_type, phase)
|
||||
↓ forced?
|
||||
┌─YES─→ use forced model
|
||||
└─NO──→ reward table → blast-score routing
|
||||
↓
|
||||
build prompt + conventions_for(task_type)
|
||||
↓
|
||||
send to CLI with team conventions embedded
|
||||
↓
|
||||
record_outcome() → update YAML success rates
|
||||
```
|
||||
|
||||
#### D. Expected Impact on Re-run
|
||||
|
||||
If the benchmark were re-run with these improvements:
|
||||
|
||||
| Gap (Before) | Expected Result (After) |
|
||||
|-------------|----------------------|
|
||||
| No product spec from ollama | EXP-1 (`docs`) now routes to claude → spec generated |
|
||||
| No tests from any model | TDD pipeline with `tdd_red` → claude → tests generated |
|
||||
| Inconsistent quality | All models receive team conventions (mWP, quality gates, security rules) |
|
||||
| No self-improvement | Outcome recording feeds back into routing rules YAML |
|
||||
|
||||
**Net effect:** Quality score expected to converge with Claude Code's 7.8+ while maintaining the 83% cost reduction.
|
||||
|
||||
@@ -192,3 +192,67 @@ This represents ~83% reduction in Claude subscription consumption.
|
||||
| Model diversity | 4 models | 1 model | Maggy |
|
||||
|
||||
**Summary:** Claude Code is faster and produces marginally higher overall quality (driven by tests and spec). Maggy's multi-model approach provides cost efficiency and subscription risk distribution, plus deeper security review via dedicated model routing. The main gaps to close: add TDD pipeline (test generation step), and improve docs routing (don't send prose tasks to coding-optimized local models).
|
||||
|
||||
---
|
||||
|
||||
## 9. Post-Benchmark Fixes (Routing Rules + Conventions)
|
||||
|
||||
Three systems were built immediately after the benchmark to close the gaps above.
|
||||
|
||||
### 9.1 Routing Rules (`~/.maggy/routing-rules.yaml`)
|
||||
|
||||
A self-updating YAML config that overrides blast-score routing for specific task types and pipeline phases. Rules are checked **before** the reward table or blast-score tier.
|
||||
|
||||
**Task-type overrides seeded from benchmark evidence:**
|
||||
|
||||
| Task Type | Forced To | Why |
|
||||
|-----------|----------|-----|
|
||||
| `docs` | claude | Ollama (code-optimized) produced no spec file |
|
||||
| `security` | claude | Security review needs deep reasoning |
|
||||
| `tests` | claude | Only claude generated test files in benchmark |
|
||||
| `architecture` | claude | Architecture needs cross-context awareness |
|
||||
| `planning` | claude | Planning requires structured reasoning |
|
||||
|
||||
**Pipeline phase overrides from TDD workflow:**
|
||||
|
||||
| Phase | Forced To | Why |
|
||||
|-------|----------|-----|
|
||||
| `spec` | claude | SPEC phase needs comprehensive docs |
|
||||
| `tdd_red` | claude | RED phase needs test design expertise |
|
||||
| `tdd_green` | auto | GREEN uses blast-score routing (cheap models can implement) |
|
||||
| `review` | claude | Review needs security + architecture depth |
|
||||
|
||||
**Self-learning:** `record_outcome()` updates rolling success rates per model. `learn_override()` lets Maggy add new rules when outcome data supports it. Manual YAML edits are preserved.
|
||||
|
||||
### 9.2 Team Conventions Injection
|
||||
|
||||
Five conventions from claude-bootstrap's CLAUDE.md are embedded in routing rules and injected into every prompt sent to any CLI:
|
||||
|
||||
1. **mWP** — Build minimum wowable product. No feature flags, no premature abstractions.
|
||||
2. **TDD** — RED → GREEN → VALIDATE. Coverage >= 80%.
|
||||
3. **Security** — No secrets in code. Parameterized SQL. Validate input at boundaries.
|
||||
4. **Quality gates** — 20 lines/fn, 3 params, 2 nesting levels, 200 lines/file.
|
||||
5. **Existing patterns** — Read codebase before changing. Keep changes minimal.
|
||||
|
||||
All four executor prompt methods (`_plan_prompt`, `_analysis_prompt`, `_tests_prompt`, `_impl_prompt`) now append matching conventions. This standardizes quality expectations across kimi, codex, ollama, and claude.
|
||||
|
||||
### 9.3 Expected Re-run Improvements
|
||||
|
||||
| Benchmark Gap | Root Cause | Fix Applied | Expected Result |
|
||||
|--------------|-----------|-------------|-----------------|
|
||||
| No product spec (EXP-1) | `docs` routed to ollama | `docs → claude` override | Claude generates spec |
|
||||
| No tests from any model | No TDD step in pipeline | `tdd_red → claude` + `tests → claude` overrides | Claude writes failing tests |
|
||||
| Inconsistent quality across models | No shared standards | Conventions injected into all prompts | mWP + quality gates enforced everywhere |
|
||||
| No learning from outcomes | Static routing only | `record_outcome()` + `learn_override()` | Routing improves with each task |
|
||||
|
||||
**Projected scores if re-run:**
|
||||
|
||||
| Dimension | Before | After (est.) | Change |
|
||||
|-----------|--------|-------------|--------|
|
||||
| Product spec | 0/10 | 9/10 | `docs → claude` |
|
||||
| Test coverage | 0/10 | 8/10 | `tdd_red → claude` |
|
||||
| Security | 10/10 | 10/10 | No change (already strong) |
|
||||
| Architecture | 8/10 | 9/10 | Conventions enforce patterns |
|
||||
| **Weighted avg** | **7.4/10** | **~8.5/10** | **+1.1 points** |
|
||||
|
||||
Cost efficiency would remain at ~83% savings — the new overrides only force claude for `docs` (1 task) and `tests` (new TDD step), not for CRUD/API/frontend work.
|
||||
|
||||
+59
-4
@@ -13,9 +13,15 @@ from pathlib import Path
|
||||
from maggy.calibration.tracker import CalibrationTracker
|
||||
from maggy.config import MaggyConfig
|
||||
from maggy.process.model_router import (
|
||||
DEFAULT_TIERS,
|
||||
RoutingDecision,
|
||||
route_task,
|
||||
)
|
||||
from maggy.routing_rules import (
|
||||
apply_override,
|
||||
load as load_rules,
|
||||
record_outcome as rules_record,
|
||||
)
|
||||
from maggy.scores import RewardTable
|
||||
|
||||
MIN_CALIBRATION_ACCURACY = 0.5
|
||||
@@ -29,19 +35,29 @@ class RoutingContext:
|
||||
task_type: str = "general"
|
||||
security_sensitive: bool = False
|
||||
project_key: str = ""
|
||||
pipeline_phase: str = ""
|
||||
|
||||
|
||||
class RoutingService:
|
||||
"""Blast-score aware routing with reward-based learning."""
|
||||
"""Blast-score aware routing with rule overrides."""
|
||||
|
||||
def __init__(self, cfg: MaggyConfig):
|
||||
self.cfg = cfg
|
||||
self.rewards = RewardTable(cfg)
|
||||
db_dir = Path(cfg.storage.path).expanduser().parent
|
||||
self.calibration = CalibrationTracker(db_dir / "calibration.db")
|
||||
self.calibration = CalibrationTracker(
|
||||
db_dir / "calibration.db",
|
||||
)
|
||||
self.rules = load_rules()
|
||||
|
||||
def route(self, ctx: RoutingContext) -> RoutingDecision:
|
||||
"""Pick the best model for this task context."""
|
||||
forced = apply_override(
|
||||
self.rules, ctx.task_type, ctx.pipeline_phase,
|
||||
)
|
||||
if forced:
|
||||
return self._forced_decision(forced, ctx)
|
||||
|
||||
override = self.rewards.best_model(
|
||||
ctx.task_type, self._blast_tier(ctx.blast_score),
|
||||
)
|
||||
@@ -50,8 +66,10 @@ class RoutingService:
|
||||
primary=override,
|
||||
validator=None,
|
||||
fallback_chain=[],
|
||||
reason=f"Learned: best for {ctx.task_type} "
|
||||
f"at blast {ctx.blast_score}",
|
||||
reason=(
|
||||
f"Learned: best for {ctx.task_type} "
|
||||
f"at blast {ctx.blast_score}"
|
||||
),
|
||||
)
|
||||
|
||||
decision = route_task(
|
||||
@@ -72,6 +90,12 @@ class RoutingService:
|
||||
tier = self._blast_tier(blast_score)
|
||||
self.rewards.record(model, task_type, tier, reward)
|
||||
self.calibration.record(model, task_type, reward, reward)
|
||||
success = reward > 0.0
|
||||
rules_record(self.rules, model, task_type, success)
|
||||
|
||||
def reload_rules(self) -> None:
|
||||
"""Reload rules from disk (after Maggy self-update)."""
|
||||
self.rules = load_rules()
|
||||
|
||||
def get_heatmap(self) -> list[dict]:
|
||||
"""Return reward heatmap data for dashboard."""
|
||||
@@ -88,6 +112,29 @@ class RoutingService:
|
||||
acc = self.calibration.accuracy(model)
|
||||
return acc == 0.0 or acc >= MIN_CALIBRATION_ACCURACY
|
||||
|
||||
def _forced_decision(
|
||||
self, model_name: str, ctx: RoutingContext,
|
||||
) -> RoutingDecision:
|
||||
"""Build decision from a rules override."""
|
||||
tier = _find_tier(model_name)
|
||||
if tier is None:
|
||||
return route_task(
|
||||
ctx.blast_score,
|
||||
ctx.task_type,
|
||||
ctx.security_sensitive,
|
||||
)
|
||||
validator = None
|
||||
if ctx.blast_score >= 8 or ctx.security_sensitive:
|
||||
validator = _find_tier("codex")
|
||||
return RoutingDecision(
|
||||
primary=tier,
|
||||
validator=validator,
|
||||
fallback_chain=[],
|
||||
reason=f"Rule override: {ctx.task_type}"
|
||||
f"{f'/{ctx.pipeline_phase}' if ctx.pipeline_phase else ''}"
|
||||
f" → {model_name}",
|
||||
)
|
||||
|
||||
def _penalize_uncalibrated(
|
||||
self, decision: RoutingDecision,
|
||||
) -> RoutingDecision:
|
||||
@@ -101,3 +148,11 @@ class RoutingService:
|
||||
reason="Calibration penalty",
|
||||
)
|
||||
return decision
|
||||
|
||||
|
||||
def _find_tier(name: str):
|
||||
"""Look up a ModelTier by name from defaults."""
|
||||
for t in DEFAULT_TIERS:
|
||||
if t.name == name:
|
||||
return t
|
||||
return None
|
||||
|
||||
@@ -0,0 +1,363 @@
|
||||
"""Routing rules — task-type and pipeline-phase model assignments.
|
||||
|
||||
Loaded from ~/.maggy/routing-rules.yaml. Maggy can self-update
|
||||
this file when benchmark or outcome data provides evidence for
|
||||
better routing decisions. Manual edits are preserved.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from maggy.config import CONFIG_DIR
|
||||
|
||||
RULES_PATH = CONFIG_DIR / "routing-rules.yaml"
|
||||
MIN_CONFIDENCE = 0.6
|
||||
|
||||
|
||||
@dataclass
|
||||
class ModelOverride:
|
||||
"""Force a specific model for a task type or phase."""
|
||||
|
||||
model: str
|
||||
reason: str = ""
|
||||
confidence: float = 1.0
|
||||
source: str = "rule"
|
||||
|
||||
|
||||
@dataclass
|
||||
class PerformanceRecord:
|
||||
"""Tracked model performance from outcomes."""
|
||||
|
||||
strengths: list[str] = field(default_factory=list)
|
||||
weaknesses: list[str] = field(default_factory=list)
|
||||
tasks_completed: int = 0
|
||||
success_rate: float = 0.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class Convention:
|
||||
"""A team convention injected into prompts."""
|
||||
|
||||
text: str
|
||||
applies_to: list[str] = field(default_factory=list)
|
||||
source: str = "manual"
|
||||
|
||||
|
||||
@dataclass
|
||||
class RoutingRules:
|
||||
"""All routing rules Maggy uses for orchestration."""
|
||||
|
||||
version: int = 1
|
||||
updated_at: str = ""
|
||||
task_type_overrides: dict[str, ModelOverride] = field(
|
||||
default_factory=dict,
|
||||
)
|
||||
pipeline_phases: dict[str, ModelOverride] = field(
|
||||
default_factory=dict,
|
||||
)
|
||||
model_performance: dict[str, PerformanceRecord] = field(
|
||||
default_factory=dict,
|
||||
)
|
||||
conventions: list[Convention] = field(
|
||||
default_factory=list,
|
||||
)
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _default_conventions() -> list[Convention]:
|
||||
"""Team conventions from claude-bootstrap skills."""
|
||||
return [
|
||||
Convention(
|
||||
"Build minimum wowable product (mWP). Ship the smallest "
|
||||
"thing that makes someone say 'wow'. No feature flags, no "
|
||||
"backwards-compat shims, no premature abstractions.",
|
||||
["all"], "claude-bootstrap",
|
||||
),
|
||||
Convention(
|
||||
"Follow TDD: RED (write failing tests) → GREEN (minimal "
|
||||
"code to pass) → VALIDATE (lint, types, coverage >= 80%).",
|
||||
["feature", "bug", "refactor"], "claude-bootstrap",
|
||||
),
|
||||
Convention(
|
||||
"No secrets in code. Parameterized SQL only. Validate all "
|
||||
"input at API boundaries. Hash passwords with bcrypt/argon2.",
|
||||
["all"], "claude-bootstrap",
|
||||
),
|
||||
Convention(
|
||||
"Quality gates: max 20 lines/function, max 3 params, "
|
||||
"max 2 nesting levels, max 200 lines/file.",
|
||||
["all"], "claude-bootstrap",
|
||||
),
|
||||
Convention(
|
||||
"Use existing patterns. Read the codebase before changing it. "
|
||||
"Keep changes minimal and focused on the task.",
|
||||
["all"], "claude-bootstrap",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def default_rules() -> RoutingRules:
|
||||
"""Seed rules from benchmark evidence + team conventions."""
|
||||
return RoutingRules(
|
||||
version=1,
|
||||
updated_at=_now_iso(),
|
||||
conventions=_default_conventions(),
|
||||
task_type_overrides={
|
||||
"docs": ModelOverride(
|
||||
"claude",
|
||||
"Local models are code-optimized, not prose",
|
||||
0.9, "benchmark",
|
||||
),
|
||||
"security": ModelOverride(
|
||||
"claude",
|
||||
"Security review needs deep reasoning",
|
||||
1.0, "rule",
|
||||
),
|
||||
"architecture": ModelOverride(
|
||||
"claude",
|
||||
"Architecture needs cross-context awareness",
|
||||
0.8, "rule",
|
||||
),
|
||||
"tests": ModelOverride(
|
||||
"claude",
|
||||
"Only claude generated test files in benchmark",
|
||||
0.9, "benchmark",
|
||||
),
|
||||
"planning": ModelOverride(
|
||||
"claude",
|
||||
"Planning requires structured reasoning",
|
||||
0.8, "rule",
|
||||
),
|
||||
},
|
||||
pipeline_phases={
|
||||
"spec": ModelOverride(
|
||||
"claude",
|
||||
"SPEC phase needs comprehensive docs",
|
||||
1.0, "rule",
|
||||
),
|
||||
"tdd_red": ModelOverride(
|
||||
"claude",
|
||||
"RED phase needs test design expertise",
|
||||
0.9, "rule",
|
||||
),
|
||||
"tdd_green": ModelOverride(
|
||||
"auto",
|
||||
"GREEN phase uses blast-score routing",
|
||||
1.0, "rule",
|
||||
),
|
||||
"review": ModelOverride(
|
||||
"claude",
|
||||
"Review needs security + architecture depth",
|
||||
1.0, "rule",
|
||||
),
|
||||
},
|
||||
model_performance={
|
||||
"claude": PerformanceRecord(
|
||||
["security", "tests", "docs", "architecture"],
|
||||
["cost"],
|
||||
6, 1.0,
|
||||
),
|
||||
"codex": PerformanceRecord(
|
||||
["crud", "api_design"],
|
||||
["frontend_speed", "tests"],
|
||||
3, 1.0,
|
||||
),
|
||||
"kimi": PerformanceRecord(
|
||||
["schema", "simple_tasks"],
|
||||
["complex_reasoning"],
|
||||
1, 1.0,
|
||||
),
|
||||
"local": PerformanceRecord(
|
||||
["code_formatting", "simple_edits"],
|
||||
["docs", "prose", "planning"],
|
||||
1, 1.0,
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def load(path: Path | None = None) -> RoutingRules:
|
||||
"""Load rules from YAML. Seeds defaults if missing."""
|
||||
target = path or RULES_PATH
|
||||
if not target.exists():
|
||||
rules = default_rules()
|
||||
save(rules, target)
|
||||
return rules
|
||||
rules = _from_yaml(target)
|
||||
if not rules.conventions:
|
||||
rules.conventions = _default_conventions()
|
||||
save(rules, target)
|
||||
return rules
|
||||
|
||||
|
||||
def save(rules: RoutingRules, path: Path | None = None) -> None:
|
||||
"""Write rules to YAML."""
|
||||
target = path or RULES_PATH
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
data = _to_dict(rules)
|
||||
target.write_text(yaml.safe_dump(data, sort_keys=False))
|
||||
|
||||
|
||||
def apply_override(
|
||||
rules: RoutingRules,
|
||||
task_type: str,
|
||||
phase: str | None = None,
|
||||
) -> str | None:
|
||||
"""Return model name if rules override routing.
|
||||
|
||||
Returns None if blast-score routing should be used.
|
||||
"""
|
||||
if phase and phase in rules.pipeline_phases:
|
||||
override = rules.pipeline_phases[phase]
|
||||
if override.model != "auto" and _trusted(override):
|
||||
return override.model
|
||||
|
||||
if task_type in rules.task_type_overrides:
|
||||
override = rules.task_type_overrides[task_type]
|
||||
if _trusted(override):
|
||||
return override.model
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def record_outcome(
|
||||
rules: RoutingRules,
|
||||
model: str,
|
||||
task_type: str,
|
||||
success: bool,
|
||||
path: Path | None = None,
|
||||
) -> None:
|
||||
"""Update performance data from a task outcome."""
|
||||
perf = rules.model_performance.get(model)
|
||||
if perf is None:
|
||||
perf = PerformanceRecord()
|
||||
rules.model_performance[model] = perf
|
||||
|
||||
total = perf.tasks_completed
|
||||
rate = perf.success_rate
|
||||
new_total = total + 1
|
||||
new_rate = (rate * total + (1.0 if success else 0.0)) / new_total
|
||||
perf.tasks_completed = new_total
|
||||
perf.success_rate = round(new_rate, 3)
|
||||
|
||||
if success and task_type not in perf.strengths:
|
||||
perf.strengths.append(task_type)
|
||||
if not success and task_type not in perf.weaknesses:
|
||||
perf.weaknesses.append(task_type)
|
||||
|
||||
rules.updated_at = _now_iso()
|
||||
save(rules, path)
|
||||
|
||||
|
||||
def learn_override(
|
||||
rules: RoutingRules,
|
||||
task_type: str,
|
||||
model: str,
|
||||
reason: str,
|
||||
confidence: float = 0.7,
|
||||
path: Path | None = None,
|
||||
) -> None:
|
||||
"""Maggy learns a new routing override from data."""
|
||||
rules.task_type_overrides[task_type] = ModelOverride(
|
||||
model=model,
|
||||
reason=reason,
|
||||
confidence=confidence,
|
||||
source="learned",
|
||||
)
|
||||
rules.updated_at = _now_iso()
|
||||
save(rules, path)
|
||||
|
||||
|
||||
def conventions_for(
|
||||
rules: RoutingRules, task_type: str,
|
||||
) -> str:
|
||||
"""Return conventions text relevant to a task type."""
|
||||
lines: list[str] = []
|
||||
for c in rules.conventions:
|
||||
if "all" in c.applies_to or task_type in c.applies_to:
|
||||
lines.append(f"- {c.text}")
|
||||
if not lines:
|
||||
return ""
|
||||
return "## Team Conventions\n" + "\n".join(lines)
|
||||
|
||||
|
||||
def _trusted(override: ModelOverride) -> bool:
|
||||
return override.confidence >= MIN_CONFIDENCE
|
||||
|
||||
|
||||
def _to_dict(rules: RoutingRules) -> dict:
|
||||
return {
|
||||
"version": rules.version,
|
||||
"updated_at": rules.updated_at,
|
||||
"conventions": [
|
||||
{
|
||||
"text": c.text,
|
||||
"applies_to": c.applies_to,
|
||||
"source": c.source,
|
||||
}
|
||||
for c in rules.conventions
|
||||
],
|
||||
"task_type_overrides": {
|
||||
k: {
|
||||
"model": v.model,
|
||||
"reason": v.reason,
|
||||
"confidence": v.confidence,
|
||||
"source": v.source,
|
||||
}
|
||||
for k, v in rules.task_type_overrides.items()
|
||||
},
|
||||
"pipeline_phases": {
|
||||
k: {
|
||||
"model": v.model,
|
||||
"reason": v.reason,
|
||||
"confidence": v.confidence,
|
||||
"source": v.source,
|
||||
}
|
||||
for k, v in rules.pipeline_phases.items()
|
||||
},
|
||||
"model_performance": {
|
||||
k: {
|
||||
"strengths": v.strengths,
|
||||
"weaknesses": v.weaknesses,
|
||||
"tasks_completed": v.tasks_completed,
|
||||
"success_rate": v.success_rate,
|
||||
}
|
||||
for k, v in rules.model_performance.items()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _from_yaml(path: Path) -> RoutingRules:
|
||||
data = yaml.safe_load(path.read_text()) or {}
|
||||
overrides = {
|
||||
k: ModelOverride(**v)
|
||||
for k, v in (data.get("task_type_overrides") or {}).items()
|
||||
}
|
||||
phases = {
|
||||
k: ModelOverride(**v)
|
||||
for k, v in (data.get("pipeline_phases") or {}).items()
|
||||
}
|
||||
perf = {
|
||||
k: PerformanceRecord(**v)
|
||||
for k, v in (data.get("model_performance") or {}).items()
|
||||
}
|
||||
convs = [
|
||||
Convention(**c)
|
||||
for c in (data.get("conventions") or [])
|
||||
]
|
||||
return RoutingRules(
|
||||
version=data.get("version", 1),
|
||||
updated_at=data.get("updated_at", ""),
|
||||
task_type_overrides=overrides,
|
||||
pipeline_phases=phases,
|
||||
model_performance=perf,
|
||||
conventions=convs,
|
||||
)
|
||||
@@ -25,6 +25,7 @@ from maggy.process.model_router import RoutingDecision
|
||||
from maggy.providers.base import IssueTrackerProvider, Task
|
||||
from maggy.recovery.rollback import RollbackManager
|
||||
from maggy.routing import RoutingContext, RoutingService
|
||||
from maggy.routing_rules import conventions_for
|
||||
from maggy.services.planner import DualPlanner
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -342,36 +343,40 @@ class ExecutorService:
|
||||
})
|
||||
|
||||
def _plan_prompt(self, task: Task, icpg_ctx: str) -> str:
|
||||
conv = self._conventions_block(task)
|
||||
return (
|
||||
"Create an implementation plan for this ticket. No code changes — just a plan.\n\n"
|
||||
f"Ticket: {task.title}\n{task.description[:1500]}"
|
||||
f"{self._icpg_block(icpg_ctx)}\n"
|
||||
f"{self._icpg_block(icpg_ctx)}{conv}\n"
|
||||
"Output: numbered steps, files to touch, risks, tests to add."
|
||||
)
|
||||
|
||||
def _analysis_prompt(self, task: Task, icpg_ctx: str) -> str:
|
||||
conv = self._conventions_block(task)
|
||||
return (
|
||||
"Analyze this ticket against the codebase and output a concise plan.\n"
|
||||
"Identify: files to change, functions affected, tests needed, risks.\n\n"
|
||||
f"Ticket: {task.title}\n{task.description[:1500]}"
|
||||
f"{self._icpg_block(icpg_ctx)}"
|
||||
f"{self._icpg_block(icpg_ctx)}{conv}"
|
||||
)
|
||||
|
||||
def _tests_prompt(self, task: Task, icpg_ctx: str, analysis: str) -> str:
|
||||
conv = self._conventions_block(task)
|
||||
return (
|
||||
"Write failing test cases for this ticket (TDD — no implementation yet).\n"
|
||||
"Use the project's existing test patterns. Commit tests separately.\n\n"
|
||||
f"Ticket: {task.title}\n{task.description[:1500]}"
|
||||
f"{self._icpg_block(icpg_ctx)}\n"
|
||||
f"{self._icpg_block(icpg_ctx)}{conv}\n"
|
||||
f"Analysis:\n{analysis[:1000]}"
|
||||
)
|
||||
|
||||
def _impl_prompt(self, task: Task, icpg_ctx: str) -> str:
|
||||
conv = self._conventions_block(task)
|
||||
return (
|
||||
"Implement the feature to make the failing tests pass.\n"
|
||||
"Follow existing code patterns. Keep changes minimal.\n\n"
|
||||
f"Ticket: {task.title}\n{task.description[:1500]}"
|
||||
f"{self._icpg_block(icpg_ctx)}\n"
|
||||
f"{self._icpg_block(icpg_ctx)}{conv}\n"
|
||||
"Run tests to verify, then commit with a conventional commit message."
|
||||
)
|
||||
|
||||
@@ -380,6 +385,14 @@ class ExecutorService:
|
||||
return ""
|
||||
return f"\n\n{icpg_ctx}\n"
|
||||
|
||||
def _conventions_block(self, task: Task) -> str:
|
||||
raw = task.raw if isinstance(task.raw, dict) else {}
|
||||
task_type = str(raw.get("task_type") or self._task_type(task))
|
||||
text = conventions_for(self._routing.rules, task_type)
|
||||
if not text:
|
||||
return ""
|
||||
return f"\n\n{text}\n"
|
||||
|
||||
async def _post_plan(self, task_id: str, output: str) -> None:
|
||||
try:
|
||||
await self.provider.add_comment(
|
||||
|
||||
@@ -108,8 +108,9 @@ class TestRoutingAccuracy:
|
||||
name = decision.primary if isinstance(decision.primary, str) else decision.primary.name
|
||||
results[task.id] = name
|
||||
|
||||
# Low blast (1-3) → cheap tier
|
||||
assert results["T-1"] in ("local", "kimi", "deepseek")
|
||||
# Low blast (1-3) → cheap tier unless rules override
|
||||
# T-1 is docs → rules force claude (local can't do prose)
|
||||
assert results["T-1"] == "claude"
|
||||
assert results["T-2"] in ("local", "kimi", "deepseek")
|
||||
assert results["T-3"] in ("local", "kimi", "deepseek")
|
||||
# Mid blast (4-6) → gpt
|
||||
@@ -131,7 +132,7 @@ class TestRoutingAccuracy:
|
||||
correct = 0
|
||||
|
||||
expected_tiers = {
|
||||
"T-1": "cheap", "T-2": "cheap", "T-3": "cheap",
|
||||
"T-1": "premium", "T-2": "cheap", "T-3": "cheap",
|
||||
"T-4": "medium", "T-5": "medium",
|
||||
"T-6": "medium", # blast 7 overlaps gpt/claude
|
||||
"T-7": "premium", "T-8": "premium",
|
||||
|
||||
@@ -90,7 +90,8 @@ async def test_plan_mode_records_spend_for_selected_model(mock_cfg, tmp_path, mo
|
||||
|
||||
await executor._run("session-1", _task(3, "security"), str(tmp_path), "plan")
|
||||
|
||||
assert executor._budget.today_spend("openai") == pytest.approx(1.25)
|
||||
# security tasks are now rule-overridden to claude (anthropic)
|
||||
assert executor._budget.today_spend("anthropic") == pytest.approx(1.25)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -138,3 +139,31 @@ async def test_fatigue_tracked_during_steps(mock_cfg, tmp_path, monkeypatch):
|
||||
await executor._run("session-1", _task(3, "docs"), str(tmp_path), "plan")
|
||||
|
||||
assert executor._fatigue.dimensions["context_load"] > 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_conventions_injected_into_prompts(mock_cfg, tmp_path, monkeypatch):
|
||||
provider = AsyncMock()
|
||||
executor = ExecutorService(mock_cfg, provider)
|
||||
executor._sessions["session-1"] = _session()
|
||||
prompts: list[str] = []
|
||||
|
||||
async def fake_context(task: Task, wd: str) -> str:
|
||||
return ""
|
||||
|
||||
async def fake_send(
|
||||
model_name: str, prompt: str, working_dir: str,
|
||||
max_turns: int = 20, timeout: int = 600,
|
||||
) -> RunResult:
|
||||
prompts.append(prompt)
|
||||
return RunResult(model=model_name, success=True, output="ok")
|
||||
|
||||
monkeypatch.setattr(executor, "_build_icpg_context", fake_context)
|
||||
monkeypatch.setattr(executor._pi, "send_prompt", fake_send)
|
||||
|
||||
await executor._run(
|
||||
"session-1", _task(5, "feature"), str(tmp_path), "plan",
|
||||
)
|
||||
assert prompts, "At least one prompt should have been sent"
|
||||
assert "Team Conventions" in prompts[0]
|
||||
assert "minimum wowable product" in prompts[0]
|
||||
|
||||
@@ -81,7 +81,8 @@ class TestRoutingDecisions:
|
||||
cfg = _project_cfg(tmp_path)
|
||||
svc = RoutingService(cfg)
|
||||
for blast in (1, 2):
|
||||
ctx = RoutingContext(blast_score=blast, task_type="docs")
|
||||
# Use "formatting" — "docs" is now rules-overridden
|
||||
ctx = RoutingContext(blast_score=blast, task_type="formatting")
|
||||
decision = svc.route(ctx)
|
||||
assert decision.primary.cost_rank <= 2, (
|
||||
f"blast={blast} should route to cheap tier"
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
"""Tests for routing rules — load, save, apply, learn."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from maggy.routing_rules import (
|
||||
ModelOverride,
|
||||
PerformanceRecord,
|
||||
RoutingRules,
|
||||
apply_override,
|
||||
default_rules,
|
||||
learn_override,
|
||||
load,
|
||||
record_outcome,
|
||||
save,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def rules_path(tmp_path: Path) -> Path:
|
||||
return tmp_path / "routing-rules.yaml"
|
||||
|
||||
|
||||
class TestDefaultRules:
|
||||
def test_seeds_task_type_overrides(self):
|
||||
rules = default_rules()
|
||||
assert "docs" in rules.task_type_overrides
|
||||
assert "security" in rules.task_type_overrides
|
||||
assert "tests" in rules.task_type_overrides
|
||||
|
||||
def test_seeds_pipeline_phases(self):
|
||||
rules = default_rules()
|
||||
assert "spec" in rules.pipeline_phases
|
||||
assert "tdd_red" in rules.pipeline_phases
|
||||
assert rules.pipeline_phases["tdd_green"].model == "auto"
|
||||
|
||||
def test_seeds_model_performance(self):
|
||||
rules = default_rules()
|
||||
assert "claude" in rules.model_performance
|
||||
assert "local" in rules.model_performance
|
||||
|
||||
|
||||
class TestLoadSave:
|
||||
def test_load_creates_default(self, rules_path: Path):
|
||||
rules = load(rules_path)
|
||||
assert rules_path.exists()
|
||||
assert "docs" in rules.task_type_overrides
|
||||
|
||||
def test_roundtrip(self, rules_path: Path):
|
||||
original = default_rules()
|
||||
save(original, rules_path)
|
||||
loaded = load(rules_path)
|
||||
assert loaded.version == original.version
|
||||
assert set(loaded.task_type_overrides) == set(
|
||||
original.task_type_overrides,
|
||||
)
|
||||
|
||||
def test_load_existing(self, rules_path: Path):
|
||||
save(default_rules(), rules_path)
|
||||
rules = load(rules_path)
|
||||
assert rules.task_type_overrides["security"].model == "claude"
|
||||
|
||||
|
||||
class TestApplyOverride:
|
||||
def test_phase_takes_priority(self):
|
||||
rules = default_rules()
|
||||
result = apply_override(rules, "feature", "spec")
|
||||
assert result == "claude"
|
||||
|
||||
def test_auto_phase_returns_none(self):
|
||||
rules = default_rules()
|
||||
result = apply_override(rules, "feature", "tdd_green")
|
||||
assert result is None
|
||||
|
||||
def test_task_type_override(self):
|
||||
rules = default_rules()
|
||||
result = apply_override(rules, "security")
|
||||
assert result == "claude"
|
||||
|
||||
def test_no_override_returns_none(self):
|
||||
rules = default_rules()
|
||||
result = apply_override(rules, "feature")
|
||||
assert result is None
|
||||
|
||||
def test_low_confidence_ignored(self):
|
||||
rules = RoutingRules(
|
||||
task_type_overrides={
|
||||
"test": ModelOverride("kimi", "weak", 0.3),
|
||||
},
|
||||
)
|
||||
result = apply_override(rules, "test")
|
||||
assert result is None
|
||||
|
||||
|
||||
class TestRecordOutcome:
|
||||
def test_updates_success_rate(self, rules_path: Path):
|
||||
rules = default_rules()
|
||||
record_outcome(rules, "claude", "feature", True, rules_path)
|
||||
perf = rules.model_performance["claude"]
|
||||
assert perf.tasks_completed == 7
|
||||
assert perf.success_rate > 0.9
|
||||
|
||||
def test_creates_new_model(self, rules_path: Path):
|
||||
rules = default_rules()
|
||||
record_outcome(rules, "gemini", "feature", True, rules_path)
|
||||
assert "gemini" in rules.model_performance
|
||||
assert rules.model_performance["gemini"].success_rate == 1.0
|
||||
|
||||
def test_records_failure(self, rules_path: Path):
|
||||
rules = RoutingRules(
|
||||
model_performance={
|
||||
"test": PerformanceRecord(
|
||||
tasks_completed=1, success_rate=1.0,
|
||||
),
|
||||
},
|
||||
)
|
||||
record_outcome(rules, "test", "security", False, rules_path)
|
||||
assert rules.model_performance["test"].success_rate == 0.5
|
||||
assert "security" in rules.model_performance["test"].weaknesses
|
||||
|
||||
|
||||
class TestLearnOverride:
|
||||
def test_adds_new_override(self, rules_path: Path):
|
||||
rules = default_rules()
|
||||
learn_override(
|
||||
rules, "frontend", "claude",
|
||||
"Codex too slow for frontend (280s vs 122s)",
|
||||
0.8, rules_path,
|
||||
)
|
||||
assert rules.task_type_overrides["frontend"].model == "claude"
|
||||
assert rules.task_type_overrides["frontend"].source == "learned"
|
||||
|
||||
def test_persists_to_disk(self, rules_path: Path):
|
||||
rules = default_rules()
|
||||
save(rules, rules_path)
|
||||
learn_override(
|
||||
rules, "frontend", "claude", "test", 0.9, rules_path,
|
||||
)
|
||||
reloaded = load(rules_path)
|
||||
assert "frontend" in reloaded.task_type_overrides
|
||||
Reference in New Issue
Block a user