mirror of
https://github.com/borghei/Claude-Skills.git
synced 2026-09-14 19:41:02 +08:00
feat(pm): tier 3 — output eval harness for 12 artifact-generating PM skills
Final layer of the depth investment. Deterministic scoring of any markdown artifact against per-skill rubrics. No LLM in the loop, stdlib only. FRAMEWORK (evals/) - engine.py — generic check evaluator with 13 check types: regex, regex_not, section_present, section_word_count, section_sentence_count, keyword_any, keyword_all, keyword_none, has_table, has_list, url_count, length_in_range, line_count_range. - run.py — runner with --all, --skill, --artifact, --format, --threshold, --output flags. - README.md — framework documentation + how to add a new rubric. RUBRICS (12 skills, 190+ criteria total) Each rubric anchors every criterion to a specific entry in the skill's references/red-flags.md or SKILL.md Success Criteria: - create-prd — 17 criteria scoring against 8-section PRD model - prfaq — 19 criteria scoring against Amazon Working Backwards - ai-feature-prd — 21 criteria scoring against eval/guardrails/model-fallback - brainstorm-okrs — 19 criteria scoring against SMART + Wodtke confidence - status-update — 19 criteria scoring against SBNR + R/Y/G + Asks - post-mortem — 19 criteria scoring against blameless + 5 Whys + owners - north-star-metric — 19 criteria scoring against NSM + input tree + counter - product-vision — 19 criteria scoring against Pichler / Raskin structure - pricing-prd — 20 criteria scoring against Westendorp + grandfathering - roadmap-comms — 19 criteria scoring against 3-variant audience fit - release-notes — 19 criteria scoring against value framing + categories - feedback-triage — 19 criteria scoring against Kano + RICE + ack templates SMOKE TEST (all 12 worked examples self-scored) - 6 skills @ 100/100 (ai-feature-prd, customer-feedback-triage, north-star-metric, post-mortem, product-vision, status-update-generator) - 6 skills @ 89-95/100 (brainstorm-okrs 95, pricing-prd 95, prfaq 94, roadmap-communication 94, release-notes 90, create-prd 89) - 12/12 pass at threshold 70 PM README + CHANGELOG updated. This commit completes the 3-tier PM depth investment: - Tier 1 (505cb8e): worked examples (54), data adapters (3), MCP tools (15) - Tier 2 (a8063e1): red-flag libraries (54), runnable pipelines (5) - Tier 3 (this): output eval harness (12 rubrics) + framework PM domain: 54 skills, 78 sub-files of worked examples, 54 red-flag libraries, 12 deterministic scorers, 5 chain pipelines, 3 live data adapters, 15 MCP tools. The skills moved from "documentation" to "production toolkit."
This commit is contained in:
Brian Borghei
@@ -5,6 +5,45 @@ All notable changes to the Claude Skills Library will be documented in this file
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [4.7.0] - 2026-05-22 (Tier 3 of PM depth)
|
||||
|
||||
### Added
|
||||
|
||||
**Output evaluation harness for artifact-generating PM skills (`evals/`).** Deterministic scoring (no LLM) for any markdown artifact, scored 0-100 against the skill's red-flags + success criteria.
|
||||
|
||||
Framework:
|
||||
- `evals/engine.py` — generic check evaluator. 13 supported check types: regex, regex_not, section_present, section_word_count, section_sentence_count, keyword_any, keyword_all, keyword_none, has_table, has_list, url_count, length_in_range, line_count_range.
|
||||
- `evals/run.py` — runner. Supports `--all`, `--skill <name>`, `--artifact <path>`, `--format markdown|json`, `--threshold <int>`, `--output <file>`.
|
||||
- `evals/README.md` — framework documentation.
|
||||
|
||||
12 rubrics (190+ criteria total, each anchored to a specific red-flag or success-criterion item):
|
||||
- create-prd · prfaq · ai-feature-prd · brainstorm-okrs · status-update-generator
|
||||
- post-mortem · north-star-metric · product-vision · pricing-prd
|
||||
- roadmap-communication · release-notes · customer-feedback-triage
|
||||
|
||||
Smoke-test (all 12 worked examples scored against their own rubric):
|
||||
|
||||
| Skill | Score |
|
||||
|---|---|
|
||||
| ai-feature-prd | 100 |
|
||||
| customer-feedback-triage | 100 |
|
||||
| north-star-metric | 100 |
|
||||
| post-mortem | 100 |
|
||||
| product-vision | 100 |
|
||||
| status-update-generator | 100 |
|
||||
| brainstorm-okrs | 95 |
|
||||
| pricing-prd | 95 |
|
||||
| prfaq | 94 |
|
||||
| roadmap-communication | 94 |
|
||||
| release-notes | 90 |
|
||||
| create-prd | 89 |
|
||||
|
||||
12/12 pass at threshold 70. High scores reflect that the worked examples were authored as exemplars; the rubrics are calibrated to catch realistic failures (drafts with missing sections, blame language, output-as-KR, watermelon status, etc.).
|
||||
|
||||
### Changed
|
||||
|
||||
- PM README updated with new "Output evaluation harness" section.
|
||||
|
||||
## [4.6.0] - 2026-05-22 (Tier 2 of PM depth)
|
||||
|
||||
### Added
|
||||
|
||||
+139
@@ -0,0 +1,139 @@
|
||||
# PM Skill Output Evaluation Harness
|
||||
|
||||
Deterministic scoring for artifacts generated by PM skills. Catches output-quality regressions without an LLM in the loop.
|
||||
|
||||
**Why:** PM skills produce artifacts (PRDs, post-mortems, OKRs, status updates, etc.). Without a scoring mechanism, "is this output good?" is a vibes call. The harness encodes the *red flags* and *success criteria* from each skill as concrete checks against the artifact.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
evals/
|
||||
├── README.md # this file
|
||||
├── engine.py # generic check evaluator (stdlib only)
|
||||
├── run.py # runner: iterates rubrics, scores examples
|
||||
└── <skill>/
|
||||
└── rubric.json # per-skill rubric — list of weighted checks
|
||||
```
|
||||
|
||||
For each skill, a `rubric.json` lists weighted criteria. Each criterion has a `check` block with a `type` and parameters. The engine knows how to evaluate each type. The runner finds artifacts (typically in the skill's `examples/` folder) and scores them.
|
||||
|
||||
## Supported check types
|
||||
|
||||
| Type | Purpose | Parameters |
|
||||
|---|---|---|
|
||||
| `regex` | Pattern must match | `pattern`, `flags` (default `im`) |
|
||||
| `regex_not` | Pattern must NOT match | `pattern`, `flags` |
|
||||
| `section_present` | A markdown heading exists | `heading`, `level` (default 2) |
|
||||
| `section_word_count` | Section length in range | `heading`, `min`, `max` |
|
||||
| `section_sentence_count` | Section sentence count in range | `heading`, `min`, `max` |
|
||||
| `keyword_any` | At least one keyword present | `keywords` (list) |
|
||||
| `keyword_none` | All listed keywords absent | `keywords` (list) |
|
||||
| `keyword_all` | All listed keywords present | `keywords` (list) |
|
||||
| `has_table` | Contains a markdown table | `min_rows` (optional) |
|
||||
| `has_list` | Contains a bullet or numbered list | `min_items` (optional) |
|
||||
| `url_count` | URL count in range | `min`, `max` |
|
||||
| `length_in_range` | Total length (chars) in range | `min`, `max` |
|
||||
| `line_count_range` | Line count in range | `min`, `max` |
|
||||
|
||||
A criterion passes (full weight) or fails (zero). For partial credit, use multiple smaller criteria.
|
||||
|
||||
## Rubric format
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": "create-prd",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores an 8-section PRD against red-flags + success criteria.",
|
||||
"max_score": 100,
|
||||
"criteria": [
|
||||
{
|
||||
"id": "summary_present",
|
||||
"name": "Section 1 (Summary) present",
|
||||
"weight": 8,
|
||||
"check": {"type": "section_present", "heading": "Summary"}
|
||||
},
|
||||
{
|
||||
"id": "summary_concise",
|
||||
"name": "Summary is 2-4 sentences (10-second exec test)",
|
||||
"weight": 5,
|
||||
"check": {"type": "section_sentence_count", "heading": "Summary", "min": 2, "max": 4}
|
||||
},
|
||||
{
|
||||
"id": "no_jargon",
|
||||
"name": "No consultant-speak",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_none", "keywords": ["synergy", "leverage", "deep-dive", "circle back", "low-hanging fruit"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
# Score one skill's worked example
|
||||
python evals/run.py --skill create-prd
|
||||
|
||||
# Score all skills
|
||||
python evals/run.py --all
|
||||
|
||||
# Score a specific artifact file (any markdown)
|
||||
python evals/run.py --skill create-prd --artifact path/to/my-prd.md
|
||||
|
||||
# JSON output for CI
|
||||
python evals/run.py --all --format json
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
Markdown summary by default:
|
||||
|
||||
```
|
||||
# Evaluation Report
|
||||
|
||||
## create-prd / examples/shared-dashboards-prd.md
|
||||
Score: 87 / 100 (PASS — threshold 70)
|
||||
|
||||
### Passed (8 of 10 criteria)
|
||||
- ✅ Section 1 (Summary) present (8)
|
||||
- ✅ Summary is 2-4 sentences (5)
|
||||
- ...
|
||||
|
||||
### Failed (2 of 10)
|
||||
- ❌ Has explicit Assumptions table (8) — section "Assumptions" not found
|
||||
- ❌ KR1 has baseline -> target -> deadline format (6)
|
||||
```
|
||||
|
||||
## Adding a new rubric
|
||||
|
||||
1. Create `evals/<skill>/rubric.json`
|
||||
2. Anchor each criterion to a specific entry in `<skill>/references/red-flags.md` or the SKILL.md Success Criteria
|
||||
3. Total weights should sum to 100 (the runner normalizes if they don't)
|
||||
4. Run `python evals/run.py --skill <skill>` to test
|
||||
|
||||
## Limitations
|
||||
|
||||
- **No semantic understanding.** Checks are structural / lexical. The engine can verify "does the artifact have a Section 7 Solution?" but not "is the solution actually a solution?"
|
||||
- **No autograding of judgment.** A PR/FAQ scoring 95/100 may still describe a bad product. The harness catches *form* failures, not *substance* failures.
|
||||
- **Calibration matters.** Initial weights are author-curated guesses. Adjust after running against 5-10 real artifacts.
|
||||
|
||||
## Covered skills
|
||||
|
||||
Current rubrics:
|
||||
|
||||
| Skill | Rubric | Anchored to |
|
||||
|---|---|---|
|
||||
| create-prd | rubric.json | SKILL.md 8 sections + red-flags |
|
||||
| prfaq | rubric.json | Amazon Working Backwards |
|
||||
| ai-feature-prd | rubric.json | eval-spec + guardrails + model-selection sections |
|
||||
| brainstorm-okrs | rubric.json | SMART + Wodtke confidence |
|
||||
| status-update-generator | rubric.json | SBNR + R/Y/G + Asks |
|
||||
| post-mortem | rubric.json | Blameless + 5 Whys + action items with owners |
|
||||
| north-star-metric | rubric.json | NSM + input tree + counter-metrics |
|
||||
| product-vision | rubric.json | Pichler Vision Board / 5-10-year horizon |
|
||||
| pricing-prd | rubric.json | Westendorp + grandfathering + rollback |
|
||||
| roadmap-communication | rubric.json | Three-variant audience fit |
|
||||
| release-notes | rubric.json | Value framing + categorization |
|
||||
| customer-feedback-triage | rubric.json | Kano + scoring + ack template |
|
||||
|
||||
12 rubrics total.
|
||||
@@ -0,0 +1,133 @@
|
||||
{
|
||||
"skill": "ai-feature-prd",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores an 11-section AI Feature PRD against AI-specific success criteria: model selection w/ fallback, numeric eval thresholds, guardrails, deployment ramp w/ gates, HIL, EU AI Act tier — anchored to references/red-flags.md.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "summary_present",
|
||||
"name": "Section 1 (Summary) names the model class explicitly",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Claude", "GPT", "Sonnet", "Haiku", "Llama", "Gemini", "BERT", "fine-tuned", "RAG", "LLM"]}
|
||||
},
|
||||
{
|
||||
"id": "objective_present",
|
||||
"name": "Section 4 (Objective) with Key Results",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "KR\\d|Key Result", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "ai_system_design_section",
|
||||
"name": "Section 9 (AI System Design) present",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:9\\.?\\s+|Section\\s+9:?\\s+)?AI System Design", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "model_selection_with_fallback",
|
||||
"name": "Model selection includes a primary AND a fallback",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_all", "keywords": ["primary", "fallback"]}
|
||||
},
|
||||
{
|
||||
"id": "switch_trigger",
|
||||
"name": "Model switch trigger named (outage, regression, cost cap)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["switch trigger", "fail over", "outage", "regression", "cost cap"]}
|
||||
},
|
||||
{
|
||||
"id": "architecture_pattern",
|
||||
"name": "Architecture pattern declared (prompt / RAG / fine-tune / agent) with rejected alternatives",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Prompt + RAG", "RAG", "fine-tune", "agentic", "rejected", "alternatives"]}
|
||||
},
|
||||
{
|
||||
"id": "eval_safety_section",
|
||||
"name": "Section 10 (Eval & Safety Plan) present",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:10\\.?\\s+|Section\\s+10:?\\s+)?Eval", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "eval_numeric_thresholds",
|
||||
"name": "Eval section has numeric thresholds (%, ms, $)",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "(acceptance.*\\d+%|hallucinat.*\\d+%|>= ?\\d+%|<= ?\\d+%|p95.*\\d+\\s?ms|\\$0\\.\\d+)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "hallucination_target",
|
||||
"name": "Hallucination rate target named",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "hallucinat[a-z]+\\s*(rate)?\\s*(<=|<|under|less than|of)?\\s*\\d", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "refusal_policy",
|
||||
"name": "Refusal policy explicit (enumerated categories, not one sentence)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Refusal policy", "refuses to", "never:", "redirect"]}
|
||||
},
|
||||
{
|
||||
"id": "golden_set_committed",
|
||||
"name": "Golden set / eval set is concrete (size + composition, not 'TBD')",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["golden set", "gold set", "200-item", "evals/", "adversarial", "common", "edge cases"]}
|
||||
},
|
||||
{
|
||||
"id": "no_tbd_eval",
|
||||
"name": "No 'golden set TBD' or 'eval TBD' deferral",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "(golden\\s*set|gold\\s*set|eval[a-z]*)\\s*:?\\s*TBD", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "guardrails_layers",
|
||||
"name": "Guardrails section enumerates layers (input / output / HIL / etc.)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_all", "keywords": ["input", "output", "guardrail"]}
|
||||
},
|
||||
{
|
||||
"id": "human_in_the_loop",
|
||||
"name": "Human-in-the-loop checkpoints declared",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["human-in-the-loop", "human in the loop", "HIL", "hard gate", "soft gate", "sampling review"]}
|
||||
},
|
||||
{
|
||||
"id": "failure_modes_table",
|
||||
"name": "Failure modes enumerated with detection + response",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_all", "keywords": ["failure", "detection"]}
|
||||
},
|
||||
{
|
||||
"id": "eu_ai_act_tier",
|
||||
"name": "EU AI Act risk tier declared (minimal / limited / high / unacceptable)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["EU AI Act", "Limited Risk", "High Risk", "High-risk", "Minimal Risk", "risk tier"]}
|
||||
},
|
||||
{
|
||||
"id": "operations_cost_section",
|
||||
"name": "Section 11 (Operations & Cost) present",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:11\\.?\\s+|Section\\s+11:?\\s+)?(Operations|Cost)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "cost_model_with_scale",
|
||||
"name": "Cost model includes 10x / scale scenario or per-tenant alerts",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["10x", "per-tenant", "80% of budget", "auto-throttle", "throttle at", "cost-per", "$/Mtok"]}
|
||||
},
|
||||
{
|
||||
"id": "deployment_ramp_gates",
|
||||
"name": "Deployment ramp shadow -> internal -> canary -> GA with gate metrics",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_all", "keywords": ["shadow", "canary"]}
|
||||
},
|
||||
{
|
||||
"id": "no_100_percent_target",
|
||||
"name": "No naive 100% acceptance target (red flag #4)",
|
||||
"weight": 3,
|
||||
"check": {"type": "regex_not", "pattern": "acceptance.{0,30}(100%|>=\\s*99%|99\\.9%)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "no_buzzwords",
|
||||
"name": "No marketing buzzwords",
|
||||
"weight": 2,
|
||||
"check": {"type": "keyword_none", "keywords": ["revolutionary", "next-generation", "world-class", "seamless"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "brainstorm-okrs",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores an OKR set against Wodtke's Radical Focus: qualitative Objective, measurable outcome-focused KRs with baseline/target/deadline + confidence ratings (60-70%), counter-metrics, no output-disguised KRs.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "objective_present",
|
||||
"name": "Objective section present",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(^###?\\s+Objective|^\\*\\*Objective\\*\\*|^Objective:)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "key_results_present",
|
||||
"name": "Key Results section present (KR1/KR2/KR3 markers)",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "KR\\d|Key Result", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "objective_qualitative",
|
||||
"name": "Objective contains no naked numeric metric (qualitative, not numeric)",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex_not", "pattern": "^[#*]*\\s*(Objective[:\\)]).{0,80}(\\d+%|\\d+k|\\$\\d|\\d+ users|reach \\d|>= ?\\d)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "baseline_target_format",
|
||||
"name": "KRs use from/to/baseline/target language (measurable)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["from ", "to ", "baseline", "target", ">= ", "increase", "reduce"]}
|
||||
},
|
||||
{
|
||||
"id": "confidence_present",
|
||||
"name": "Confidence rating per KR (Wodtke 60-70% bar)",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "(confidence[:\\s]*[0-9]{2}%|[0-9]{2}%\\s+confidence|confidence\\s+(rating|level)|10/7/5/3)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "confidence_in_wodtke_range",
|
||||
"name": "At least one KR confidence in Wodtke 50-75% range",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "\\b(5[0-9]|6[0-9]|7[0-5])%\\b", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "counter_metric_present",
|
||||
"name": "Counter-metric explicitly named",
|
||||
"weight": 8,
|
||||
"check": {"type": "keyword_any", "keywords": ["counter-metric", "counter metric", "(counter)", "Counter:"]}
|
||||
},
|
||||
{
|
||||
"id": "deadline_window",
|
||||
"name": "KR window / deadline (quarterly bounds)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Q1", "Q2", "Q3", "Q4", "by end of", "by 20", "2026-", "deadline"]}
|
||||
},
|
||||
{
|
||||
"id": "no_output_ship_verbs",
|
||||
"name": "KRs are not output-disguised ('Ship X', 'Launch X', 'Build X' as the only verb)",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex_not", "pattern": "KR\\d?:?\\s*(Ship|Launch|Build|Publish|Implement)\\s+[A-Z]", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "outcome_verbs",
|
||||
"name": "KRs use outcome verbs (increase / reduce / improve / lift)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["increase", "reduce", "improve", "lift", "grow", "maintain", "boost", "decrease"]}
|
||||
},
|
||||
{
|
||||
"id": "max_two_objectives",
|
||||
"name": "Single objective (Radical Focus) — not a list of 5",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex_not", "pattern": "Obj(ective)?\\s*5|^###?\\s+(Objective 4|Objective 5)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "rationale_present",
|
||||
"name": "Rationale tying OKR to theme/strategy present",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["Rationale", "Why", "Theme:", "connects to", "tied to", "aligned"]}
|
||||
},
|
||||
{
|
||||
"id": "weekly_cadence",
|
||||
"name": "Weekly check-in cadence referenced (no set-and-forget)",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["weekly check-in", "weekly cadence", "Monday cadence", "weekly health", "weekly OKR", "check-in"]}
|
||||
},
|
||||
{
|
||||
"id": "kill_or_review_criteria",
|
||||
"name": "Kill criteria / mid-quarter review mentioned",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["kill criteria", "mid-Q", "mid-quarter", "learning quarter", "pivot"]}
|
||||
},
|
||||
{
|
||||
"id": "no_activity_count_kr",
|
||||
"name": "No 'Publish 12 blog posts' / 'Run X interviews' activity KRs",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex_not", "pattern": "KR\\d?:?\\s+(Publish|Run|Conduct|Hold|Organize)\\s+\\d+\\s+(blog|posts|interview|meeting|webinar|standup)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "metric_with_source",
|
||||
"name": "KR cites measurement source (analytics tool, survey, etc.)",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["HubSpot", "Amplitude", "survey", "NPS", "Mixpanel", "GA4", "via ", "source:", "telemetry"]}
|
||||
},
|
||||
{
|
||||
"id": "three_or_fewer_krs_per_obj",
|
||||
"name": "No more than 4 KRs per objective (anti-roadmap-disguised-as-OKR)",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "KR5|KR6|KR7", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "alignment_company",
|
||||
"name": "Alignment to company / org-level objective referenced",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["company OKR", "org OKR", "company-level", "org-level", "alignment", "cascade", "company priorities"]}
|
||||
},
|
||||
{
|
||||
"id": "no_objective_with_percent",
|
||||
"name": "Objective is not just a number/percent (Red Flag 6)",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "^[#*]*\\s*Objective:?\\s+(Reach|Achieve|Hit)\\s+\\d+", "flags": "im"}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
{
|
||||
"skill": "create-prd",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores an 8-section PRD against SKILL.md Success Criteria and references/red-flags.md.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "summary_present",
|
||||
"name": "Section 1 (Summary) present",
|
||||
"weight": 8,
|
||||
"check": {"type": "section_present", "heading": "Summary"}
|
||||
},
|
||||
{
|
||||
"id": "summary_concise",
|
||||
"name": "Summary is 2-4 sentences (passes 10-second exec test)",
|
||||
"weight": 5,
|
||||
"check": {"type": "section_sentence_count", "heading": "Summary", "min": 2, "max": 5}
|
||||
},
|
||||
{
|
||||
"id": "background_present",
|
||||
"name": "Section 3 (Background) present with context + why-now",
|
||||
"weight": 6,
|
||||
"check": {"type": "section_present", "heading": "Background"}
|
||||
},
|
||||
{
|
||||
"id": "objective_present",
|
||||
"name": "Section 4 (Objective) present",
|
||||
"weight": 6,
|
||||
"check": {"type": "section_present", "heading": "Objective"}
|
||||
},
|
||||
{
|
||||
"id": "key_results_format",
|
||||
"name": "Has Key Results in OKR format (KR1/KR2/KR3)",
|
||||
"weight": 8,
|
||||
"check": {"type": "regex", "pattern": "KR\\d|Key Result", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "key_results_measurable",
|
||||
"name": "Key Results contain measurable language (from/to/by, %, baseline, target)",
|
||||
"weight": 8,
|
||||
"check": {"type": "keyword_any", "keywords": ["from", "to", "baseline", "target", "%", "by 20"]}
|
||||
},
|
||||
{
|
||||
"id": "segments_present",
|
||||
"name": "Section 5 (Market Segments) present",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:Section\\s+5:?\\s+)?Market Segment", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "segments_not_demographic",
|
||||
"name": "Segments defined by jobs/problems, not raw demographics",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_none", "keywords": ["millennials aged", "gen z aged", "demographics", "18-34", "25-35 in urban"]}
|
||||
},
|
||||
{
|
||||
"id": "value_prop_present",
|
||||
"name": "Section 6 (Value Proposition) present",
|
||||
"weight": 5,
|
||||
"check": {"type": "section_present", "heading": "Value Proposition"}
|
||||
},
|
||||
{
|
||||
"id": "solution_present",
|
||||
"name": "Section 7 (Solution) present",
|
||||
"weight": 6,
|
||||
"check": {"type": "section_present", "heading": "Solution"}
|
||||
},
|
||||
{
|
||||
"id": "priorities_marked",
|
||||
"name": "Features are tagged P0/P1/P2",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "P[012]\\b", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "assumptions_table",
|
||||
"name": "Assumptions are listed (Section 7 sub-bullet or explicit table)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["Assumption", "We believe", "Hypothesis"]}
|
||||
},
|
||||
{
|
||||
"id": "release_present",
|
||||
"name": "Section 8 (Release) present",
|
||||
"weight": 5,
|
||||
"check": {"type": "section_present", "heading": "Release"}
|
||||
},
|
||||
{
|
||||
"id": "v1_scope_defined",
|
||||
"name": "v1 scope or Now/Next/Later breakdown present",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["v1 scope", "Now", "Next", "Later", "MVP"]}
|
||||
},
|
||||
{
|
||||
"id": "deferred_explicit",
|
||||
"name": "Explicitly deferred items called out (anti-scope-creep)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["deferred", "out of scope", "v2", "future version", "explicitly excluded"]}
|
||||
},
|
||||
{
|
||||
"id": "no_consultant_speak",
|
||||
"name": "No consultant-speak / buzzwords",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_none", "keywords": ["synergy", "leverage best", "deep-dive", "circle back", "low-hanging fruit", "move the needle"]}
|
||||
},
|
||||
{
|
||||
"id": "no_solution_before_problem",
|
||||
"name": "Problem framing present before solution (negative check on 'solution' appearing before 'problem' in the doc)",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(problem|background)[\\s\\S]+?(solution|key features)", "flags": "is"}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "customer-feedback-triage",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores a customer-feedback triage artifact against Cagan Request/Opportunity/Solution separation, Kano categorization, deduplicated clusters w/ scoring, acknowledgment templates for 3 outcomes, and routing to discovery/prioritization.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "clusters_not_raw_list",
|
||||
"name": "Items clustered (not raw enumerated list)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["cluster", "Cluster", "clustered", "C-001", "C-002", "opportunity_label", "Opportunity label", "deduplicated"]}
|
||||
},
|
||||
{
|
||||
"id": "channel_field",
|
||||
"name": "Channel field present (support / sales / NPS / in-app / exec_ask)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["channel:", "Channel:", "support", "sales", "NPS", "in_app", "in-app", "exec_ask", "customer_interview"]}
|
||||
},
|
||||
{
|
||||
"id": "customer_or_segment",
|
||||
"name": "Customer / segment field present (enterprise / mid-market / SMB)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["segment", "enterprise", "mid-market", "mid_market", "SMB", "Segment:", "customer_id"]}
|
||||
},
|
||||
{
|
||||
"id": "raw_text_captured",
|
||||
"name": "Raw verbatim text captured (not paraphrased only)",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["raw_text", "verbatim", "Raw text", "submitted text", "asked for", "\""]}
|
||||
},
|
||||
{
|
||||
"id": "opportunity_normalized",
|
||||
"name": "Normalized opportunity / job statement (not raw request)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["opportunity", "Opportunity:", "job statement", "underlying job", "underlying problem", "Cagan", "Request vs Opportunity"]}
|
||||
},
|
||||
{
|
||||
"id": "kano_category",
|
||||
"name": "Kano category present (basic / performance / delight / indifferent / reverse)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["Kano", "kano_category", "basic", "performance", "delight", "must-be", "Attractive"]}
|
||||
},
|
||||
{
|
||||
"id": "scoring_rubric",
|
||||
"name": "Scoring rubric applied (priority / RICE-like / weighted)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["priority_score", "priority score", "Score:", "RICE", "weighted", "scored against"]}
|
||||
},
|
||||
{
|
||||
"id": "distinct_customers_volume",
|
||||
"name": "Volume counted via dedup / source-request count (anti-squeaky-wheel)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["distinct_customers", "distinct customers", "Volume:", "customer count", "deduplication", "after dedup", "dedup", "distinct count", "Source requests"]}
|
||||
},
|
||||
{
|
||||
"id": "ack_template_will_build",
|
||||
"name": "Will-build response template referenced",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["will-build", "Will-build", "will build", "committed", "Will build:"]}
|
||||
},
|
||||
{
|
||||
"id": "ack_template_wont_build",
|
||||
"name": "Won't-build / Decline response template referenced",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["won't-build", "won't build", "Won't-build", "wont-build", "Will not build", "decided not to build", "Decline", "decline politely", "politely decline"]}
|
||||
},
|
||||
{
|
||||
"id": "ack_template_exploring",
|
||||
"name": "Exploring / Discovery response template referenced",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["exploring", "Exploring", "evaluating", "Discovery", "discovery (investigate)", "investigate", "under exploration"]}
|
||||
},
|
||||
{
|
||||
"id": "routing_destinations",
|
||||
"name": "Routing destinations named (prioritization / discovery / bug tracker / strategy)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["Routing", "routing", "to_prioritization", "to_bug_tracker", "to_strategy", "route to", "routed to"]}
|
||||
},
|
||||
{
|
||||
"id": "bug_vs_feature_categorization",
|
||||
"name": "Bug vs Feature vs Question vs Strategy categorization",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Bug", "Feature request", "feature_request", "Strategy", "Question", "categoriz"]}
|
||||
},
|
||||
{
|
||||
"id": "no_sales_weighting_only",
|
||||
"name": "Sales-driven roadmap not the only weighting (anti-pattern)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex_not", "pattern": "(sole|only)\\s+(scoring|weighting)\\s+(criteri[ao]n|factor)[:\\s]*(deal size|ARR|enterprise)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "not_request_literally",
|
||||
"name": "Does not treat raw request as the build target only (Red Flag 4)",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "Solution[:\\s]+(Build|Implement)\\s+(PDF|Excel|CSV)\\s+export\\s+button(?:\\s|\\.|,|$)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "exec_ask_through_triage",
|
||||
"name": "Exec asks routed through same triage (no HiPPO bypass)",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["exec_ask", "exec ask", "HiPPO", "same triage", "exec-channel", "exec channel", "CEO request", "CEO's", "Sponsorship is not prioritization", "exec one-liner"]}
|
||||
},
|
||||
{
|
||||
"id": "acknowledgment_sla",
|
||||
"name": "Acknowledgment SLA / rate referenced",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["acknowledgment", "Acknowledge", "14 days", "acknowledgment rate", "respond within", "auto-acknowledge"]}
|
||||
},
|
||||
{
|
||||
"id": "downstream_handoff",
|
||||
"name": "Handoff to downstream discovery / prioritization (prevents triage = roadmap)",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["prioritization-frameworks", "interview-synthesis", "identify-assumptions", "RICE", "discovery", "downstream"]}
|
||||
},
|
||||
{
|
||||
"id": "no_pure_volume",
|
||||
"name": "Volume mentioned alongside dedup (not raw submission count only)",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["after deduplication", "deduplication", "distinct customers", "after dedup", "dedup", "collapsed into", "Source requests"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
+271
@@ -0,0 +1,271 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
engine.py — Deterministic check evaluator for PM skill output rubrics.
|
||||
|
||||
Stdlib only. See evals/README.md for the supported check types.
|
||||
|
||||
Public API:
|
||||
evaluate_artifact(rubric: dict, artifact_text: str) -> dict
|
||||
extract_section(text, heading, level=2) -> str | None
|
||||
check_passes(check: dict, text: str) -> tuple[bool, str]
|
||||
"""
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Section extraction (markdown)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _heading_regex(heading: str, level: int = 2) -> re.Pattern:
|
||||
"""Match a markdown heading like '## <heading>' (allowing 'Section N:' prefix)."""
|
||||
hashes = "#" * max(1, min(level, 6))
|
||||
# allow optional "Section N:" or numeric prefix
|
||||
h = re.escape(heading).replace(r"\ ", r"\s+")
|
||||
return re.compile(
|
||||
rf"^{hashes}\s+(?:Section\s+\d+:\s+|\d+\.\s+)?{h}\s*$",
|
||||
re.MULTILINE | re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def extract_section(text: str, heading: str, level: int = 2) -> str | None:
|
||||
"""Extract the body of a section starting with the given heading.
|
||||
|
||||
Returns the text from after the heading line until the next heading of
|
||||
the same or higher level, or end of file. Returns None if not found.
|
||||
"""
|
||||
pat = _heading_regex(heading, level)
|
||||
m = pat.search(text)
|
||||
if not m:
|
||||
return None
|
||||
start = m.end()
|
||||
# Find next heading at same or higher level
|
||||
boundary = re.compile(
|
||||
rf"^#{{1,{level}}}\s+",
|
||||
re.MULTILINE,
|
||||
)
|
||||
rest = text[start:]
|
||||
nxt = boundary.search(rest)
|
||||
return rest[:nxt.start()] if nxt else rest
|
||||
|
||||
|
||||
def _sentences(text: str) -> list[str]:
|
||||
"""Naive sentence splitter — splits on . ? ! followed by space + capital."""
|
||||
chunks = re.split(r"(?<=[.!?])\s+(?=[A-Z\"'])", text.strip())
|
||||
return [c.strip() for c in chunks if c.strip()]
|
||||
|
||||
|
||||
def _words(text: str) -> list[str]:
|
||||
return [w for w in re.split(r"\s+", text) if w]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check evaluators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _flag_to_re(flags: str) -> int:
|
||||
f = 0
|
||||
if "i" in flags: f |= re.IGNORECASE
|
||||
if "m" in flags: f |= re.MULTILINE
|
||||
if "s" in flags: f |= re.DOTALL
|
||||
return f
|
||||
|
||||
|
||||
def check_regex(params: dict, text: str) -> tuple[bool, str]:
|
||||
pat = params.get("pattern", "")
|
||||
flags = _flag_to_re(params.get("flags", "im"))
|
||||
if not pat:
|
||||
return False, "missing 'pattern'"
|
||||
found = bool(re.search(pat, text, flags))
|
||||
return (found, "matched" if found else "pattern not found")
|
||||
|
||||
|
||||
def check_regex_not(params: dict, text: str) -> tuple[bool, str]:
|
||||
pat = params.get("pattern", "")
|
||||
flags = _flag_to_re(params.get("flags", "im"))
|
||||
if not pat:
|
||||
return False, "missing 'pattern'"
|
||||
found = bool(re.search(pat, text, flags))
|
||||
return (not found, "absent (good)" if not found else "forbidden pattern found")
|
||||
|
||||
|
||||
def check_section_present(params: dict, text: str) -> tuple[bool, str]:
|
||||
h = params.get("heading", "")
|
||||
lvl = int(params.get("level", 2))
|
||||
if not h: return False, "missing 'heading'"
|
||||
found = extract_section(text, h, lvl) is not None
|
||||
return (found, "found" if found else f"section '{h}' not found")
|
||||
|
||||
|
||||
def check_section_word_count(params: dict, text: str) -> tuple[bool, str]:
|
||||
h = params.get("heading", "")
|
||||
lvl = int(params.get("level", 2))
|
||||
mn = int(params.get("min", 0))
|
||||
mx = int(params.get("max", 10**6))
|
||||
body = extract_section(text, h, lvl)
|
||||
if body is None:
|
||||
return False, f"section '{h}' not found"
|
||||
n = len(_words(body))
|
||||
ok = mn <= n <= mx
|
||||
return (ok, f"{n} words (need {mn}-{mx})")
|
||||
|
||||
|
||||
def check_section_sentence_count(params: dict, text: str) -> tuple[bool, str]:
|
||||
h = params.get("heading", "")
|
||||
lvl = int(params.get("level", 2))
|
||||
mn = int(params.get("min", 0))
|
||||
mx = int(params.get("max", 10**6))
|
||||
body = extract_section(text, h, lvl)
|
||||
if body is None:
|
||||
return False, f"section '{h}' not found"
|
||||
# Strip tables and code blocks from sentence count
|
||||
body_clean = re.sub(r"```.*?```", " ", body, flags=re.DOTALL)
|
||||
body_clean = re.sub(r"^\|.*\|\s*$", " ", body_clean, flags=re.MULTILINE)
|
||||
n = len(_sentences(body_clean))
|
||||
ok = mn <= n <= mx
|
||||
return (ok, f"{n} sentences (need {mn}-{mx})")
|
||||
|
||||
|
||||
def check_keyword_any(params: dict, text: str) -> tuple[bool, str]:
|
||||
kws = params.get("keywords", [])
|
||||
if not kws: return False, "no keywords"
|
||||
low = text.lower()
|
||||
matched = [k for k in kws if k.lower() in low]
|
||||
return (len(matched) > 0, f"matched: {matched[:3]}" if matched else "none of " + ", ".join(kws[:5]))
|
||||
|
||||
|
||||
def check_keyword_all(params: dict, text: str) -> tuple[bool, str]:
|
||||
kws = params.get("keywords", [])
|
||||
if not kws: return False, "no keywords"
|
||||
low = text.lower()
|
||||
missing = [k for k in kws if k.lower() not in low]
|
||||
return (len(missing) == 0, "all present" if not missing else f"missing: {missing}")
|
||||
|
||||
|
||||
def check_keyword_none(params: dict, text: str) -> tuple[bool, str]:
|
||||
kws = params.get("keywords", [])
|
||||
if not kws: return False, "no keywords"
|
||||
low = text.lower()
|
||||
found = [k for k in kws if k.lower() in low]
|
||||
return (len(found) == 0, "clean" if not found else f"found forbidden: {found}")
|
||||
|
||||
|
||||
def check_has_table(params: dict, text: str) -> tuple[bool, str]:
|
||||
rows = re.findall(r"^\|.+\|\s*$", text, re.MULTILINE)
|
||||
# subtract separator rows
|
||||
real_rows = [r for r in rows if not re.match(r"^\|\s*:?-+:?\s*(\|\s*:?-+:?\s*)+\|\s*$", r)]
|
||||
min_rows = int(params.get("min_rows", 2))
|
||||
n = len(real_rows)
|
||||
ok = n >= min_rows
|
||||
return (ok, f"{n} rows (need >= {min_rows})")
|
||||
|
||||
|
||||
def check_has_list(params: dict, text: str) -> tuple[bool, str]:
|
||||
items = re.findall(r"^\s*(?:[-*+]\s+|\d+\.\s+)", text, re.MULTILINE)
|
||||
min_items = int(params.get("min_items", 3))
|
||||
n = len(items)
|
||||
ok = n >= min_items
|
||||
return (ok, f"{n} list items (need >= {min_items})")
|
||||
|
||||
|
||||
def check_url_count(params: dict, text: str) -> tuple[bool, str]:
|
||||
urls = re.findall(r"https?://\S+", text)
|
||||
mn = int(params.get("min", 0))
|
||||
mx = int(params.get("max", 10**6))
|
||||
n = len(urls)
|
||||
ok = mn <= n <= mx
|
||||
return (ok, f"{n} URLs (need {mn}-{mx})")
|
||||
|
||||
|
||||
def check_length_in_range(params: dict, text: str) -> tuple[bool, str]:
|
||||
n = len(text)
|
||||
mn = int(params.get("min", 0))
|
||||
mx = int(params.get("max", 10**6))
|
||||
ok = mn <= n <= mx
|
||||
return (ok, f"{n} chars (need {mn}-{mx})")
|
||||
|
||||
|
||||
def check_line_count_range(params: dict, text: str) -> tuple[bool, str]:
|
||||
n = text.count("\n") + 1
|
||||
mn = int(params.get("min", 0))
|
||||
mx = int(params.get("max", 10**6))
|
||||
ok = mn <= n <= mx
|
||||
return (ok, f"{n} lines (need {mn}-{mx})")
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"regex": check_regex,
|
||||
"regex_not": check_regex_not,
|
||||
"section_present": check_section_present,
|
||||
"section_word_count": check_section_word_count,
|
||||
"section_sentence_count": check_section_sentence_count,
|
||||
"keyword_any": check_keyword_any,
|
||||
"keyword_all": check_keyword_all,
|
||||
"keyword_none": check_keyword_none,
|
||||
"has_table": check_has_table,
|
||||
"has_list": check_has_list,
|
||||
"url_count": check_url_count,
|
||||
"length_in_range": check_length_in_range,
|
||||
"line_count_range": check_line_count_range,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rubric evaluation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def check_passes(check: dict, text: str) -> tuple[bool, str]:
|
||||
ctype = check.get("type", "")
|
||||
fn = CHECKS.get(ctype)
|
||||
if not fn:
|
||||
return False, f"unknown check type: {ctype}"
|
||||
try:
|
||||
return fn(check, text)
|
||||
except Exception as exc:
|
||||
return False, f"check error: {exc}"
|
||||
|
||||
|
||||
def evaluate_artifact(rubric: dict, artifact_text: str) -> dict:
|
||||
"""Run a rubric against an artifact. Returns a structured result.
|
||||
|
||||
Result shape:
|
||||
{
|
||||
"skill": str,
|
||||
"max_score": int,
|
||||
"raw_score": int,
|
||||
"score": int, # 0-100, weight-normalized
|
||||
"passed": int,
|
||||
"failed": int,
|
||||
"results": [
|
||||
{"id", "name", "weight", "passed", "detail"}
|
||||
]
|
||||
}
|
||||
"""
|
||||
criteria = rubric.get("criteria", []) or []
|
||||
raw_max = sum(int(c.get("weight", 1)) for c in criteria) or 1
|
||||
rows: list[dict] = []
|
||||
raw_score = 0
|
||||
for c in criteria:
|
||||
w = int(c.get("weight", 1))
|
||||
passed, detail = check_passes(c.get("check", {}), artifact_text)
|
||||
if passed:
|
||||
raw_score += w
|
||||
rows.append({
|
||||
"id": c.get("id", ""),
|
||||
"name": c.get("name", c.get("id", "")),
|
||||
"weight": w,
|
||||
"passed": passed,
|
||||
"detail": detail,
|
||||
})
|
||||
score = round(100 * raw_score / raw_max) if raw_max > 0 else 0
|
||||
return {
|
||||
"skill": rubric.get("skill", ""),
|
||||
"rubric_version": rubric.get("version", ""),
|
||||
"max_score": raw_max,
|
||||
"raw_score": raw_score,
|
||||
"score": score,
|
||||
"passed": sum(1 for r in rows if r["passed"]),
|
||||
"failed": sum(1 for r in rows if not r["passed"]),
|
||||
"results": rows,
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "north-star-metric",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores an NSM specification against the Amplitude / Sean Ellis framework: single NSM, 3-5 input metrics with formula, leading indicators per input, counter-metrics with thresholds, anti-metrics, archetype declaration, baseline+target.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "nsm_section_present",
|
||||
"name": "North Star Metric section present",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:The\\s+)?North Star Metric", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "single_nsm_named",
|
||||
"name": "Single NSM stated (not a list of candidates)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["NSM:", "**NSM", "north star metric (NSM)", "The North Star Metric"]}
|
||||
},
|
||||
{
|
||||
"id": "baseline_present",
|
||||
"name": "Current baseline value present",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["Current:", "Current baseline", "current value", "baseline:", "trajectory"]}
|
||||
},
|
||||
{
|
||||
"id": "target_with_deadline",
|
||||
"name": "NSM target with deadline",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(target|Q[1-4]).{0,40}(20\\d{2}|Q[1-4])", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "input_metrics_section",
|
||||
"name": "Input metrics section present",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(Input(s| metrics|s? \\()|Input metric tree)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "three_to_five_inputs",
|
||||
"name": "3-5 input metrics named",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(Input\\s*[1-5]|IN[1-5]|Input #[1-5])", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "formula_explicit",
|
||||
"name": "Explicit formula relating inputs to NSM",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["Formula", "Math:", "NSM =", "= Input", "multiplies", "multiplicative", "additive"]}
|
||||
},
|
||||
{
|
||||
"id": "leading_indicators",
|
||||
"name": "Leading indicators per input",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["Leading indicator", "leading indicators", "Lead:", "early-warning", "leads"]}
|
||||
},
|
||||
{
|
||||
"id": "counter_metrics",
|
||||
"name": "Counter-metrics section present",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["Counter-metric", "counter metrics", "Counter:", "guardrail"]}
|
||||
},
|
||||
{
|
||||
"id": "anti_metrics",
|
||||
"name": "Anti-metrics section present",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Anti-metric", "anti metrics", "anti-metrics"]}
|
||||
},
|
||||
{
|
||||
"id": "counter_threshold",
|
||||
"name": "Counter-metric has explicit threshold (number/%)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(must\\s+(stay|not)\\s+(below|above|exceed)|threshold[:\\s]+[<>]|stay below \\d|stay above \\d|<\\s*\\d|>\\s*\\d)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "archetype_declared",
|
||||
"name": "NSM archetype declared (attention/transaction/productivity/communication/subscriber)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["archetype", "Productivity", "Attention", "Transaction", "Communication", "Subscriber"]}
|
||||
},
|
||||
{
|
||||
"id": "not_just_revenue",
|
||||
"name": "NSM is not naked revenue / MRR / ARR alone",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex_not", "pattern": "^##\\s+(?:The\\s+)?North Star Metric\\s*$[\\s\\S]{0,200}(NSM[:\\s]+(MRR|ARR|Revenue|Total revenue)\\s*$)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "not_dau_mau_alone",
|
||||
"name": "NSM is not raw DAU/MAU alone",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "^\\s*\\*?\\*?NSM[:\\s]+(DAU|MAU)\\s*\\*?\\*?\\s*$", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "value_oriented_language",
|
||||
"name": "NSM language oriented to customer value (action verb / completion / engagement)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["completed", "published", "engaged", "delivered", "share", "complete a deliverable", "active", "consumed"]}
|
||||
},
|
||||
{
|
||||
"id": "tree_visualization",
|
||||
"name": "Tree or table representation (mermaid or markdown table)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(```mermaid|graph\\s+(TD|LR)|\\|---|^\\|.*\\|.*\\|)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "five_tests_acknowledged",
|
||||
"name": "5 tests / criteria acknowledged (customer value, leading, single number, movable, strategic)",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["Customer value", "leading, not lagging", "single number", "Movable", "Strategic alignment", "passes the", "5 tests"]}
|
||||
},
|
||||
{
|
||||
"id": "owner_per_input",
|
||||
"name": "Owner or squad named per input metric",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["Owner:", "squad", "PM:", "team:"]}
|
||||
},
|
||||
{
|
||||
"id": "no_vanity_only",
|
||||
"name": "Does not list vanity-only metrics ('total dashboards', 'total seats')",
|
||||
"weight": 3,
|
||||
"check": {"type": "regex_not", "pattern": "(?:^|\\n)\\s*\\*?\\*?NSM[:\\s]+Total\\s+(dashboards|seats|signups)\\s*\\*?\\*?\\s*$", "flags": "im"}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "post-mortem",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores a blameless post-mortem against Google SRE / Allspaw / Dekker standards: severity, timeline, what-went-well, contributing factors (plural), 5 Whys or causal tree, action items with owners + dates + tickets.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "severity_classification",
|
||||
"name": "Severity classification (Sev0/1/2/3) declared",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "Sev\\s?[0-4]|Severity[:\\s]+(Sev|S)?\\s?[0-4]|SEV-?[0-4]", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "summary_section",
|
||||
"name": "Summary section present",
|
||||
"weight": 5,
|
||||
"check": {"type": "section_present", "heading": "Summary"}
|
||||
},
|
||||
{
|
||||
"id": "impact_section",
|
||||
"name": "Impact section present",
|
||||
"weight": 5,
|
||||
"check": {"type": "section_present", "heading": "Impact"}
|
||||
},
|
||||
{
|
||||
"id": "timeline_section",
|
||||
"name": "Timeline section present",
|
||||
"weight": 7,
|
||||
"check": {"type": "section_present", "heading": "Timeline"}
|
||||
},
|
||||
{
|
||||
"id": "timeline_has_timestamps",
|
||||
"name": "Timeline contains timestamps",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "\\b\\d{1,2}:\\d{2}\\b", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "what_went_well_section",
|
||||
"name": "What went well section present (often-skipped)",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:\\d+\\.\\s+)?(What went well|Went well|Successes)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "what_went_wrong_section",
|
||||
"name": "What went wrong section present",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:\\d+\\.\\s+)?(What went wrong|Went wrong)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "contributing_factors_plural",
|
||||
"name": "Contributing factors enumerated (plural — not single root cause)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["Contributing factors", "contributing factor", "Causal tree", "causal-tree"]}
|
||||
},
|
||||
{
|
||||
"id": "five_whys_or_causal_tree",
|
||||
"name": "5 Whys or Causal Tree method used and named",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["5 Whys", "Five Whys", "5-whys", "Causal Tree", "causal tree", "Why did"]}
|
||||
},
|
||||
{
|
||||
"id": "action_items_section",
|
||||
"name": "Action items section present",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:\\d+\\.\\s+)?(Action [Ii]tems|Action-?items)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "action_items_have_owners",
|
||||
"name": "Action items have named owners and due dates",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_all", "keywords": ["Owner", "Due"]}
|
||||
},
|
||||
{
|
||||
"id": "action_items_have_tickets",
|
||||
"name": "Action items have tracker ticket IDs",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(PAY|ENG|DOC|PROJ|JIRA|LIN|AI-)\\-?\\d+", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "no_blame_should_have",
|
||||
"name": "No blame language ('should have', 'failed to', 'if only')",
|
||||
"weight": 8,
|
||||
"check": {"type": "regex_not", "pattern": "(should have (noticed|caught|known|escalated)|failed to (deploy|notice|catch|escalate)|if only)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "no_human_error",
|
||||
"name": "Does not declare 'human error' as a cause (Dekker)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex_not", "pattern": "(root cause:?\\s+human\\s+error|human\\s+error\\s+(is the|was the)\\s+(cause|root))", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "blameless_role_labels",
|
||||
"name": "Uses role labels (on-call, engineer, SRE) — preferred over names in narrative",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["on-call", "on call", "the engineer", "the SRE", "the team", "incident commander", "Eng Lead", "PM facilitator"]}
|
||||
},
|
||||
{
|
||||
"id": "detect_prevent_mitigate_categories",
|
||||
"name": "Action items categorized (prevent / detect / mitigate / respond / process)",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["Prevent", "Detect", "Mitigate", "Respond", "Process", "category"]}
|
||||
},
|
||||
{
|
||||
"id": "lessons_learned_section",
|
||||
"name": "Lessons learned section present",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:\\d+\\.\\s+)?Lessons learned", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "no_generic_action_items",
|
||||
"name": "Action items not all vague ('improve monitoring' alone)",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "^\\s*[-*]\\s*Improve monitoring\\s*$", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "allspaw_test_or_signoff",
|
||||
"name": "Allspaw test or sign-off referenced",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["Allspaw", "Allspaw test", "sign-off", "approved by", "Sign-off"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "prfaq",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores an Amazon Working Backwards PR/FAQ against the press release structure, internal FAQ (10-20 Q&A across 9 categories), and external FAQ — anchored to references/red-flags.md and SKILL.md Success Criteria.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "press_release_section",
|
||||
"name": "Part 1 (Press Release) present",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:Part\\s+1[:.]?\\s+)?Press Release", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "internal_faq_section",
|
||||
"name": "Part 2 (Internal FAQ) present",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:Part\\s+2[:.]?\\s+)?Internal FAQ", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "external_faq_section",
|
||||
"name": "Part 3 (External FAQ) present",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(?:Part\\s+3[:.]?\\s+)?External FAQ", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "dateline_present",
|
||||
"name": "Press release has dateline (location + future date)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(FOR IMMEDIATE RELEASE|^[A-Z][a-zA-Z .]+,\\s*[A-Z]{2,}\\s*[-—]\\s*[A-Z][a-z]+\\s+\\d{1,2},\\s*20\\d{2})", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "headline_present",
|
||||
"name": "Press release headline is a customer-facing sentence (### heading)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^###\\s+\\S+", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "customer_quote_present",
|
||||
"name": "Customer quote present (named persona with outcome)",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(said\\s+[A-Z][a-zA-Z .]+,?\\s+(VP|Head|Director|Manager|CEO|Founder|Lead)|[\"'][^\"']{20,}[\"']\\s*[-—]\\s*[A-Z])", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "exec_quote_present",
|
||||
"name": "Leader/exec quote present",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex", "pattern": "(said|according to)\\s+[A-Z][a-zA-Z .]+,\\s+(VP|Chief|CEO|CPO|CTO|Vice President|President|Head)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "magnitude_quantified",
|
||||
"name": "Quantified outcomes in PR (numbers, percentages, time savings)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["minutes", "hours", "%", "$", "from ", "to ", "x faster", "reduction"]}
|
||||
},
|
||||
{
|
||||
"id": "availability_present",
|
||||
"name": "Pricing / availability / how-to-get-started language present",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["available", "pricing", "tier", "today at", "launch", "get started"]}
|
||||
},
|
||||
{
|
||||
"id": "no_buzzwords",
|
||||
"name": "No buzzword stuffing (next-generation, revolutionary, seamless, etc.)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_none", "keywords": ["revolutionary", "next-generation", "seamless", "cutting-edge", "world-class", "best-in-class", "game-changer"]}
|
||||
},
|
||||
{
|
||||
"id": "no_internal_codenames",
|
||||
"name": "No internal codename / project tag in PR headline",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "ProjectCobra|Project [A-Z][a-z]+ |Q[1-4] (deliverable|milestone|roadmap)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "no_vague_audience",
|
||||
"name": "No vague audience ('for businesses', 'for everyone')",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_none", "keywords": ["for businesses everywhere", "for everyone", "for all teams", "for modern teams"]}
|
||||
},
|
||||
{
|
||||
"id": "internal_faq_categories",
|
||||
"name": "Internal FAQ spans required categories (customer demand, business, strategic fit, competition, feasibility, ops, legal/privacy, risk, scope)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_all", "keywords": ["customer", "business", "competit", "feasib", "legal", "risk", "scope"]}
|
||||
},
|
||||
{
|
||||
"id": "internal_faq_min_questions",
|
||||
"name": "Internal FAQ has at least 10 Q&A pairs (counted via 'Q' markers or bolded questions)",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(^\\*\\*Q\\d+|^Q\\d+:|^\\*\\*Q:|\\*\\*Q\\d+:)", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "not_doing_in_v1",
|
||||
"name": "'What we are NOT doing in v1' question answered explicitly",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["explicitly NOT", "not building", "not doing", "out of scope", "deferred", "won't ", "will not "]}
|
||||
},
|
||||
{
|
||||
"id": "cited_evidence",
|
||||
"name": "Quantitative claims have cited evidence (interview counts, % usage, $ARR, MVP data)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["interview", "n=", "MVP", "ARR", "concierge", "research", "survey"]}
|
||||
},
|
||||
{
|
||||
"id": "external_faq_buyer_questions",
|
||||
"name": "External FAQ covers buyer questions (pricing, tier, get started, cancel/refund)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["pricing", "tier", "get started", "cancel", "refund", "available on"]}
|
||||
},
|
||||
{
|
||||
"id": "external_faq_privacy",
|
||||
"name": "External FAQ addresses privacy / data stance",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["privacy", "data", "GDPR", "EU", "secure", "password"]}
|
||||
},
|
||||
{
|
||||
"id": "no_fake_quote_phrases",
|
||||
"name": "No fake-sounding superlative customer quote phrases",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_none", "keywords": ["transform our entire industry", "truly the next generation", "best product ever", "literally life-changing"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"skill": "pricing-prd",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores a tactical pricing PRD against Ramanujam / Van Westendorp / Reforge discipline: pricing model + packaging + WTP research + grandfathering + A/B design + rollback + communication plan + regional considerations.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "summary_present",
|
||||
"name": "Summary section present",
|
||||
"weight": 4,
|
||||
"check": {"type": "section_present", "heading": "Summary"}
|
||||
},
|
||||
{
|
||||
"id": "background_with_market_context",
|
||||
"name": "Background includes market / competitor context",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["competitor", "GitHub Copilot", "Linear", "anchors", "market context", "competing"]}
|
||||
},
|
||||
{
|
||||
"id": "pricing_model_declared",
|
||||
"name": "Pricing model declared (tier / usage / hybrid)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["pricing model", "Tier structure", "usage-based", "per-seat", "Hybrid", "per-conversation", "flat fee", "metered"]}
|
||||
},
|
||||
{
|
||||
"id": "wtp_research_cited",
|
||||
"name": "Willingness-to-pay research cited (Van Westendorp PSM, conjoint, OPP, IPP)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["Van Westendorp", "PSM", "Price Sensitivity Meter", "OPP", "IPP", "Optimal Price Point", "conjoint", "willingness to pay", "willingness-to-pay"]}
|
||||
},
|
||||
{
|
||||
"id": "wtp_segmented",
|
||||
"name": "WTP segmented (not aggregate)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(per segment|segmented|by segment|SMB.*\\$|enterprise.*\\$|segment-specific|per-segment)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "packaging_decisions",
|
||||
"name": "Packaging decisions (tier structure, value carrier, boundaries)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["packaging", "Tier structure", "tier boundaries", "value carrier", "Good/Better/Best", "feature mix"]}
|
||||
},
|
||||
{
|
||||
"id": "grandfathering_policy",
|
||||
"name": "Grandfathering policy explicit",
|
||||
"weight": 8,
|
||||
"check": {"type": "keyword_any", "keywords": ["Grandfathering", "grandfathered", "grandfather policy", "existing customers stay", "price-freeze"]}
|
||||
},
|
||||
{
|
||||
"id": "grandfathering_duration",
|
||||
"name": "Grandfathering duration stated (months/years)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(grandfather[a-z]*\\s+(for\\s+)?\\d+\\s*(month|year)|(\\d+)\\s*(month|year)s?\\s+grandfather)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "ab_test_design",
|
||||
"name": "A/B test design section (hypothesis, primary metric, MDE, holdout)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["A/B test", "A/B design", "experiment design", "hypothesis", "MDE", "holdout"]}
|
||||
},
|
||||
{
|
||||
"id": "ab_primary_metric_arpu",
|
||||
"name": "Primary metric is ARPU / revenue / gross margin (not just conversion)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["ARPU", "revenue per visitor", "gross margin", "net revenue", "LTV"]}
|
||||
},
|
||||
{
|
||||
"id": "ab_sample_or_holdout",
|
||||
"name": "Holdout / sample size specified",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["holdout", "sample size", "5% holdout", "10% holdout", "power", "significance"]}
|
||||
},
|
||||
{
|
||||
"id": "rollback_criteria",
|
||||
"name": "Rollback criteria with numeric thresholds",
|
||||
"weight": 8,
|
||||
"check": {"type": "keyword_any", "keywords": ["Rollback criteria", "rollback threshold", "rollback if", "reverse if", "trigger", "pause if"]}
|
||||
},
|
||||
{
|
||||
"id": "rollback_numeric_threshold",
|
||||
"name": "Rollback thresholds are numeric (%, x baseline)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(>\\s*\\d+\\s*(pp|%|x baseline|points)|drops\\s+>\\s*\\d|\\d+x\\s+baseline)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "counter_metrics_downstream",
|
||||
"name": "Counter / secondary metrics for downstream impact (churn, NPS, retention)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["churn", "NPS", "D30 retention", "D90 retention", "downgrade", "secondary metrics", "counter"]}
|
||||
},
|
||||
{
|
||||
"id": "communication_plan",
|
||||
"name": "Communication plan section present",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["Communication plan", "communication strategy", "customer email", "sales enablement", "support enablement"]}
|
||||
},
|
||||
{
|
||||
"id": "comm_channels_covered",
|
||||
"name": "Communication covers multiple channels (sales / support / in-app / email / pricing page)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_all", "keywords": ["sales", "support"]}
|
||||
},
|
||||
{
|
||||
"id": "legal_reviewed",
|
||||
"name": "Legal / regulatory review referenced",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["Legal", "GDPR", "consumer law", "ToS", "Terms of Service", "EU consumer", "CCPA"]}
|
||||
},
|
||||
{
|
||||
"id": "no_public_enterprise_pricing",
|
||||
"name": "Top tier is sales-led ('Contact sales' or 'Custom') — no public enterprise number",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["Contact sales", "Contact us", "Custom", "sales-led", "Enterprise (custom)", "Enterprise tier", "talk to sales"]}
|
||||
},
|
||||
{
|
||||
"id": "no_per_seat_for_ai",
|
||||
"name": "No naive per-seat-only pricing for an AI product (per-usage / per-conversation considered)",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["per-conversation", "per conversation", "usage-based", "metered", "per-token", "flat fee"]}
|
||||
},
|
||||
{
|
||||
"id": "no_buzzwords",
|
||||
"name": "No marketing buzzwords",
|
||||
"weight": 2,
|
||||
"check": {"type": "keyword_none", "keywords": ["revolutionary", "next-generation", "seamless", "world-class"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "product-vision",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores a Product Vision document against Pichler / Moore / Raskin / Cagan frameworks: 5-10 year horizon, customer-named, differentiated, narrative not statement, with non-goals and bottom-up grounded numbers.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "vision_section_present",
|
||||
"name": "Vision section / heading present",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(^##\\s+(?:The\\s+)?(Product\\s+)?Vision|^###\\s+Vision\\b|^#\\s+.*Vision)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "horizon_5_to_10_years",
|
||||
"name": "Time horizon 3+ years (5-10 preferred) — not 12-month",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "(By\\s+20(2[8-9]|3[0-9])|2029|2030|2031|2032|2034|2036|10[-\\s]year|5[-\\s]year|3[-\\s]year)", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "not_one_year_only",
|
||||
"name": "No 12-month-only horizon (Red Flag 2)",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "(?:^|\\s)Vision\\s+for\\s+(next\\s+year|the\\s+next\\s+12\\s+months|2026\\s*$)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "customer_named",
|
||||
"name": "Specific customer / persona named (not 'everyone')",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["engineer", "finance lead", "EM ", "designer", "Head of", "PM ", "teams of", "applied-AI", "developer", "manager", "Series A"]}
|
||||
},
|
||||
{
|
||||
"id": "not_vague_audience",
|
||||
"name": "No 'for businesses / for everyone' vague audience",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_none", "keywords": ["for everyone", "for businesses everywhere", "modern teams of every size", "every team and every company"]}
|
||||
},
|
||||
{
|
||||
"id": "differentiation_named",
|
||||
"name": "Differentiation from competition stated",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["Unlike", "differentiation", "competitor", "vs", "differs from", "competitive", "what makes us different", "our edge"]}
|
||||
},
|
||||
{
|
||||
"id": "outcome_for_customer",
|
||||
"name": "Outcome for the customer named (verb + result)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["so that", "spend < ", "from ", "to ", "close the books", "save", "recover", "reduce"]}
|
||||
},
|
||||
{
|
||||
"id": "non_goals_present",
|
||||
"name": "Non-goals / what we will NOT do explicitly named (Red Flag 5)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["Non-goals", "non goals", "what we will not", "out of scope", "we will not chase", "we are not", "Excluded:"]}
|
||||
},
|
||||
{
|
||||
"id": "narrative_not_one_line",
|
||||
"name": "Document is a narrative (not just a tagline)",
|
||||
"weight": 5,
|
||||
"check": {"type": "length_in_range", "min": 1000, "max": 30000}
|
||||
},
|
||||
{
|
||||
"id": "framework_used",
|
||||
"name": "At least one named framework used (Pichler / Moore / Raskin / Cagan / Working Backwards)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["Pichler", "Vision Board", "Moore", "Elevator pitch", "Crossing the Chasm", "Raskin", "Strategic narrative", "Cagan", "10-year", "Working Backwards"]}
|
||||
},
|
||||
{
|
||||
"id": "not_mission_statement",
|
||||
"name": "Not a generic mission ('empower every person')",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex_not", "pattern": "(empower\\s+(every|all)|enable\\s+everyone|build\\s+the\\s+future\\s+of\\s+everything)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "business_goals",
|
||||
"name": "Business goals / metrics stated",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["ARR", "Business goals", "market position", "category leadership", "% market"]}
|
||||
},
|
||||
{
|
||||
"id": "needs_or_pains",
|
||||
"name": "Customer needs / pains / jobs named",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Needs", "pain", "job", "JTBD", "jobs", "outcomes the customer", "what customers need"]}
|
||||
},
|
||||
{
|
||||
"id": "numeric_grounding",
|
||||
"name": "Numerical grounding present (TAM, current baseline, customer count)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(\\$\\d|\\d+%|\\d+[kM]\\s*(ARR|users|customers|teams|engineers)|\\d+\\s+paying|\\d+\\s+teams)", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "no_feature_lock",
|
||||
"name": "Does not lock vision to specific product features (Red Flag 9)",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex_not", "pattern": "Vision[:\\s]+.{0,80}(dropdown|button|widget|integration\\s+with\\s+[A-Z][a-z]+)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "no_buzzwords",
|
||||
"name": "No buzzword vision ('next-generation', 'revolutionary')",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_none", "keywords": ["next-generation", "revolutionary", "cutting-edge", "world-class", "best-in-class"]}
|
||||
},
|
||||
{
|
||||
"id": "strategic_narrative_arc",
|
||||
"name": "Strategic narrative arc present (change / stakes / promised land / obstacles)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["promised land", "the change", "obstacles", "the stakes", "what changed", "why now"]}
|
||||
},
|
||||
{
|
||||
"id": "review_or_validation",
|
||||
"name": "Review / validation / stress-test referenced",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["review checklist", "stress-test", "board-reviewed", "tested with", "red-team", "validated"]}
|
||||
},
|
||||
{
|
||||
"id": "no_arbitrary_big_numbers",
|
||||
"name": "Does not present arbitrary big numbers without derivation",
|
||||
"weight": 3,
|
||||
"check": {"type": "regex_not", "pattern": "^\\s*(By 20\\d{2}\\s+we will be\\s+a\\s+\\$\\d+B\\s+(company|ARR))\\s*\\.?\\s*$", "flags": "im"}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "release-notes",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores release notes against the 5-category structure (Features / Improvements / Fixes / Breaking / Deprecations) with value framing per item, plain language (no raw PR/ticket IDs in user-facing copy), and date + version present.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "version_number",
|
||||
"name": "Version number present (vX.Y or similar)",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(v\\d+\\.\\d+(\\.\\d+)?|Version\\s+\\d+\\.\\d+|Release\\s+\\d+\\.\\d+)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "release_date",
|
||||
"name": "Release date present",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(Release\\s+Date|Released|released\\s+on|20\\d{2}-\\d{2}-\\d{2}|[A-Z][a-z]+\\s+\\d{1,2},\\s+20\\d{2})", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "new_features_section",
|
||||
"name": "New Features section present",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(New\\s+)?Features?", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "improvements_section",
|
||||
"name": "Improvements section present",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+Improvements", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "bug_fixes_section",
|
||||
"name": "Bug Fixes section present",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(Bug\\s+)?Fixes", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "breaking_changes_section",
|
||||
"name": "Breaking Changes section present or no breaking changes",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Breaking Changes", "Breaking change", "Action Required", "ACTION REQUIRED", "breaking changes", "no breaking changes"]}
|
||||
},
|
||||
{
|
||||
"id": "deprecations_section_or_none",
|
||||
"name": "Deprecations section present (or no deprecations)",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["Deprecation", "Planned Removal", "deprecated", "will be removed", "no deprecations"]}
|
||||
},
|
||||
{
|
||||
"id": "value_framing_per_item",
|
||||
"name": "Items lead with user value (now / can / faster / lets you / reduces)",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_any", "keywords": ["now ", "can now", "lets you", "you can", "faster", "reduces", "no more", "available", "supported"]}
|
||||
},
|
||||
{
|
||||
"id": "no_raw_ticket_titles",
|
||||
"name": "No raw 'SEG-NNNN Refactor of X' titles as bullets in user-facing copy",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex_not", "pattern": "^\\s*[-*]\\s*[A-Z]{2,5}-\\d+\\s+(Refactor|Migrate|Migrated|Refactored)", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "no_internal_jargon",
|
||||
"name": "Limited internal jargon (no 'null pointer exception', 'context manager', etc.)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_none", "keywords": ["null pointer exception", "stack trace", "context manager", "race condition fix", "WebSocket reconnection logic"]}
|
||||
},
|
||||
{
|
||||
"id": "no_marketing_hype",
|
||||
"name": "No empty marketing copy ('revolutionary', 'game-changing')",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_none", "keywords": ["revolutionary", "game-changing", "next-generation", "powered by AI", "groundbreaking"]}
|
||||
},
|
||||
{
|
||||
"id": "breaking_action_required",
|
||||
"name": "Breaking changes have explicit Action Required + deadline/migration",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["Action Required", "ACTION REQUIRED", "Migrate", "migration guide", "before ", "will be removed", "by April", "by June", "by May"]}
|
||||
},
|
||||
{
|
||||
"id": "specific_features_described",
|
||||
"name": "Features described concretely (not abstract 'AI-powered insights')",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["dashboard", "export", "settings", "search", "filter", "report", "shortcut", "view", "page", "API", "endpoint"]}
|
||||
},
|
||||
{
|
||||
"id": "no_internal_oncall_in_copy",
|
||||
"name": "No 'on-call', 'rollback', 'known issue: internal' in customer-facing copy",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_none", "keywords": ["on-call: @", "rollback in progress", "known issue: internal", "Sev1", "incident channel"]}
|
||||
},
|
||||
{
|
||||
"id": "categorized_items",
|
||||
"name": "Items categorized (not a flat list)",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+\\w", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "structured_with_bullets",
|
||||
"name": "Items presented as a structured list",
|
||||
"weight": 4,
|
||||
"check": {"type": "has_list", "min_items": 3}
|
||||
},
|
||||
{
|
||||
"id": "product_name_present",
|
||||
"name": "Product / release name present in header",
|
||||
"weight": 3,
|
||||
"check": {"type": "regex", "pattern": "^#\\s+\\S+", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "no_pure_pr_numbers",
|
||||
"name": "No 'PR-1234' raw references as the only bullet content",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex_not", "pattern": "^\\s*[-*]\\s*PR-?\\d+\\s*$", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "user_facing_language",
|
||||
"name": "User-facing language used (you / your / customer / users)",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["you ", "your ", "users", "customers", "team", "our customers"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "roadmap-communication",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores a 3-variant roadmap (executive / customer / internal) against Now/Next/Later structure, outcome framing, confidence ratings, audience-appropriate detail, and absence of date-promised commitments.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "three_variants_named",
|
||||
"name": "Three audience variants present (executive / customer / internal)",
|
||||
"weight": 8,
|
||||
"check": {"type": "keyword_all", "keywords": ["executive", "customer", "internal"]}
|
||||
},
|
||||
{
|
||||
"id": "executive_variant_section",
|
||||
"name": "Executive variant section present",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "^#{2,4}\\s+(?:Variant\\s+\\d:?\\s+)?(Executive|Exec|Board)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "customer_variant_section",
|
||||
"name": "Customer variant section present",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "^#{2,4}\\s+(?:Variant\\s+\\d:?\\s+)?Customer\\s+(roadmap|view|variant|blog|post|view)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "internal_variant_section",
|
||||
"name": "Internal variant section present",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "^#{2,4}\\s+(?:Variant\\s+\\d:?\\s+)?(Internal|Engineering)\\s+(roadmap|view|variant)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "now_next_later",
|
||||
"name": "Now / Next / Later structure used",
|
||||
"weight": 7,
|
||||
"check": {"type": "keyword_all", "keywords": ["Now", "Next", "Later"]}
|
||||
},
|
||||
{
|
||||
"id": "confidence_levels",
|
||||
"name": "Confidence levels (H/M/L or risk register) per item",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(Confidence[:\\s]+(High|Med|Low|HIGH|MED|LOW|H/M/L)|HIGH|MEDIUM|LOW|L/M/H|risk register|H | M | L)", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "themes_present",
|
||||
"name": "Items bucketed under themes",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Theme:", "*Theme*", "Themes", "*Activation*", "Enterprise readiness", "themed cards", "theme-led"]}
|
||||
},
|
||||
{
|
||||
"id": "outcome_oriented_exec",
|
||||
"name": "Outcome framing in executive variant (outcomes, not just features)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["Outcome:", "Outcomes", "outcome-based", "outcome-led", "outcome metric", "named outcome", "outcome per"]}
|
||||
},
|
||||
{
|
||||
"id": "no_dates_in_customer",
|
||||
"name": "Customer variant avoids specific date promises beyond current quarter",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex_not", "pattern": "(Customer roadmap|Customer view|Customer variant)[\\s\\S]{0,500}(SCIM|SAML|SOC ?2)[:\\s]+(January|February|March|April|May|June|July|August|September|October|November|December)\\s+\\d{1,2},\\s+20\\d{2}", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "directional_language_customer",
|
||||
"name": "Customer variant uses directional language ('later this year', 'mid-half', 'early next year')",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["exploring", "in development", "shipping soon", "later this year", "early 2027", "mid-half", "early next year", "directional", "may change", "approximations"]}
|
||||
},
|
||||
{
|
||||
"id": "internal_has_owners",
|
||||
"name": "Internal variant has explicit owners (PM/EM/Lead)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["PM:", "EM:", "Owner:", "Lead:", "Design lead", "Eng lead"]}
|
||||
},
|
||||
{
|
||||
"id": "internal_has_dependencies",
|
||||
"name": "Internal variant has explicit dependencies / risks",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["dependencies", "Dependencies", "depends on", "Risk:", "risks", "mitigation", "Blocked by"]}
|
||||
},
|
||||
{
|
||||
"id": "not_doing_section",
|
||||
"name": "Variants name what is NOT being done",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["NOT doing", "Not doing", "not working on", "explicitly not", "out of scope", "trade-offs", "tradeoffs", "deferred", "not building", "we are *not*"]}
|
||||
},
|
||||
{
|
||||
"id": "no_dates_as_commitments",
|
||||
"name": "Customer variant deliberately avoids fixed customer-facing dates",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["planning approximations", "may shift", "directional", "no committed dates", "without committing", "later this year", "early next year", "no committed dates outside", "no committed dates"]}
|
||||
},
|
||||
{
|
||||
"id": "triangulation_or_consistency",
|
||||
"name": "Triangulation / consistency mechanism between variants",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_any", "keywords": ["triangulation", "consistent identity", "same ID", "trace", "source of truth"]}
|
||||
},
|
||||
{
|
||||
"id": "visualization_or_format",
|
||||
"name": "Visualization / format (mermaid, swimlanes, table)",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex", "pattern": "(```mermaid|graph\\s+(TD|LR)|swimlane|themed card|\\|---)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "executive_outcome_metric",
|
||||
"name": "Executive variant has outcome metric + target",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Expected outcome", "outcome metric", "metric target", "target ARR", "target:", "expected outcome", "Composite goal"]}
|
||||
},
|
||||
{
|
||||
"id": "internal_kr_or_dep_graph",
|
||||
"name": "Internal variant has KR/OKR mapping OR explicit dependency graph",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["KR mapped", "KR:", "OKR", "Key Result", "Dependencies", "dependency graph", "Critical-path", "critical path"]}
|
||||
},
|
||||
{
|
||||
"id": "no_marketing_buzzwords",
|
||||
"name": "No internal marketing buzzwords across variants",
|
||||
"weight": 2,
|
||||
"check": {"type": "keyword_none", "keywords": ["revolutionary", "next-generation", "world-class", "leveraging", "synergizing"]}
|
||||
}
|
||||
]
|
||||
}
|
||||
+175
@@ -0,0 +1,175 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
run.py — Runner for the PM eval harness.
|
||||
|
||||
Loads rubrics from evals/<skill>/rubric.json and scores worked-example
|
||||
artifacts under project-management/**/examples/*.md.
|
||||
|
||||
Usage:
|
||||
python evals/run.py --all
|
||||
python evals/run.py --skill create-prd
|
||||
python evals/run.py --skill create-prd --artifact path/to/file.md
|
||||
python evals/run.py --all --format json
|
||||
python evals/run.py --all --threshold 70
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Local import
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from engine import evaluate_artifact # noqa: E402
|
||||
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
EVALS_DIR = Path(__file__).resolve().parent
|
||||
PM_DIR = REPO / "project-management"
|
||||
|
||||
|
||||
def find_rubric(skill: str) -> Path | None:
|
||||
p = EVALS_DIR / skill / "rubric.json"
|
||||
return p if p.exists() else None
|
||||
|
||||
|
||||
def all_rubrics() -> list[tuple[str, Path]]:
|
||||
rubrics = []
|
||||
for sub in sorted(EVALS_DIR.iterdir()):
|
||||
if sub.is_dir() and (sub / "rubric.json").exists():
|
||||
rubrics.append((sub.name, sub / "rubric.json"))
|
||||
return rubrics
|
||||
|
||||
|
||||
def find_skill_dir(skill_name: str) -> Path | None:
|
||||
"""Find the skill folder under project-management/ (handles nested sub-areas)."""
|
||||
# Direct match
|
||||
direct = PM_DIR / skill_name
|
||||
if direct.is_dir() and (direct / "SKILL.md").exists():
|
||||
return direct
|
||||
# Under subareas
|
||||
for sub in ("discovery", "execution", "career"):
|
||||
candidate = PM_DIR / sub / skill_name
|
||||
if candidate.is_dir() and (candidate / "SKILL.md").exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def find_examples(skill_name: str) -> list[Path]:
|
||||
skill_dir = find_skill_dir(skill_name)
|
||||
if not skill_dir:
|
||||
return []
|
||||
examples_dir = skill_dir / "examples"
|
||||
if not examples_dir.is_dir():
|
||||
return []
|
||||
return sorted(p for p in examples_dir.glob("*.md") if not p.name.startswith("README"))
|
||||
|
||||
|
||||
def score_one(skill_name: str, rubric_path: Path, artifacts: list[Path]) -> list[dict]:
|
||||
rubric = json.loads(rubric_path.read_text(encoding="utf-8"))
|
||||
results: list[dict] = []
|
||||
for art in artifacts:
|
||||
text = art.read_text(encoding="utf-8")
|
||||
result = evaluate_artifact(rubric, text)
|
||||
result["artifact"] = str(art.relative_to(REPO))
|
||||
results.append(result)
|
||||
return results
|
||||
|
||||
|
||||
def render_markdown(all_results: list[dict], threshold: int) -> str:
|
||||
lines = ["# PM Eval Report", ""]
|
||||
if not all_results:
|
||||
lines.append("_No results._")
|
||||
return "\n".join(lines)
|
||||
|
||||
# Aggregate
|
||||
total = len(all_results)
|
||||
passed_n = sum(1 for r in all_results if r["score"] >= threshold)
|
||||
lines.append(f"**{passed_n}/{total} artifacts at or above threshold ({threshold}/100).**")
|
||||
lines.append("")
|
||||
lines.append("## Summary")
|
||||
lines.append("")
|
||||
lines.append("| Skill | Artifact | Score | Pass? |")
|
||||
lines.append("|---|---|---:|:---:|")
|
||||
for r in all_results:
|
||||
skill = r["skill"]
|
||||
path = r["artifact"]
|
||||
score = r["score"]
|
||||
ok = "✓" if score >= threshold else "✗"
|
||||
lines.append(f"| {skill} | `{path}` | {score} | {ok} |")
|
||||
lines.append("")
|
||||
|
||||
# Per-artifact detail
|
||||
for r in all_results:
|
||||
lines.append(f"## {r['skill']} — `{r['artifact']}`")
|
||||
lines.append(f"Score: **{r['score']}/100** (raw {r['raw_score']}/{r['max_score']}) ")
|
||||
lines.append(f"Passed {r['passed']}/{r['passed']+r['failed']} criteria.")
|
||||
lines.append("")
|
||||
if r["failed"]:
|
||||
lines.append("### Failed criteria")
|
||||
lines.append("")
|
||||
for row in r["results"]:
|
||||
if not row["passed"]:
|
||||
lines.append(f"- **{row['name']}** (weight {row['weight']}) — {row['detail']}")
|
||||
lines.append("")
|
||||
if r["passed"]:
|
||||
lines.append("<details><summary>Passed criteria</summary>")
|
||||
lines.append("")
|
||||
for row in r["results"]:
|
||||
if row["passed"]:
|
||||
lines.append(f"- {row['name']} (weight {row['weight']}) — {row['detail']}")
|
||||
lines.append("")
|
||||
lines.append("</details>")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description="Run PM skill output evaluations.")
|
||||
g = p.add_mutually_exclusive_group(required=True)
|
||||
g.add_argument("--all", action="store_true", help="Run all rubrics")
|
||||
g.add_argument("--skill", help="Run a single skill's rubric")
|
||||
p.add_argument("--artifact", help="Explicit artifact path (overrides examples/ lookup)")
|
||||
p.add_argument("--format", choices=("markdown", "json"), default="markdown")
|
||||
p.add_argument("--threshold", type=int, default=70, help="Pass threshold (default 70)")
|
||||
p.add_argument("--output", help="Write to file instead of stdout")
|
||||
args = p.parse_args()
|
||||
|
||||
all_results: list[dict] = []
|
||||
skills_to_run: list[tuple[str, Path]] = []
|
||||
|
||||
if args.all:
|
||||
skills_to_run = all_rubrics()
|
||||
if not skills_to_run:
|
||||
sys.exit("ERROR: no rubrics found under evals/")
|
||||
else:
|
||||
rubric_path = find_rubric(args.skill)
|
||||
if not rubric_path:
|
||||
sys.exit(f"ERROR: no rubric for skill '{args.skill}' at evals/{args.skill}/rubric.json")
|
||||
skills_to_run = [(args.skill, rubric_path)]
|
||||
|
||||
for skill_name, rubric_path in skills_to_run:
|
||||
if args.artifact:
|
||||
arts = [Path(args.artifact)]
|
||||
else:
|
||||
arts = find_examples(skill_name)
|
||||
if not arts:
|
||||
print(f"[warn] no examples for {skill_name}", file=sys.stderr)
|
||||
continue
|
||||
results = score_one(skill_name, rubric_path, arts)
|
||||
all_results.extend(results)
|
||||
|
||||
if args.format == "json":
|
||||
out = json.dumps({"results": all_results, "threshold": args.threshold}, indent=2) + "\n"
|
||||
else:
|
||||
out = render_markdown(all_results, args.threshold)
|
||||
|
||||
if args.output:
|
||||
Path(args.output).write_text(out, encoding="utf-8")
|
||||
else:
|
||||
sys.stdout.write(out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"skill": "status-update-generator",
|
||||
"version": "1.0.0",
|
||||
"description": "Scores a weekly executive status update against the 6-section template (Header + Highlights / Blockers / Risks / Asks / What's Next) with R/Y/G traffic light and structured risk/ask formats.",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "header_period",
|
||||
"name": "Header has period / date range",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["Period:", "Week of", "Week of 20"]}
|
||||
},
|
||||
{
|
||||
"id": "header_author",
|
||||
"name": "Header has author / team",
|
||||
"weight": 4,
|
||||
"check": {"type": "keyword_any", "keywords": ["Author:", "Team:", "Owner:", "PM:"]}
|
||||
},
|
||||
{
|
||||
"id": "traffic_light_present",
|
||||
"name": "Traffic-light status (R/Y/G) declared",
|
||||
"weight": 7,
|
||||
"check": {"type": "regex", "pattern": "\\b(Status[:\\s]+)?(RED|YELLOW|GREEN|Red|Yellow|Green)\\b", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "status_rationale",
|
||||
"name": "Status rationale sentence explicit",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_any", "keywords": ["rationale:", "Status rationale", "reason:", "why this color"]}
|
||||
},
|
||||
{
|
||||
"id": "highlights_section",
|
||||
"name": "Highlights section present",
|
||||
"weight": 6,
|
||||
"check": {"type": "section_present", "heading": "Highlights"}
|
||||
},
|
||||
{
|
||||
"id": "blockers_section",
|
||||
"name": "Blockers section present",
|
||||
"weight": 6,
|
||||
"check": {"type": "section_present", "heading": "Blockers"}
|
||||
},
|
||||
{
|
||||
"id": "risks_section",
|
||||
"name": "Risks section present",
|
||||
"weight": 6,
|
||||
"check": {"type": "section_present", "heading": "Risks"}
|
||||
},
|
||||
{
|
||||
"id": "asks_section",
|
||||
"name": "Asks section present (mandatory, even if 'none this week')",
|
||||
"weight": 8,
|
||||
"check": {"type": "section_present", "heading": "Asks"}
|
||||
},
|
||||
{
|
||||
"id": "whats_next_section",
|
||||
"name": "What's Next section present",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "^##\\s+(What['s]+|What is)\\s+next", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "highlights_outcome_led",
|
||||
"name": "Highlights are outcome-led (not raw 'Shipped PR-NNNN' tickets only)",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex_not", "pattern": "^\\s*[-*]\\s*Shipped\\s+PR-\\d+\\s*$", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "highlights_quantified",
|
||||
"name": "Highlights contain numerical anchors (%, ms, $, counts)",
|
||||
"weight": 6,
|
||||
"check": {"type": "regex", "pattern": "(\\d+%|\\d+\\s?ms|\\$\\d+|\\d+ of \\d+|\\d+x )", "flags": ""}
|
||||
},
|
||||
{
|
||||
"id": "blocker_has_what_who_need",
|
||||
"name": "Blockers structured (what / blocked by / need)",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_all", "keywords": ["block", "need"]}
|
||||
},
|
||||
{
|
||||
"id": "risk_owner_due",
|
||||
"name": "Risks have Owner + Due date",
|
||||
"weight": 6,
|
||||
"check": {"type": "keyword_all", "keywords": ["Owner", "Due"]}
|
||||
},
|
||||
{
|
||||
"id": "risk_likelihood_impact",
|
||||
"name": "Risks include Likelihood x Impact rating",
|
||||
"weight": 4,
|
||||
"check": {"type": "regex", "pattern": "(L[:\\s]*[HML]|Likelihood[:\\s]*[HML]|[HML][:\\s]*x\\s*[HML]|Likelihood.*Impact)", "flags": "im"}
|
||||
},
|
||||
{
|
||||
"id": "ask_decision_maker",
|
||||
"name": "Asks name decision-maker / from-whom",
|
||||
"weight": 5,
|
||||
"check": {"type": "keyword_any", "keywords": ["VP", "Director", "Head of", "from whom", "from:", "decision needed", "Decision needed"]}
|
||||
},
|
||||
{
|
||||
"id": "ask_by_when",
|
||||
"name": "Asks have by-when date",
|
||||
"weight": 5,
|
||||
"check": {"type": "regex", "pattern": "(by\\s+(Monday|Tuesday|Wednesday|Thursday|Friday)|by\\s+EOD|by\\s+20\\d{2}-\\d{2}-\\d{2}|by\\s+[A-Z][a-z]+\\s+\\d|Decision needed by)", "flags": "i"}
|
||||
},
|
||||
{
|
||||
"id": "no_emotional_drama",
|
||||
"name": "No internal drama / emotional language",
|
||||
"weight": 3,
|
||||
"check": {"type": "keyword_none", "keywords": ["frustrated with", "annoying", "pissed", "fed up", "lack of engagement"]}
|
||||
},
|
||||
{
|
||||
"id": "no_pure_activity_highlight",
|
||||
"name": "Highlights not pure activity (held kickoff / had meeting / discussed)",
|
||||
"weight": 3,
|
||||
"check": {"type": "regex_not", "pattern": "^\\s*[-*]\\s*(Held|Had|Discussed|Started\\s+working|Began\\s+)", "flags": "m"}
|
||||
},
|
||||
{
|
||||
"id": "one_page_density",
|
||||
"name": "Update fits a reasonable length (under 4000 words)",
|
||||
"weight": 3,
|
||||
"check": {"type": "length_in_range", "min": 200, "max": 15000}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -196,6 +196,27 @@ A few highlight per-skill examples:
|
||||
|
||||
---
|
||||
|
||||
## Output evaluation harness
|
||||
|
||||
Twelve of the artifact-generating PM skills now have **deterministic eval rubrics** at [`evals/`](../evals/) — scores any markdown artifact 0-100 against the skill's red flags and success criteria. No LLM in the loop, stdlib only.
|
||||
|
||||
```bash
|
||||
# Score all worked examples
|
||||
python evals/run.py --all --threshold 70
|
||||
|
||||
# Score one skill against its own example
|
||||
python evals/run.py --skill post-mortem
|
||||
|
||||
# Score a custom artifact
|
||||
python evals/run.py --skill create-prd --artifact ./my-prd.md
|
||||
```
|
||||
|
||||
Skills with rubrics: `create-prd`, `prfaq`, `ai-feature-prd`, `brainstorm-okrs`, `status-update-generator`, `post-mortem`, `north-star-metric`, `product-vision`, `pricing-prd`, `roadmap-communication`, `release-notes`, `customer-feedback-triage`.
|
||||
|
||||
Use case: PMs paste their drafted artifact and get back specific failures ("Summary is 12 sentences, need 2-5", "Section 5 Market Segments not found", "Found forbidden buzzword: synergy"). Faster than a human review, catches form failures consistently.
|
||||
|
||||
---
|
||||
|
||||
## Red flags (anti-pattern library) per skill
|
||||
|
||||
Every PM skill now ships with a `references/red-flags.md` file: 10-12 concrete anti-patterns with paired *bad* and *good* artifact snippets, plus a "How to catch it" check question. Use it before sharing your output to scan for the most common failures.
|
||||
|
||||
Reference in New Issue
Block a user