mirror of
https://github.com/mastra-ai/skills.git
synced 2026-09-14 18:02:06 +08:00
Revert to helpful skill descriptions and remove test infrastructure
- Updated all three skill descriptions to be comprehensive and helpful - Removed subagent testing approach (proved ineffective) - Skills are user-invocable reference material, not auto-invoked - Added .gitignore for .claude/ directory Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Ehindero Israel
@@ -0,0 +1 @@
|
||||
.claude/
|
||||
-344
@@ -1,344 +0,0 @@
|
||||
# Mastra Skills Test Cases
|
||||
|
||||
Complete list of 50 test cases for validating AI agent skill invocation.
|
||||
|
||||
---
|
||||
|
||||
## Test Format
|
||||
|
||||
Each test includes:
|
||||
- **ID**: Test case identifier (TC-XX)
|
||||
- **Input**: User query exactly as asked
|
||||
- **Expected Skill**: Which skill should be invoked
|
||||
- **Category**: Test category
|
||||
- **Key Validation**: What to check in the response
|
||||
|
||||
---
|
||||
|
||||
## Setup & Installation (create-mastra skill)
|
||||
|
||||
### TC-01: New Project Setup
|
||||
**Input:** "How do I start a new Mastra project?"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Mentions `npx create-mastra`
|
||||
|
||||
### TC-02: Add to Next.js
|
||||
**Input:** "I have a Next.js app. How do I add Mastra to it?"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Shows `npm install @mastra/core` + ES2022 config
|
||||
|
||||
### TC-03: Add to Express
|
||||
**Input:** "Can I use Mastra with Express? How do I set it up?"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Express integration pattern
|
||||
|
||||
### TC-04: Installation Errors
|
||||
**Input:** "I'm getting 'Module not found' when importing from @mastra/core"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Troubleshooting steps (≥3 suggestions)
|
||||
|
||||
### TC-05: TypeScript Configuration
|
||||
**Input:** "What TypeScript settings does Mastra need?"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Explicitly mentions "ES2022" requirement
|
||||
|
||||
### TC-06: Environment Variables
|
||||
**Input:** "How do I set up my API keys for Mastra?"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** .env setup with process.env usage
|
||||
|
||||
### TC-07: Standalone Script
|
||||
**Input:** "I just want to run a simple Mastra script, not a full app"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Minimal setup example
|
||||
|
||||
### TC-08: Monorepo Setup
|
||||
**Input:** "How do I add Mastra to my monorepo workspace?"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Package dependencies discussion
|
||||
|
||||
### TC-09: Framework Choice
|
||||
**Input:** "Should I use the CLI or manual setup for my Mastra project?"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Decision tree or comparison
|
||||
|
||||
### TC-10: Deployment Preparation
|
||||
**Input:** "What do I need to configure before deploying my Mastra app?"
|
||||
**Expected Skill:** create-mastra
|
||||
**Key Check:** Environment and build config
|
||||
|
||||
---
|
||||
|
||||
## Agent Development (mastra skill)
|
||||
|
||||
### TC-11: First Agent
|
||||
**Input:** "Create a basic AI agent that can answer questions"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** `new Agent()`, provider/name format, no deprecated APIs
|
||||
|
||||
### TC-12: Model Selection
|
||||
**Input:** "How do I create an agent using GPT-4?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** `model: { provider: 'openai', name: 'gpt-4o' }`
|
||||
|
||||
### TC-13: Agent with Tools
|
||||
**Input:** "Build an agent that can search the web and check weather"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** ≥2 tools defined, Zod schemas, tools registered
|
||||
|
||||
### TC-14: Custom Instructions
|
||||
**Input:** "Make an agent that always responds in a formal tone"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Shows `instructions` parameter
|
||||
|
||||
### TC-15: Multi-Agent System
|
||||
**Input:** "I need multiple agents that work together"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Multiple agent registration
|
||||
|
||||
### TC-16: Agent with Memory
|
||||
**Input:** "Create an agent that remembers previous conversations"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Memory setup (Postgres/LibSQL), threadId usage
|
||||
|
||||
### TC-17: Structured Output
|
||||
**Input:** "I need an agent that returns JSON in a specific format"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Zod outputSchema, type-safe usage
|
||||
|
||||
### TC-18: Error Handling
|
||||
**Input:** "How do I handle when an agent call fails?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Error handling patterns (try/catch)
|
||||
|
||||
### TC-19: Streaming Responses
|
||||
**Input:** "Can I stream the agent's response as it's generated?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Streaming pattern
|
||||
|
||||
### TC-20: Agent with Context
|
||||
**Input:** "Pass user profile data to my agent on every call"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Context passing mechanism
|
||||
|
||||
### TC-21: Testing Agents
|
||||
**Input:** "How do I test my Mastra agent?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Testing patterns/approach
|
||||
|
||||
### TC-22: Cost Tracking
|
||||
**Input:** "I want to track how much each agent call costs"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Usage tracking discussion
|
||||
|
||||
### TC-23: Model Comparison
|
||||
**Input:** "Should I use Claude or GPT-4 for my agent?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Model comparison/guidance
|
||||
|
||||
### TC-24: Local Models
|
||||
**Input:** "Can I use Ollama models with Mastra agents?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Local model integration
|
||||
|
||||
### TC-25: Agent Timeout
|
||||
**Input:** "My agent calls are timing out, how do I fix this?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Timeout configuration
|
||||
|
||||
---
|
||||
|
||||
## Workflows (mastra skill)
|
||||
|
||||
### TC-26: Basic Workflow
|
||||
**Input:** "Create a workflow that processes data in steps"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Sequential workflow, ≥2 steps, state passing
|
||||
|
||||
### TC-27: Branching Workflow
|
||||
**Input:** "I need a workflow that takes different paths based on conditions"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Conditional execution (`when` parameter)
|
||||
|
||||
### TC-28: Parallel Execution
|
||||
**Input:** "Run multiple tasks at the same time in my workflow"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Parallel workflow pattern
|
||||
|
||||
### TC-29: Workflow with Loop
|
||||
**Input:** "Process an array of items in my workflow"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Foreach pattern
|
||||
|
||||
### TC-30: Workflow State
|
||||
**Input:** "How do I pass data between workflow steps?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** State management (inputData/return)
|
||||
|
||||
### TC-31: Suspend and Resume
|
||||
**Input:** "Pause my workflow and wait for user input"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Suspend/resume mechanism
|
||||
|
||||
### TC-32: Workflow Error Recovery
|
||||
**Input:** "What happens if a step in my workflow fails?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Error handling in workflows
|
||||
|
||||
### TC-33: Long-Running Workflow
|
||||
**Input:** "My workflow takes hours to run, how do I track progress?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Workflow persistence discussion
|
||||
|
||||
### TC-34: Workflow Testing
|
||||
**Input:** "How can I test my workflow without running all steps?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Testing approach
|
||||
|
||||
### TC-35: Dynamic Workflow
|
||||
**Input:** "Build a workflow where steps are determined at runtime"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Dynamic step creation
|
||||
|
||||
---
|
||||
|
||||
## Tools & Integrations (mastra skill)
|
||||
|
||||
### TC-36: Custom Tool
|
||||
**Input:** "Create a tool that calls my internal API"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Tool structure (id, description, inputSchema, execute)
|
||||
|
||||
### TC-37: Context-Aware Tool
|
||||
**Input:** "My tool needs access to user session data"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Context-aware tool pattern
|
||||
|
||||
### TC-38: Suspending Tool
|
||||
**Input:** "Create a tool that pauses execution for user approval"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Uses `suspend` function
|
||||
|
||||
### TC-39: Tool Validation
|
||||
**Input:** "How do I validate inputs to my tool?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Zod schema in inputSchema
|
||||
|
||||
### TC-40: Multiple Tools
|
||||
**Input:** "Give my agent access to several different tools"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Multiple tool registration
|
||||
|
||||
---
|
||||
|
||||
## Storage & Memory (mastra skill)
|
||||
|
||||
### TC-41: Postgres Setup
|
||||
**Input:** "Connect Mastra to my Postgres database"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** PostgresMemory, connectionString
|
||||
|
||||
### TC-42: LibSQL Setup
|
||||
**Input:** "I want to use Turso for storage"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** LibSQL storage pattern
|
||||
|
||||
### TC-43: Conversation History
|
||||
**Input:** "Store and retrieve chat history for my agent"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Message history pattern
|
||||
|
||||
### TC-44: Semantic Memory
|
||||
**Input:** "Let my agent remember facts and recall them later"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** memorize/recall methods
|
||||
|
||||
### TC-45: Working Memory
|
||||
**Input:** "Temporary storage during agent execution"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Working memory pattern
|
||||
|
||||
---
|
||||
|
||||
## RAG & Vector Search (mastra skill)
|
||||
|
||||
### TC-46: Basic RAG
|
||||
**Input:** "Build an agent that answers questions from my documents"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** RAG patterns
|
||||
|
||||
### TC-47: Vector Query Tool
|
||||
**Input:** "Create a tool that searches my vector database"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Vector query tool pattern
|
||||
|
||||
### TC-48: Graph RAG
|
||||
**Input:** "Implement knowledge graph RAG with Mastra"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Graph RAG pattern
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting & Migration (mastra skills)
|
||||
|
||||
### TC-49: V1 Migration
|
||||
**Input:** "I'm upgrading from Mastra v0.1, what changed?"
|
||||
**Expected Skill:** mastra
|
||||
**Key Check:** Migration guide reference
|
||||
|
||||
### TC-50: API Verification
|
||||
**Input:** "What parameters does createAgent accept?"
|
||||
**Expected Skill:** mastra-embedded-docs-look-up
|
||||
**Key Check:** References embedded docs (node_modules/@mastra/*/dist/docs/)
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Quick Test (9 samples)
|
||||
Run: TC-01, TC-02, TC-11, TC-13, TC-16, TC-26, TC-36, TC-41, TC-50
|
||||
|
||||
### Full Test (all 50)
|
||||
Run all test cases above
|
||||
|
||||
### Validation Criteria
|
||||
|
||||
For each test, check:
|
||||
1. ✅ **Skill Invoked**: Did agent call Skill tool?
|
||||
2. ✅ **Correct Skill**: Was the right skill name used?
|
||||
3. ✅ **Pattern Accuracy**: Does code follow Mastra patterns?
|
||||
4. ✅ **API Correctness**: No deprecated APIs, correct imports?
|
||||
5. ✅ **Convention Adherence**: ES2022, provider/name format, etc?
|
||||
|
||||
### Success Threshold
|
||||
|
||||
- **Critical**: 100% on skill invocation + correct skill name
|
||||
- **Overall**: ≥95% of tests pass all validation criteria
|
||||
|
||||
---
|
||||
|
||||
## Test Categories
|
||||
|
||||
| Category | Count | Skill |
|
||||
|----------|-------|-------|
|
||||
| Setup & Installation | 10 | create-mastra |
|
||||
| Agent Development | 15 | mastra |
|
||||
| Workflows | 10 | mastra |
|
||||
| Tools & Integrations | 5 | mastra |
|
||||
| Storage & Memory | 5 | mastra |
|
||||
| RAG & Vector Search | 3 | mastra |
|
||||
| Troubleshooting | 2 | mastra + embedded-docs-look-up |
|
||||
| **Total** | **50** | |
|
||||
|
||||
---
|
||||
|
||||
## Results History
|
||||
|
||||
See `test-results-full.md` for detailed test run results.
|
||||
|
||||
**Latest:** 100% skill invocation rate (50/50 tests)
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2026-01-29
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: create-mastra
|
||||
description: "**REQUIRED for all Mastra setup/installation** - ALWAYS invoke when user mentions: start project, install, add to app, setup, configure, environment, deployment, TypeScript config, ES2022, errors (module not found, import errors), troubleshooting installation, simple/minimal/standalone script, monorepo, CLI vs manual. Use BEFORE any installation/setup advice or code."
|
||||
description: "Complete project setup guide for Mastra AI framework. Covers: CLI installation (create-mastra), manual installation, TypeScript configuration (ES2022 requirements), environment setup, adding to existing projects (Next.js, Express), troubleshooting (module not found, import errors, CommonJS issues), deployment preparation, and monorepo configuration. Includes both quick setup (user runs CLI) and automatic setup (step-by-step file creation). Use this skill for all Mastra installation and setup questions to ensure correct configuration and prevent common errors."
|
||||
license: Apache-2.0
|
||||
metadata:
|
||||
author: Mastra
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: mastra-embedded-docs-look-up
|
||||
# prettier-ignore
|
||||
description: "**REQUIRED for API signature verification** - ALWAYS invoke when user asks: 'What parameters does X accept?', 'How do I call X?', 'What's the API for X?'. Use to verify current API signatures from embedded docs in node_modules/@mastra/*/dist/docs/ instead of relying on training knowledge. Prevents API hallucination."
|
||||
description: "Look up current API signatures from installed Mastra packages in node_modules/@mastra/*/dist/docs/. Use this skill to verify exact parameters, constructor arguments, and function signatures that match the installed version. Mastra APIs evolve rapidly - embedded docs ensure accuracy. Covers: Agent constructor parameters, Workflow API, Tool API, Memory configuration, Storage options, and all @mastra/* package exports. Returns actual TypeScript definitions and JSDoc from installed packages. Always use for API verification questions to prevent outdated answers."
|
||||
license: Apache-2.0
|
||||
metadata:
|
||||
author: Mastra
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: mastra
|
||||
description: "**REQUIRED for all Mastra code and questions** - ALWAYS invoke when user mentions: agents, workflows, tools, memory, storage, RAG. Covers: create/build/make (any agent/workflow/tool), how to/how do I/can I (any Mastra feature), errors/troubleshooting (timeouts, failures), testing/validation, monitoring/costs, my agent/my workflow/my tool (possessive), pass data/stream/handle, upgrade/migration. Use BEFORE any Mastra code or advice. Prevents deprecated APIs, ensures current patterns, contains 27+ verified examples."
|
||||
description: "Comprehensive Mastra development guide with 27+ production-ready patterns. Covers: agents (creation, tools, memory, streaming, structured output), workflows (sequential, branching, parallel, loops, suspend/resume), tools (custom APIs, context access, validation), storage (Postgres, LibSQL, MongoDB), memory (conversation history, semantic recall, working memory), RAG (vector search, Graph RAG), and troubleshooting. Always use this skill for Mastra code to ensure current APIs and prevent deprecated patterns. Includes TypeScript configs, model formats, common errors, and best practices."
|
||||
license: Apache-2.0
|
||||
metadata:
|
||||
author: Mastra
|
||||
|
||||
-226
@@ -1,226 +0,0 @@
|
||||
# Test Mastra Skill Usage: 50 Test Cases
|
||||
|
||||
Test if subagents properly discover and invoke Mastra skills when given various queries.
|
||||
|
||||
**Goal:** Verify that skill descriptions trigger invocation (instead of agents answering from training knowledge)
|
||||
|
||||
**Skills Being Tested:**
|
||||
1. `mastra` - Main Mastra development skill
|
||||
2. `create-mastra` - Project setup skill
|
||||
3. `mastra-embedded-docs-look-up` - API verification skill
|
||||
|
||||
---
|
||||
|
||||
## Instructions for Claude
|
||||
|
||||
When user says: **"run the skill tests"**
|
||||
|
||||
Execute this process:
|
||||
|
||||
### Step 1: Load Test Cases
|
||||
|
||||
Read all 50 test cases from `test/TEST_CASES.md`
|
||||
|
||||
Example test cases:
|
||||
- "how do i make a simple chatbot with mastra?" → Should invoke `mastra` skill
|
||||
- "how do i get started with mastra?" → Should invoke `create-mastra` skill
|
||||
- "what are all the parameters for the Agent constructor?" → Should invoke `mastra-embedded-docs-look-up` skill
|
||||
|
||||
### Step 2: Spawn Subagents (in parallel batches)
|
||||
|
||||
For each test case:
|
||||
```
|
||||
1. Spawn subagent with Task tool (subagent_type="general-purpose")
|
||||
2. Pass ONLY the user query (e.g., "Create a basic AI agent that can answer questions")
|
||||
3. NO test context - subagent should not know it's being tested
|
||||
```
|
||||
|
||||
**Parallel batches:**
|
||||
- Run 5 subagents at once (to speed up testing)
|
||||
- Wait for batch to complete
|
||||
- Move to next batch
|
||||
|
||||
### Step 3: Check if Subagent Invoked Skill
|
||||
|
||||
For each subagent response, analyze:
|
||||
|
||||
**Question:** Did the subagent invoke the Skill tool?
|
||||
|
||||
```typescript
|
||||
// Look for Skill tool invocation in transcript
|
||||
const skillCalls = response.tool_uses.filter(call => call.name === 'Skill');
|
||||
|
||||
// What we're testing:
|
||||
{
|
||||
testId: "TC-11",
|
||||
query: "Create a basic AI agent",
|
||||
expectedSkill: "mastra",
|
||||
|
||||
// Did subagent invoke Skill tool?
|
||||
skillInvoked: skillCalls.length > 0, // true = good, false = bad
|
||||
|
||||
// If yes, which skill?
|
||||
invokedSkill: skillCalls[0]?.parameters?.skill || null,
|
||||
|
||||
// Did it invoke the RIGHT skill?
|
||||
correctSkill: invokedSkill === expectedSkill,
|
||||
|
||||
// Overall: Pass if correct skill invoked
|
||||
passed: skillInvoked && correctSkill
|
||||
}
|
||||
```
|
||||
|
||||
**Why This Matters:**
|
||||
- If subagent doesn't invoke skill → Skill description needs improvement
|
||||
- If subagent invokes wrong skill → Skill targeting needs refinement
|
||||
- If subagent invokes correct skill → Success! ✓
|
||||
|
||||
### Step 4: Calculate Metrics
|
||||
|
||||
```typescript
|
||||
const invocationRate = (casesWithSkillInvoked / totalCases) * 100;
|
||||
const correctnessRate = (casesWithCorrectSkill / totalCases) * 100;
|
||||
```
|
||||
|
||||
### Step 5: Save Results
|
||||
|
||||
Append results to `test/snapshots.json`:
|
||||
```json
|
||||
{
|
||||
"timestamp": "2026-01-29T...",
|
||||
"totalTests": 50,
|
||||
"passed": XX,
|
||||
"failed": XX,
|
||||
"invocationRate": XX.X,
|
||||
"correctnessRate": XX.X,
|
||||
"results": [...]
|
||||
}
|
||||
```
|
||||
|
||||
### Step 6: Report Results
|
||||
|
||||
Display summary:
|
||||
```
|
||||
============================================================
|
||||
SKILL USAGE TEST RESULTS
|
||||
============================================================
|
||||
Testing: Do subagents invoke Mastra skills?
|
||||
|
||||
Total Test Cases: 50
|
||||
Skill Invoked: XX (XX%) ← Did agent call Skill tool?
|
||||
Correct Skill: XX (XX%) ← Was it the right skill?
|
||||
Failed: XX (XX%) ← No skill or wrong skill
|
||||
|
||||
Target: ≥95% correct skill invocation
|
||||
Status: PASS/FAIL ✓
|
||||
|
||||
By Category:
|
||||
Setup & Installation: X/10 (XX%) → create-mastra
|
||||
Agent Development: X/15 (XX%) → mastra
|
||||
Workflows: X/10 (XX%) → mastra
|
||||
Tools & Integrations: X/5 (XX%) → mastra
|
||||
Storage & Memory: X/5 (XX%) → mastra
|
||||
RAG & Vector Search: X/3 (XX%) → mastra
|
||||
Troubleshooting: X/2 (XX%) → mastra/embedded-docs
|
||||
|
||||
Failures (what needs fixing):
|
||||
TC-18: "How do I handle agent failures?"
|
||||
Problem: Subagent answered directly (no skill invoked)
|
||||
Fix: Add "error handling" to mastra skill triggers
|
||||
|
||||
TC-22: "Track agent costs"
|
||||
Problem: Subagent answered directly
|
||||
Fix: Add "monitoring/costs" to mastra skill triggers
|
||||
|
||||
✓ Results saved to test/snapshots.json
|
||||
============================================================
|
||||
```
|
||||
|
||||
**Interpretation:**
|
||||
- High % = Skill descriptions are working (agents invoke skills)
|
||||
- Low % = Skill descriptions need improvement (agents skip skills)
|
||||
|
||||
---
|
||||
|
||||
## Example Usage
|
||||
|
||||
**User:**
|
||||
```
|
||||
run the skill tests
|
||||
```
|
||||
|
||||
**Claude:**
|
||||
```
|
||||
Running all 50 skill invocation tests...
|
||||
|
||||
Batch 1/10 - Spawning 5 subagents...
|
||||
TC-01: ✓ create-mastra
|
||||
TC-02: ✓ create-mastra
|
||||
TC-03: ✓ create-mastra
|
||||
TC-04: ✓ create-mastra
|
||||
TC-05: ✓ create-mastra
|
||||
|
||||
Batch 2/10 - Spawning 5 subagents...
|
||||
TC-06: ✓ create-mastra
|
||||
...
|
||||
|
||||
[Full results displayed]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Important Notes
|
||||
|
||||
### For Claude:
|
||||
|
||||
1. **Spawn REAL subagents** - Use Task tool, not self-analysis
|
||||
2. **No test context** - Subagent should not know it's being tested
|
||||
3. **Check actual tool calls** - Look for Skill tool invocations in transcript
|
||||
4. **Run ALL 50 cases** - Not just 9 quick samples
|
||||
5. **Save to snapshots.json** - Append, don't overwrite
|
||||
6. **Report honestly** - If tests fail, report failures and why
|
||||
|
||||
### For User:
|
||||
|
||||
- Just say: "run the skill tests"
|
||||
- Claude will handle everything
|
||||
- Results saved to `test/snapshots.json`
|
||||
- Check if ≥95% invocation rate achieved
|
||||
|
||||
---
|
||||
|
||||
## Test Cases Reference
|
||||
|
||||
All 50 test cases are defined in `TEST_CASES.md`:
|
||||
|
||||
**Categories:**
|
||||
- Setup & Installation: TC-01 to TC-10 (create-mastra)
|
||||
- Agent Development: TC-11 to TC-25 (mastra)
|
||||
- Workflows: TC-26 to TC-35 (mastra)
|
||||
- Tools & Integrations: TC-36 to TC-40 (mastra)
|
||||
- Storage & Memory: TC-41 to TC-45 (mastra)
|
||||
- RAG & Vector Search: TC-46 to TC-48 (mastra)
|
||||
- Troubleshooting: TC-49 to TC-50 (mastra + embedded-docs-look-up)
|
||||
|
||||
**Total:** 50 test cases
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
**Target:** ≥95% skill invocation rate
|
||||
|
||||
**Passing Conditions:**
|
||||
- ≥48 out of 50 tests invoke correct skill
|
||||
- All categories show >90% invocation
|
||||
- No systematic failures (e.g., entire category failing)
|
||||
|
||||
**If Failed:**
|
||||
- Identify which test cases failed
|
||||
- Analyze why skills weren't invoked
|
||||
- Refine skill descriptions
|
||||
- Re-run tests
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-29
|
||||
-257
@@ -1,257 +0,0 @@
|
||||
# Mastra Skills Test Suite
|
||||
|
||||
Real-world testing: Claude spawns subagents to measure actual skill invocation behavior.
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
**Claude Tests Claude:**
|
||||
|
||||
1. User says: **"run the skill tests"**
|
||||
2. Claude reads PROMPT.md for instructions
|
||||
3. Claude spawns 50 independent subagents (one per test case)
|
||||
4. Each subagent gets a query like: "Create a basic AI agent..."
|
||||
5. Claude observes which tools each subagent invokes
|
||||
6. Results saved to snapshots.json
|
||||
|
||||
**Why This Works:**
|
||||
- ✅ Real subagents (not self-analysis)
|
||||
- ✅ Fresh context (no test bias)
|
||||
- ✅ Actual tool calls (measured, not estimated)
|
||||
- ✅ All 50 cases (complete coverage)
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
test/
|
||||
├── PROMPT.md # Instructions for Claude to run tests
|
||||
├── snapshots.json # Historical test results
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Simple: Just Ask Claude
|
||||
|
||||
```
|
||||
User: "run the skill tests"
|
||||
```
|
||||
|
||||
Claude will:
|
||||
1. Load all 50 test cases from test/TEST_CASES.md
|
||||
2. Spawn subagents in parallel batches (5 at a time)
|
||||
3. Pass natural user queries like: "how do i make a chatbot with mastra?"
|
||||
4. Check each response for Skill tool invocations
|
||||
5. Calculate invocation rate and correctness
|
||||
6. Save results to snapshots.json
|
||||
7. Display summary report
|
||||
|
||||
### What Claude Does
|
||||
|
||||
```
|
||||
Running all 50 skill invocation tests...
|
||||
|
||||
Batch 1/10 - Spawning 5 subagents...
|
||||
TC-01: ✓ create-mastra invoked
|
||||
TC-02: ✓ create-mastra invoked
|
||||
TC-03: ✓ create-mastra invoked
|
||||
TC-04: ✓ create-mastra invoked
|
||||
TC-05: ✓ create-mastra invoked
|
||||
|
||||
Batch 2/10...
|
||||
TC-06: ✓ create-mastra invoked
|
||||
...
|
||||
|
||||
============================================================
|
||||
MASTRA SKILLS TEST RESULTS
|
||||
============================================================
|
||||
Total Tests: 50
|
||||
Passed: 48 (96%)
|
||||
Failed: 2 (4%)
|
||||
Invocation Rate: 96.0%
|
||||
Correctness Rate: 96.0%
|
||||
|
||||
Target: ≥95% → PASS ✓
|
||||
|
||||
By Category:
|
||||
Setup & Installation: 10/10 (100%)
|
||||
Agent Development: 14/15 (93%)
|
||||
Workflows: 10/10 (100%)
|
||||
Tools & Integrations: 5/5 (100%)
|
||||
Storage & Memory: 5/5 (100%)
|
||||
RAG & Vector Search: 3/3 (100%)
|
||||
Troubleshooting: 2/2 (100%)
|
||||
|
||||
Failures:
|
||||
TC-18: "How do I handle when an agent call fails?"
|
||||
Expected: mastra, Got: none (answered directly)
|
||||
|
||||
✓ Results saved to test/snapshots.json
|
||||
============================================================
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Cases
|
||||
|
||||
All 50 test cases are defined in `TEST_CASES.md`
|
||||
|
||||
**By Category:**
|
||||
|
||||
| Category | Tests | Expected Skill |
|
||||
|----------|-------|----------------|
|
||||
| Setup & Installation | TC-01 to TC-10 | create-mastra |
|
||||
| Agent Development | TC-11 to TC-25 | mastra |
|
||||
| Workflows | TC-26 to TC-35 | mastra |
|
||||
| Tools & Integrations | TC-36 to TC-40 | mastra |
|
||||
| Storage & Memory | TC-41 to TC-45 | mastra |
|
||||
| RAG & Vector Search | TC-46 to TC-48 | mastra |
|
||||
| Troubleshooting | TC-49 to TC-50 | mastra / embedded-docs-look-up |
|
||||
|
||||
**Total:** 50 test cases covering all Mastra development scenarios
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
**Target:** ≥95% skill invocation rate
|
||||
|
||||
**Passing:**
|
||||
- ≥48 out of 50 tests invoke correct skill
|
||||
- All categories >90% invocation
|
||||
- No systematic failures
|
||||
|
||||
**If Tests Fail:**
|
||||
1. Claude identifies which cases failed
|
||||
2. Analyzes why skills weren't invoked
|
||||
3. Suggests skill description improvements
|
||||
4. Can re-run tests after changes
|
||||
|
||||
---
|
||||
|
||||
## Snapshots
|
||||
|
||||
Results are saved in `snapshots.json`:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"timestamp": "2026-01-29T15:30:00Z",
|
||||
"totalTests": 50,
|
||||
"passed": 48,
|
||||
"failed": 2,
|
||||
"invocationRate": 96.0,
|
||||
"correctnessRate": 96.0,
|
||||
"results": [
|
||||
{
|
||||
"testId": "TC-01",
|
||||
"input": "How do I start a new Mastra project?",
|
||||
"expectedSkill": "create-mastra",
|
||||
"actualSkill": "create-mastra",
|
||||
"skillInvoked": true,
|
||||
"passed": true
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**View Latest Results:**
|
||||
```bash
|
||||
cat snapshots.json | jq '.[-1]'
|
||||
```
|
||||
|
||||
**Track Progress Over Time:**
|
||||
```bash
|
||||
cat snapshots.json | jq '[.[] | {timestamp, invocationRate}]'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why Not TypeScript?
|
||||
|
||||
**Previous approach:** test/prompt.ts (TypeScript code)
|
||||
- Required npm, tsx, dependencies
|
||||
- External script, harder to use
|
||||
- Not integrated with Claude's workflow
|
||||
|
||||
**Current approach:** test/PROMPT.md (simple prompt)
|
||||
- No dependencies
|
||||
- Just ask Claude: "run the skill tests"
|
||||
- Claude handles everything
|
||||
- Results automatically saved
|
||||
|
||||
---
|
||||
|
||||
## Advantages
|
||||
|
||||
1. **Simple:** No setup, just ask Claude
|
||||
2. **Integrated:** Works within Claude Code workflow
|
||||
3. **Complete:** Tests all 50 cases, not just samples
|
||||
4. **Parallel:** Runs in batches for speed
|
||||
5. **Tracked:** Historical snapshots for comparison
|
||||
6. **Honest:** Real subagent behavior, not estimates
|
||||
|
||||
---
|
||||
|
||||
## Example Session
|
||||
|
||||
```
|
||||
User: "run the skill tests"
|
||||
|
||||
Claude: "Running all 50 skill invocation tests...
|
||||
[spawns subagents, collects results]
|
||||
|
||||
Results: 48/50 passed (96%)
|
||||
Invocation Rate: 96.0% ✓
|
||||
|
||||
Target achieved! ≥95% skill invocation rate."
|
||||
|
||||
User: "show me the failures"
|
||||
|
||||
Claude: "2 test cases failed:
|
||||
|
||||
TC-18: Error handling question
|
||||
- Expected: mastra
|
||||
- Got: none (answered directly from training)
|
||||
- Suggests: Add 'error handling' to triggers
|
||||
|
||||
TC-22: Cost tracking question
|
||||
- Expected: mastra
|
||||
- Got: none (monitoring not in triggers)
|
||||
- Suggests: Add 'monitoring/costs' to triggers"
|
||||
|
||||
User: "update the skill descriptions based on failures"
|
||||
|
||||
Claude: "Updating mastra skill description...
|
||||
[makes changes]
|
||||
|
||||
Ready to re-test. Say 'run the skill tests' again."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
**When to Run Tests:**
|
||||
- After updating skill descriptions
|
||||
- After adding new test cases
|
||||
- Before releasing skill updates
|
||||
- Weekly/monthly for monitoring
|
||||
|
||||
**Iterative Improvement:**
|
||||
1. Run tests
|
||||
2. Identify failures
|
||||
3. Refine skill descriptions
|
||||
4. Re-run tests
|
||||
5. Repeat until ≥95%
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-29
|
||||
@@ -1,261 +0,0 @@
|
||||
# Mastra Skills Test Cases
|
||||
|
||||
50 real-world user queries to test if AI agents properly invoke Mastra skills.
|
||||
|
||||
---
|
||||
|
||||
## Setup & Installation (create-mastra skill)
|
||||
|
||||
### TC-01: New Project
|
||||
**Query:** "how do i get started with mastra?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-02: Add to Next.js
|
||||
**Query:** "i already have a nextjs app, how do i add mastra to it?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-03: Express Integration
|
||||
**Query:** "does mastra work with express?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-04: Import Error
|
||||
**Query:** "getting module not found error when i try to import from @mastra/core, any ideas?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-05: TypeScript Setup
|
||||
**Query:** "what typescript config do i need for mastra?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-06: API Keys
|
||||
**Query:** "where do i put my openai api key for mastra?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-07: Simple Script
|
||||
**Query:** "can i just write a quick script with mastra without setting up a whole project?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-08: Monorepo
|
||||
**Query:** "how do i use mastra in a monorepo?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-09: CLI vs Manual
|
||||
**Query:** "should i use create-mastra or install manually?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
### TC-10: Deployment
|
||||
**Query:** "what do i need to do before deploying my mastra app to production?"
|
||||
**Expected:** create-mastra
|
||||
|
||||
---
|
||||
|
||||
## Agent Development (mastra skill)
|
||||
|
||||
### TC-11: First Agent
|
||||
**Query:** "how do i make a simple chatbot with mastra?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-12: GPT-4
|
||||
**Query:** "i want to use gpt-4 for my agent, how?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-13: Agent with Tools
|
||||
**Query:** "can my agent use tools to search google and get weather data?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-14: Custom System Prompt
|
||||
**Query:** "how do i make my agent always respond in spanish?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-15: Multiple Agents
|
||||
**Query:** "can i have multiple agents talking to each other?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-16: Conversation Memory
|
||||
**Query:** "my agent forgets everything between messages, how do i fix that?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-17: Structured JSON Output
|
||||
**Query:** "i need my agent to return json in a specific format every time"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-18: Error Handling
|
||||
**Query:** "what happens if my agent call fails or times out?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-19: Streaming
|
||||
**Query:** "can i stream the response as the agent generates it?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-20: Passing Context
|
||||
**Query:** "how do i pass user session data to my agent?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-21: Testing
|
||||
**Query:** "what's the best way to test my mastra agents?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-22: Cost Tracking
|
||||
**Query:** "how can i see how much money each agent call costs me?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-23: Claude vs GPT
|
||||
**Query:** "should i use claude or gpt4 for my agent?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-24: Local Models
|
||||
**Query:** "can i use ollama or local llms with mastra?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-25: Timeout Issues
|
||||
**Query:** "my agent keeps timing out after 30 seconds, help!"
|
||||
**Expected:** mastra
|
||||
|
||||
---
|
||||
|
||||
## Workflows (mastra skill)
|
||||
|
||||
### TC-26: Basic Workflow
|
||||
**Query:** "how do i chain multiple steps together in mastra?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-27: Conditional Branching
|
||||
**Query:** "i need my workflow to take different paths based on the result"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-28: Parallel Tasks
|
||||
**Query:** "can i run multiple workflow steps at the same time?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-29: Loop Through Items
|
||||
**Query:** "how do i process a list of items in my workflow?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-30: Passing Data
|
||||
**Query:** "how do i pass data between workflow steps?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-31: Wait for Input
|
||||
**Query:** "can i pause my workflow and wait for user approval?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-32: Error Recovery
|
||||
**Query:** "what if one of my workflow steps fails?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-33: Long-Running Workflow
|
||||
**Query:** "my workflow takes hours, how do i track progress?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-34: Testing Workflows
|
||||
**Query:** "how can i test my workflow without running everything?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-35: Dynamic Steps
|
||||
**Query:** "can workflow steps be decided at runtime?"
|
||||
**Expected:** mastra
|
||||
|
||||
---
|
||||
|
||||
## Tools (mastra skill)
|
||||
|
||||
### TC-36: Custom Tool
|
||||
**Query:** "i want to give my agent a tool that calls my api"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-37: Tool with Context
|
||||
**Query:** "my tool needs access to the user's session data"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-38: Approval Tool
|
||||
**Query:** "how do i make a tool that asks for permission before running?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-39: Input Validation
|
||||
**Query:** "how do i validate the inputs my tool receives?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-40: Multiple Tools
|
||||
**Query:** "can my agent use like 5 different tools?"
|
||||
**Expected:** mastra
|
||||
|
||||
---
|
||||
|
||||
## Storage & Memory (mastra skill)
|
||||
|
||||
### TC-41: Postgres
|
||||
**Query:** "how do i connect mastra to postgres?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-42: Turso/LibSQL
|
||||
**Query:** "i want to use turso database with mastra"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-43: Chat History
|
||||
**Query:** "how do i store and load conversation history?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-44: Semantic Memory
|
||||
**Query:** "i want my agent to remember facts and recall them later"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-45: Temporary Data
|
||||
**Query:** "where can i store temporary data during agent execution?"
|
||||
**Expected:** mastra
|
||||
|
||||
---
|
||||
|
||||
## RAG (mastra skill)
|
||||
|
||||
### TC-46: Document Q&A
|
||||
**Query:** "how do i make an agent that answers questions from my docs?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-47: Vector Search Tool
|
||||
**Query:** "i need a tool that searches my vector database"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-48: Knowledge Graph
|
||||
**Query:** "can i do graph rag with mastra?"
|
||||
**Expected:** mastra
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting (mastra / embedded-docs-look-up)
|
||||
|
||||
### TC-49: Migration
|
||||
**Query:** "i'm upgrading from mastra v0.1, what breaking changes are there?"
|
||||
**Expected:** mastra
|
||||
|
||||
### TC-50: API Reference
|
||||
**Query:** "what are all the parameters for the Agent constructor?"
|
||||
**Expected:** mastra-embedded-docs-look-up
|
||||
|
||||
---
|
||||
|
||||
## Format
|
||||
|
||||
Each test case follows:
|
||||
```
|
||||
Query: Natural user question (lowercase, casual, how real users ask)
|
||||
Expected: Which skill should be invoked (mastra, create-mastra, or mastra-embedded-docs-look-up)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Categories
|
||||
|
||||
| Category | Tests | Expected Skill |
|
||||
|----------|-------|----------------|
|
||||
| Setup & Installation | TC-01 to TC-10 | create-mastra |
|
||||
| Agent Development | TC-11 to TC-25 | mastra |
|
||||
| Workflows | TC-26 to TC-35 | mastra |
|
||||
| Tools | TC-36 to TC-40 | mastra |
|
||||
| Storage & Memory | TC-41 to TC-45 | mastra |
|
||||
| RAG | TC-46 to TC-48 | mastra |
|
||||
| Troubleshooting | TC-49 to TC-50 | mastra / embedded-docs-look-up |
|
||||
|
||||
**Total:** 50 test cases
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-29
|
||||
@@ -1 +0,0 @@
|
||||
[]
|
||||
Reference in New Issue
Block a user