* Add eval suite and sample test project to tests/
Adds:
- tests/evals/ — Promptfoo LLM eval suite covering all 14 agent skills
(metrics, feature flags, AI configs) using claude-3-haiku-20240307
- tests/checkout_proj/ — sample React/Vite app used by skill scenarios
that require a real codebase (e.g. metric-instrument, flag-create)
- tests/fixtures/ — MCP tool schema fixtures for eval context
- tests/package.json — promptfoo dev dependency and npm scripts
(eval, eval:metrics, eval:flags, eval:aiconfigs)
- .github/workflows/integration-tests.yml — manual workflow for
running gram-functions integration tests against catfood
Made-with: Cursor
* fix: add checkout_proj as plain files, not submodule
Made-with: Cursor
* Scope PR down to metric-create eval only
Remove all other eval files, checkout_proj sample app, fixtures, and
CI workflow added earlier. Keep only the metric-create eval with
llm-rubric assertions and the print-results.mjs helper script.
Made-with: Cursor
* Delete tests/.gitignore
* test: add output mutation tests and tighten llm-rubric assertions
Replaces the previous bad-skills negative test approach with output
mutation testing: 7 hand-crafted defective canned outputs fed through
the same llm-rubric graders via a custom echo-output provider. No LLM
call, zero variance — only the grader's judgment is under test.
- Adds tests/providers/echo-output.mjs (returns vars.canned_output directly)
- Adds 7 [NEG] mutation tests to metric-create.eval.yaml, one per positive test
- Tightens 4 llm-rubric assertions that had blind spots (no-premature-create,
shows-proposal-step, checks-for-duplicates, asks-measure-type)
- Updates print-results.mjs: [NEG]+FAIL = green (defect detected),
[NEG]+PASS = red (blind spot); adds token usage + cost summary
- Adds --no-cache to npm run eval for guaranteed fresh results
Result: 7/7 positive tests pass, 7/7 mutation tests detect their defects.
Made-with: Cursor
* refactor: replace relative file:// paths with env var aliases
Removes all ../../../ traversal from eval configs. The npm eval script
now exports SKILL_ROOT and PROVIDER_ROOT as absolute paths, which the
YAML references via {{env.SKILL_ROOT}} and {{env.PROVIDER_ROOT}}.
Adding evals for new skills only requires changing the path segment,
not counting directory levels.
Made-with: Cursor
* ci: add LLM eval workflow + path aliases + CI exit code gate
- Adds .github/workflows/llm-evals.yml — runs on push/PR when
skills/metrics/**, tests/evals/**, tests/providers/** change, plus
manual workflow_dispatch. Requires ANTHROPIC_API_KEY secret.
- Replaces file://../../../ traversal with SKILL_ROOT / PROVIDER_ROOT
env vars set in the npm eval script (absolute paths via subshell).
- print-results.mjs now exits 1 when positive tests fail, exits 0
otherwise — mutation tests are informational and don't gate CI.
Made-with: Cursor
* Add eval suites for metric-choose and metric-instrument skills
- tests/evals/metrics/metric-choose.eval.yaml: 6 positive + 6 mutation tests
covering hypothesis-first, release-policy lookup, event health, advisory-only,
inventory step, and context disambiguation behaviors
- tests/evals/metrics/metric-instrument.eval.yaml: 6 positive + 6 mutation tests
covering server-side track() signature, metricValue usage, event verification,
placement confirmation, SDK key fetching, and existing-pattern detection
- tests/package.json: split eval into named per-skill scripts (eval:create,
eval:choose, eval:instrument) with combined eval running all three
- .github/workflows/llm-evals.yml: run each skill eval as a separate CI step
Results: metric-create 7/7+7/7, metric-choose 6/6+6/6, metric-instrument 6/6+6/6.
Made-with: Cursor
* Set eval provider temperature to 0 for deterministic results
Made-with: Cursor
* chore: drop CI Promptfoo; add test:llm-evals for manual runs
Remove llm-evals GitHub Action (push/PR) so evals are not run automatically
in CI. Add npm test:llm-evals at repo root and in tests/ as an explicit
name for the full eval suite, and document in CONTRIBUTING.
Made-with: Cursor
Dakota Sanchez