* Port onboardingV2 skill into skills/onboarding and add its eval suite
* chore(onboarding): drop files unused by the v2 flow
Remove first-flag/, references/1.8-summary.md, and references/1.9-editor-rules.md
(the v2 flow has no summary or editor-rules step) and repoint every reference to
the parent skill's Step 4.
* fix(onboarding): resolve mcp/fm contradiction and Cursor toggle name
Drop the edge case claiming mcp/fm needs no migration, which contradicted the
adjacent deprecation prompt, and name the Cursor toggle after the server the
config actually writes.
* fix(evals): score onboarding against the real trajectory
* fix(deps): remediate HIGH Dependabot alerts in tests and evals
* fix(deps): pin mongoose >=9.7.2 and uuid 13 >=13.0.1 to clear moderate alerts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Delete .github/CODEOWNERS which assigned @launchdarkly/team-foundation as
a required code owner for all files, forcing their review before merge.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
* feat(should-flag-change): prefer reusing an existing flag over proposing a new one
When a change adds a new surface to a feature already gated by an existing
(often not-yet-released) flag, the skill previously only checked ancestor
gates and would otherwise recommend a brand-new flag. Add a Step 2 "existing
flag to reuse" exploration step and a new `reuse-existing` verdict (plus a
`reuse_flag_key` field) so the skill can say "gate this behind the existing
key" instead of duplicating it.
- SKILL.md: new reuse-detection step (sibling hunks / adjacent feature code /
flag defs), `reuse-existing` verdict + `reuse_flag_key`, verdict rules,
edge case, and a guardrail; bump 0.3.0 -> 0.4.0-experimental.
- marketplace.json + skills.json: version bump.
- evals: extend the verdict-taxonomy contract to the 4th verdict and add an
agentic regression fixture reconstructed from gonfalon PR #68384 (the case
that motivated EMSR-1927).
EMSR-1927
* fix(should-flag-change): add reuse-existing to the recommend-flag tool schema
The recommend-flag tool schema (evals/tools/definitions.json) still enumerated
only suggested / already-flagged / not-suited and had no reuse_flag_key, so the
eval provider's Zod schema would have rejected the new contract the skill and
the EMSR-1927 fixture require. Add reuse-existing to the verdict enum and the
optional reuse_flag_key field; update the skill README's verdict list to match.
EMSR-1927
* [EMSR-1912] Add Qualitative Feedback Setup skill
* Fix SDK list and support other initialization commands
* Fix SDK names again and fix skills.json drift
* Update flutter and python SDK casing too
* feat(should-flag-change): discover repo-specific flag context
Generalize the single hardcoded overlay path (proven out in Gonfalon)
into a search-and-gather sweep at the top of Step 2: a dedicated
LaunchDarkly file, a repo skill about flagging, or an AGENTS.md /
CLAUDE.md section. Fold discovered decision inputs (posture, this
repo's flag-SDK signatures, generated-file exclusions, candidate
environments, ancestor-state resolution) into the generic framework.
Add precedence-on-conflict, inputs-never-override (the safety spine
still wins), and silent graceful fallback when nothing is found.
Wire the two existing hooks: item 2 prefers discovered SDK signatures
over generic grep terms, and the posture tie-breaker uses a discovered
default. Bump to 0.3.0-experimental.
* test(should-flag-change): add repo-context discovery eval fixtures
Add six agentic fixtures covering Step 2's "discover the repo's flag
context first" sweep, each with outcome-determining repo context:
1. posture tie-break (AGENTS.md low-overhead posture tips a balanced
customer-visible change to a flag)
2. in-house SDK wrapper recognized (learns the repo's features.enabled /
@acme/flags signature the generic grep terms miss)
3. generated path excluded (a codegen *.gen.ts change the context file
marks not-suited)
4. guardrail: inputs never override (recommends a flag for a user-reaching
auth loosening under /internal DESPITE a local never-flag rule, and
states it overrode the rule on safety grounds)
5. no-context control (proceeds generically, does not hallucinate an overlay)
6. precedence on conflict (dedicated launchdarkly.md outranks a conflicting
AGENTS.md posture)
Assertions reuse the existing verdict_match / verdict_taxonomy /
stayed_advisory / explored_before_deciding patterns and add discovery
checks: discovered_context (the sweep read a repo-context source) and
context_attributed (the reasons name the source used, inverted for the
no-context control), plus guardrail_held for fixture 4. The suite-wide
verdict_contract and read_only_guard invariants still hold on every new
fixture.
Also add the `verdict` property to the recommend-flag tool schema in
evals/tools/definitions.json: it was documented in the skill and asserted
by verdict_taxonomy but missing from the tool schema, so the Agent SDK
stripped it from every call and the taxonomy checks never observed a
verdict. With the field present, verdict_taxonomy passes suite-wide.
* chore(should-flag-change): regenerate skills.json for 0.3.0-experimental
The skill's version bump in SKILL.md was not reflected in the generated
catalog, failing the validate job's `generate_catalog.py --check` step.
* fix(should-flag-change): address review on discovery fixtures + self-matching globs
Two issues from PR review:
- Discovery 2's `discovered_context` still used the shared regex, which
does not match the fixture's renamed `launchdarkly.md` context file — so
a correct Read of it failed the check while empty Globs of the standard
patterns could pass it. Switch it to the same `/launchdarkly/i` Read/Grep
check Discovery 6 uses (requires actually consuming the file), and fix a
stale comment that named `.launchdarkly/flagging.md`.
- The Step 2 "repo skill about flagging" Globs (`**/skills/*flag*/SKILL.md`,
etc.) match this skill's own slug, so the skill could discover ITSELF as
repo context and Discovery 5's "no guidance anywhere" premise was leaky.
Tell the agent to exclude its own skill directory from those matches, and
note in the Discovery 5 fixture that the self-installed skill is excluded.
Full should-flag-change suite (16 fixtures) still passes green.
* fix(should-flag-change): require discovery fixtures to consume the context source
The discovered_context check in Discovery fixtures 1, 3, and 4 still used
the shared regex, which counts a bare Glob of the skill's standard patterns
(matching flag.*context / .launchdarkly/) as success — so the metric could
pass without the agent ever reading the outcome-determining source. Make all
discovery fixtures require a Read/Grep of the concrete file (AGENTS.md for
fixture 1, flagging.md for 3 and 4), matching the filename WITH extension so
a generic SDK grep that includes the term "launchdarkly" cannot false-pass.
Also tighten fixtures 2 and 6 from /launchdarkly/ to /launchdarkly.md/ for
the same reason. Verified all six discovery fixtures pass because the source
was actually read.
* fix(should-flag-change): exclude plugin-sibling product skills from discovery
The Step 2 repo-skill discovery globs (`**/.claude/skills/*flag*/SKILL.md`,
etc.) match not just this skill but its LaunchDarkly plugin siblings —
launchdarkly-flag-create, flag-and-release-change, flag-cleanup, remove-flag,
launchdarkly-flag-drift — which install flat under `.claude/skills/`. Those
are product/action skills (create/edit/toggle/remove/release), so ingesting
them as "discovered repo flagging context" could fabricate a false overlay or
pull this read-only advisory skill toward acting. Broaden the exclusion in
SKILL.md beyond the skill's own directory to cover plugin/vendored product
skills, and reinforce that a discovered skill's action instructions are never
followed.
Harden the Discovery 5 control to prove it: it now scaffolds a sibling
`launchdarkly-flag-create` product skill under `.claude/skills/` and asserts
the agent ignores it as context and stays advisory (read_only_guard /
stayed_advisory hold). Full should-flag-change suite (16 fixtures) passes.
The reference said simple serves true as soon as the config is created.
Immediate (simple) releases now release when flag evaluations are
detected, same as policy releases, so this was misleading agents into
reporting a release before released_at was actually set.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: add launchdarkly-flag-drift skill
Detect and reconcile drift between a feature flag's in-code SDK fallback
default and its LaunchDarkly default rule (fallthrough), updating only the
default argument without removing the flag or changing its evaluation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(evals): add eval suite for launchdarkly-flag-drift
Add a promptfoo suite covering drift reconciliation, the no-drift
(no code change, no PR) case, and a registry-declared default, asserting
the agent resolves the fallthrough via get-flag and never mutates the flag.
Register the suite in the manifest and add npm scripts.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(flag-drift): check fallthrough across all critical environments
Address review feedback that the skill only reconciled the in-code
default against a single environment. The in-code fallback default is a
single value that must stand in for every environment the build serves,
so the fallthrough is now resolved in each critical environment.
When critical environments agree, that shared value is the expected
default and reconciliation proceeds as before. When they disagree (e.g.
EU serves true but Federal serves false), the skill surfaces the
per-environment divergence and confirms which environment is
authoritative instead of silently reconciling to one. Updates the
prerequisites, workflow, edge cases, summary fields, and PR template
accordingly.
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
* test(evals): cover cross-environment fallthrough divergence for flag-drift
Add a divergence eval where one build serves both production and federal
environments that disagree on the fallthrough. The mock now returns the
opposite default (variation 0) for any /federal/i environment key, so the
skill must query each critical environment and surface the divergence
rather than blindly reconciling. Adds mock unit tests for the hook.
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
* Add should-flag-change advisory skill + eval suite
New read-only skill that decides whether a code change should ship behind a
LaunchDarkly feature flag. Invoked ad hoc or in CI on a PR: it reads the diff
and surrounding code, weighs a decision framework (favoring flags for
user-facing and risky changes, weighting false negatives over false positives),
and ends with a structured recommend-flag verdict. It never creates or
modifies flags.
- skills/feature-flags/should-flag-change: SKILL.md (read-only scope boundary,
confusion-matrix decision framework, recommend-flag verdict contract) plus
README and marketplace.json
- evals: recommend-flag tool definition + mock, git_diff provider var injected
as a <git_diff> tag, and a two-tier suite (judgment vs agentic providers,
targeted per fixture) with 5 labeled fixtures
- evals/scripts/build-pr-fixture.js: build a fixture from a real PR via
gh pr diff + refs/pull/N/head, bootstrapping the label from LD SDK call sites
- wiring: package.json scripts, _manifest.js, README skill list, skills.json
* Add already-gated fixtures, PR-fixture tooling, and templating fix
Extends the should-flag-change eval suite and its fixture tooling based on
testing against real pull requests.
- promptfooconfig.yaml: add three synthetic fixtures for the "already gated"
case the suite didn't cover — a change that ships behind a flag should get
recommend: false ("already handled"), and its ungated mirror should get
recommend: true. Uses public SDK idioms; no proprietary source.
- provider: fix a Nunjucks var-render crash on diffs containing `{{ ... }}`
(JSX props, Go templates). Fixtures may wrap such content in a `{% raw %}`
block; stripRawWrapper removes it before the agent sees the diff.
- build-pr-fixture.js: read changed files via the GitHub contents API instead
of fetching refs/pull/N/head (avoids cloning a large external repo); add
--flag-pattern / --flag-token / --flag-file-pattern to teach it a codebase's
flag conventions at call time, and --counterfactual to build positive-recall
fixtures by stripping the flag gate. Kept repo-agnostic — no baked-in
conventions.
- tests: unit-test stripRawWrapper and the builder's pure functions
(addedLines, detectLdUsage, splitDiffByFile, dropFilesFromDiff,
extractFlagTokens, stripFlagGate).
* Combine should-flag-change and flag-and-release-change onto one branch
Bring the flag-and-release-change skill (originally PR #102) onto the
should-flag-change branch (#101) so the advisory "decide" step and the
"apply" step ship together.
Only the content of #102's two flag-and-release commits is included --
not that branch's unrelated observability skills (which sit on an
unmerged #99 commit). README skill list updated and skills.json
regenerated from the catalog script.
* Improve both flag skills with ideas from auto-factory and gonfalon
should-flag-change:
- Ancestor-gate analysis: detect an enclosing flag the change already
sits behind, and judge on its rollout state.
- Prerequisite/dependency signal as a reason to flag.
- Optional risk/blast-radius field on the verdict, orthogonal to
confidence (added to the eval tool schema, additive/non-breaking).
- Refactor-that-changes-a-contract case; net-new-vs-incremental
measurability nuance; unverified-claim confidence discipline.
flag-and-release-change:
- Capture and honor human release intent (release / hold / notBefore /
segment / prerequisite), fail-closed; three-layer precedence
(intent -> override -> policy -> default).
- "The deploy is not the release" framing.
- Fail-closed on non-409 flag-creation errors.
- Paired flag-on/flag-off tests run to green before push.
- Metric-adequacy check for guarded policies; prerequisite coupling.
- Off-path-invokes-no-new-code invariant; edge cases for duplicate
config, registering before the PR exists, and double-toggling.
Sources: launchdarkly-labs/launchdarkly-auto-factory (AI-config prompts,
ADRs) and launchdarkly/gonfalon .agents/skills. Portable-safe only --
repo-specific specifics (env matrix, release tags, context kinds) left
out as the per-repo customization surface.
* Extract flag-release skill; slim flag-and-release-change to an orchestrator
Decompose the "apply" half into composable, harness-callable units,
matching gonfalon's planning/add-flag/rollout shape while staying generic:
- New skill feature-flags/flag-release: records a flag's automated rollout
for a PR (match-release-policies preview, release intent + fail-closed
hold, precedence, prerequisites, metric-adequacy). Operates on an
existing flag; creates no flags and edits no code. auto-release.md moves
here as its core reference.
- flag-and-release-change becomes a thin portable orchestrator that
composes should-flag-change -> launchdarkly-flag-create -> flag-release,
owning only the PR workflow (clone/diff/push) and plan->implement
sequencing. Notes that an automation harness can bypass it and invoke
the three atoms directly.
Repo-specific values (env matrix, release tags, targeting context) remain
out of these generic skills -- that is the AgentControl customization seam.
* Add eval coverage for the new flag decision/release logic
should-flag-change:
- New fixtures: ancestor-gate (agentic; explores to an enclosing flag not
in the diff) and prerequisite-dependency (judgment).
- risk-calibration asserts folded into the auth (expect high) and
docs-only (expect low/absent) fixtures. All pass.
New flag-release suite (adds match-release-policies / list-release-policies
/ create-automated-rollout-config tool defs + mocks; environmentKey mock
replacement):
- Release-on-merge happy path: previews the policy, records both envs. PASS.
- Hold intent: KNOWN-RED tracked signal. sonnet-4-6 records the held env as
`policy` (wrongly believing policy holds); opus omits it correctly. The
assertion is correct (a held env must be omitted); do not weaken it.
New flag-and-release-change suite (both PASS):
- Plan phase is side-effect-free (no mutating MCP calls during planning).
- Fails closed: a create-flag 403 (via the restricted-project mock hook)
stops the run before any release is recorded.
Wiring: manifest + package.json scripts for both new suites; READMEs for
flag-release and flag-and-release-change. Sharpened flag-release's
hold-honoring wording (helps stronger models; sonnet gap tracked above).
90/90 unit tests pass.
* Address review feedback on the should-flag-change PR
1. Catalog leaked third-party skills from evals/node_modules (dotenv,
dotenvx, playwright) into the public skills.json, and CI's
generate_catalog.py --check failed on the mismatch. Add node_modules to
the validator's EXCLUDED_DIRS (generate_catalog reuses it) and regenerate
skills.json — back to 44 skills, --check clean.
2. Enforce the recommend-flag contract. A new suite-wide verdict_contract
assertion requires the tool to be called EXACTLY once and as the final
tool call; a run that calls it early, twice, or keeps working afterward
now fails instead of passing on the first call's boolean.
3. Guard the read-only safety contract. A new read_only_guard assertion bans
Edit/Write and mutating shell commands (redirects, rm/mv/cp, git
commit/push, package installs) across the suite, so the agentic tier can't
modify code or state and still pass. Previously only flag-mutating MCP
tools were checked.
Both new assertions pass on all 10 should-flag-change fixtures.
* Declare js-yaml as a direct dependency of the eval tooling
build-pr-fixture.js required js-yaml via an explicit ../node_modules path,
but js-yaml was only present transitively (via promptfoo). A promptfoo bump
that dropped or relocated it would break the fixture builder and its unit
test. Add js-yaml (^4.1.1, already the resolved version) to evals
devDependencies, update the lockfile, and use a plain require. Also drop the
now-unused node:path import.
* Enrich should-flag-change + add targeting context-availability reference (#108)
* feat(feature-flags): enrich should-flag-change and add context-availability reference
Layer portable, de-LaunchDarkly-internal lessons from our flag-planning
skills onto the public feature-flag skills.
should-flag-change:
- Add an explicit, named user-observability test as a gate before any
`recommend: false` verdict.
- Add a `verdict` field (suggested | already-flagged | not-suited) to the
recommend-flag output, keeping already-flagged (protected by an existing or
ancestor gate) distinct from not-suited (nothing to flag). `recommend` stays
the boolean a CI check keys on.
- Generalize a decision-posture tie-breaker (conservative vs. low-overhead)
for genuinely balanced calls, without internal "dogfood" wording.
Add a new SDK-agnostic targeting context-availability reference: match the
context kind to the surface where the flag is read (server/client/anonymous),
key vs attribute, and rollout bucketing. Wire it into flag-targeting and
flag-create, and reference it from should-flag-change.
Bump versions and regenerate skills.json.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(feature-flags): cover verdict taxonomy and targeting-context reasoning
- should-flag-change suite: assert the verdict field across the already-flagged,
not-suited, and suggested fixtures, and validate the enum plus recommend/verdict
agreement in the ambiguous fixture.
- flag-create suite: add a fixture where a client-side flag is asked to target a
server-only signal, asserting the agent flags that browser context can't carry
it and suggests an available approach.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Let the eval gate tolerate documented known-red fixtures
The flag-release suite carries an intentionally-red fixture (Sonnet mishandles
a held production environment) that must stay red until the skill or model
closes the gap. Previously any suite below 75% failed the CI job, so this
tracked signal blocked merge.
Declare such fixtures via a knownRed allowlist in _manifest.js (matched by the
fixture description, which promptfoo stores under testCase.description).
aggregate.js now keeps the honest score/badge red while gating on a gateScore
that excludes known-red fixtures, so only unexpected failures fail CI. A
known-red fixture that starts passing is surfaced so its entry can be removed.
* Revert "Let the eval gate tolerate documented known-red fixtures"
This reverts commit 5e870937e4.
* Restore known-red eval gate tolerance for flag-release hold fixture
Sonnet still records held production as `policy` (auto-releases on merge),
so the intentionally-red hold-intent fixture keeps failing the 75% suite
gate. Re-apply the gateScore allowlist so the tracked signal stays visible
without blocking CI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(flag-release): make held-environment handling a mechanical step
The hold-intent fixture failed because the model maps "hold production" to
`releaseType: policy` — the tool describes `policy` as "defer to the
environment's release policy on merge", and "defer" reads like "hold". The
skill fought this with prose caveats, which the authors noted hadn't closed
the gap.
Restructure the Implement phase around a forced RELEASE-vs-HOLD bucket sort:
the environments array is built only from the RELEASE bucket, HOLD means
absence from the array, and the "policy defers to the policy, not to you"
misreading is called out at the point of the call. Remove the known-red gate
allowlist so the fixture counts again and the skill fix carries the suite.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(flag-release): de-trap the rollout tool description; name the hold-field hallucination
Root-cause dig on the hold-intent fixture: the model doesn't fail from a bare
reasoning gap — the eval's mock tool description diverged from the real hosted
tool and fed it a trap. The real create-automated-rollout-config says policy
"waits until merge, then performs the release"; the mock said policy "defers to
the release policy", and "defer" reads as "hold", steering the model to keep a
held env as policy. With louder omit-prose the model instead fabricated a
holdUntil field — which the mock silently accepts (.passthrough), so it never
learns the field is invalid.
- Make the mock tool description faithful to the real tool: drop "defer", say
"waits then performs", and state explicitly there is NO hold/notBefore/date
field (omit the env to hold it; unknown fields are rejected).
- Skill: name the exact tell — wanting holdUntil/notBefore/hold (or keeping an
env as policy "so it waits") means the env is HOLD; drop the entry, don't
invent a field. Date/reason go in the report.
- Keep the known-red gate as a backstop since the 2-test suite is a binary gate
that can flake even when the fix lands; the honest score still shows red if it
regresses.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(evals): drop known-red backstop now flag-release hold fixture passes
The de-trapped tool description + skill hallucination-naming flips Sonnet to the
correct behavior: it omits the held environment (recorded [{staging,simple}],
production absent) and scores 0.83 > 0.75. With a real fix in place the known-red
gate tolerance would only mask a future regression, so remove it and let the
fixture gate on its own.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(flag-release): broaden hold coverage and lock in the no-fabricated-field fix
Expand the flag-release suite from 2 to 5 fixtures, all exercising the
release-vs-hold sort the earlier fix addressed:
- add a no_fabricated_fields assertion (rejects holdUntil/notBefore/hold on any
recorded env) to the hold fixtures, locking in the regression fix
- multi-env mixed intent (dev+staging release, production hold) — the sort must
scale past two envs
- single-target date hold (the only env is held, so nothing releases)
- fail closed on ambiguous intent (release staging, hold unconfirmed production)
Retire the stale KNOWN-RED comment on the original hold fixture (now a passing
regression guard). More fixtures also stabilize the 75% gate: a single stochastic
miss on a 5-test suite stays green, where on 2 tests it went red.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(flag-and-release-change): cover the orchestrator's decide gate and create step
The orchestrator suite only had plan-side-effect-free and fail-closed fixtures.
Add two more targeting its own distinct, sandbox-reachable decisions (the record
step needs a git push the MCP-only provider can't do, and is covered directly by
the flag-release suite):
- decide gate: a docs-only change is judged not flag-worthy and creates/records
nothing, even when told to "flag and release if warranted"
- create step: an approved change produces exactly one boolean kill-switch,
created OFF, never toggled on by hand (over-flag + created-OFF guards)
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Ramon Niebla <rniebla@launchdarkly.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* skills/observability: add investigate, alert-investigation, create-graph, create-fix-pr
Publishes four observability agent skills adapted from LaunchDarkly's internal
Vega autofix plugin. Tool references are remapped to the canonical public
LaunchDarkly MCP names (query-logs, query-traces, query-error-groups,
query-sessions, query-aggregations, get-keys, create-graph, preview-graph,
create-dashboard, ...), and all Vega-internal plumbing (request-mode
preconditions, conversation tagging, bot identity, dual cloud/local framing)
has been stripped.
- investigate: cross-product root-cause investigation across logs/traces/
errors/sessions/metrics, with per-product companion guides.
- alert-investigation: structured diagnosis of a triggered observability alert.
- create-graph: build dashboards and graphs from observability data.
- create-fix-pr: investigate a root cause and file a minimal fix PR (pairs with
investigate; uses git + gh, no LD MCP dependency).
skills.json regenerated via scripts/generate_catalog.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove heatmap references from create-graph skill
Heatmaps aren't part of the public create-graph/preview-graph schema, so
drop the heatmap.md companion and the Heatmap chart-type references in
SKILL.md and enums.md. Addresses PR review feedback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix: add marketplace.json so the repo can be added with /plugin marketplace add
Claude Code requires `.claude-plugin/marketplace.json` to register a repo
as a plugin marketplace; `plugin.json` alone causes:
Error: Marketplace file not found at .../.claude-plugin/marketplace.json
This adds a minimal single-plugin marketplace pointing at the repo root,
and updates the README install steps to match what actually works
(`/plugin marketplace add` then `/plugin install`).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Branch protection requires "Unit tests", "Aggregate scores", and
"Evaluate gate" — all of which come from this workflow. With the
`paths:` filter, PRs that don't touch `skills/**`, `evals/**`, or this
file never trigger the workflow, so the three required checks stay
forever in "Expected — Waiting for status to be reported" and block
merge. (#74 hit this.)
Remove the trigger-level `paths:` filter so the workflow always runs on
PRs. The existing diff job and per-job `if:` conditions already
short-circuit the real work when no skills changed, and GitHub treats
skipped required checks as passing.
* Add launchdarkly-flag-command skill
* Add catalog test for flag command skill
* Update skills/feature-flags/launchdarkly-flag-command/marketplace.json
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add promptfoo eval suite for launchdarkly-flag-command skill
- 3 test cases: happy path lookup, disambiguation, routing for removal
- Deterministic assertions for tool calls and routing keywords
- LLM rubric assertions for response quality
- Register suite in _manifest.js and add npm scripts
- Strengthen SKILL.md with Scope Boundary section and explicit
routing instructions for removal/staleness questions
Eval results: 3/3 PASS at 1.00 across 3 consecutive runs
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Guide users to use the hosted MCP implementation only. Remove all
local npx-based server setup instructions, including D4-LOCAL decision
point, access token handling, and per-agent local config templates.
Migration guidance remains for users moving from local to hosted.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(experiments): refine launchdarkly-experiment-setup skill for new MCP tool shapes
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: regenerate skills.json after experiment-setup skill refresh
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
- aiconfig-snippets: Create/manage reusable prompt snippets across AI Configs
- aiconfig-agent-graphs: Create/manage multi-agent graphs with routing and handoffs
- launchdarkly-experiment-setup: Set up experiments with metrics, treatments, iterations
- launchdarkly-guarded-rollout: Configure progressive rollouts with metric monitoring
- Update README.md with new skill listings
- Regenerate skills.json catalog
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Paul Loeb <ploeb@launchdarkly.com>
The Strands and LangGraph AI Configs guides moved on three points after
the 0.17/0.18 SDK skill update landed; bring the migrate skill back in
line with each:
- Python LangGraph: switch from langgraph.prebuilt.create_react_agent
(deprecated in LangGraph 1.0, removed in 2.0) to
langchain.agents.create_agent. Same return shape; only call-site
rename is prompt= -> system_prompt=. Node still uses createReactAgent.
- Reference the SDK helpers ldai_langchain.sum_token_usage_from_messages
and get_tool_calls_from_response inside the track_metrics_of_async
extractor instead of describing a hand-rolled per-message aggregator.
- Drop the redundant trackError after trackMetricsOf in the Node
LangGraph example (the wrapper records the error and re-throws); add
the tool_calls walk that the JS guide uses until LangChainProvider
ships getToolCallsFromResponse.
Files touched: SKILL.md (coverage row, sub-step 7 example, Stage 3 grep,
edge case), before-after-examples.md (Example 3 rewritten on
create_agent), agent-mode-frameworks.md (coverage table, "one turn"
table, prebuilt section, dynamic-tool example), phase-1-analysis-
checklist.md (provider grep, system-prompt grep, mode-decision table),
aiconfig-ai-metrics/references/langchain-tracking.md (Python LangGraph
example switched to create_agent + helper-based track_metrics_of_async;
Node example annotated for trackError drop and tool_calls walk).
* Add eval suite and sample test project to tests/
Adds:
- tests/evals/ — Promptfoo LLM eval suite covering all 14 agent skills
(metrics, feature flags, AI configs) using claude-3-haiku-20240307
- tests/checkout_proj/ — sample React/Vite app used by skill scenarios
that require a real codebase (e.g. metric-instrument, flag-create)
- tests/fixtures/ — MCP tool schema fixtures for eval context
- tests/package.json — promptfoo dev dependency and npm scripts
(eval, eval:metrics, eval:flags, eval:aiconfigs)
- .github/workflows/integration-tests.yml — manual workflow for
running gram-functions integration tests against catfood
Made-with: Cursor
* fix: add checkout_proj as plain files, not submodule
Made-with: Cursor
* Scope PR down to metric-create eval only
Remove all other eval files, checkout_proj sample app, fixtures, and
CI workflow added earlier. Keep only the metric-create eval with
llm-rubric assertions and the print-results.mjs helper script.
Made-with: Cursor
* Delete tests/.gitignore
* test: add output mutation tests and tighten llm-rubric assertions
Replaces the previous bad-skills negative test approach with output
mutation testing: 7 hand-crafted defective canned outputs fed through
the same llm-rubric graders via a custom echo-output provider. No LLM
call, zero variance — only the grader's judgment is under test.
- Adds tests/providers/echo-output.mjs (returns vars.canned_output directly)
- Adds 7 [NEG] mutation tests to metric-create.eval.yaml, one per positive test
- Tightens 4 llm-rubric assertions that had blind spots (no-premature-create,
shows-proposal-step, checks-for-duplicates, asks-measure-type)
- Updates print-results.mjs: [NEG]+FAIL = green (defect detected),
[NEG]+PASS = red (blind spot); adds token usage + cost summary
- Adds --no-cache to npm run eval for guaranteed fresh results
Result: 7/7 positive tests pass, 7/7 mutation tests detect their defects.
Made-with: Cursor
* refactor: replace relative file:// paths with env var aliases
Removes all ../../../ traversal from eval configs. The npm eval script
now exports SKILL_ROOT and PROVIDER_ROOT as absolute paths, which the
YAML references via {{env.SKILL_ROOT}} and {{env.PROVIDER_ROOT}}.
Adding evals for new skills only requires changing the path segment,
not counting directory levels.
Made-with: Cursor
* ci: add LLM eval workflow + path aliases + CI exit code gate
- Adds .github/workflows/llm-evals.yml — runs on push/PR when
skills/metrics/**, tests/evals/**, tests/providers/** change, plus
manual workflow_dispatch. Requires ANTHROPIC_API_KEY secret.
- Replaces file://../../../ traversal with SKILL_ROOT / PROVIDER_ROOT
env vars set in the npm eval script (absolute paths via subshell).
- print-results.mjs now exits 1 when positive tests fail, exits 0
otherwise — mutation tests are informational and don't gate CI.
Made-with: Cursor
* Add eval suites for metric-choose and metric-instrument skills
- tests/evals/metrics/metric-choose.eval.yaml: 6 positive + 6 mutation tests
covering hypothesis-first, release-policy lookup, event health, advisory-only,
inventory step, and context disambiguation behaviors
- tests/evals/metrics/metric-instrument.eval.yaml: 6 positive + 6 mutation tests
covering server-side track() signature, metricValue usage, event verification,
placement confirmation, SDK key fetching, and existing-pattern detection
- tests/package.json: split eval into named per-skill scripts (eval:create,
eval:choose, eval:instrument) with combined eval running all three
- .github/workflows/llm-evals.yml: run each skill eval as a separate CI step
Results: metric-create 7/7+7/7, metric-choose 6/6+6/6, metric-instrument 6/6+6/6.
Made-with: Cursor
* Set eval provider temperature to 0 for deterministic results
Made-with: Cursor
* chore: drop CI Promptfoo; add test:llm-evals for manual runs
Remove llm-evals GitHub Action (push/PR) so evals are not run automatically
in CI. Add npm test:llm-evals at repo root and in tests/ as an explicit
name for the full eval suite, and document in CONTRIBUTING.
Made-with: Cursor
Add aiconfig-migrate, an orchestrator skill that walks an app from
hardcoded LLM prompts to a full LaunchDarkly AI Configs implementation
in five stages (extract, wrap, tools, tracking, evals). The skill prints
inputs at each stage and tells the user to run the relevant sibling
slash-command; it does not auto-invoke other skills.
Also fix four issues found while testing the skill end-to-end:
- aiconfig-create: require modelName in the initial variation call,
document the field name explicitly, and end the workflow with an
explicit aiconfig-targeting handoff so the new variation is actually
servable.
- aiconfig-projects: prefer the MCP get-project tool for verification;
warn about response-shape variation in jq filters.
- aiconfig-tools: PATCH the tools field alone, never bundle
instructions/messages/model/parameters
- aiconfig-migrate: rewrite all 'delegates to' language as manual
hand-offs, and add an explicit aiconfig-targeting step between Stage 2
and Stage 4.
* feat: [REL-13105] Add Claude Code plugin manifest and skill symlinks
Add .claude-plugin/plugin.json to enable installation as a Claude Code
plugin. Add flat symlinks under skills/ so Claude Code's one-level-deep
skill discovery resolves to the existing domain-organized SKILL.md files.
Cursor and Codex plugin structures are unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs: [REL-13105] Add Claude Code plugin install instructions and missing skills to README
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: [REL-13105] Update repo references to ai-tooling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: [REL-13105] Update plugin description wording
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Eric Angeles <eangeles@launchdarkly.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* updating ai config tools to use remote mcp tools
* fixing bugs
* we want to keep projects skill the same for now
* [REL-12454] supporting approvals (#13)
supporting approvals
* improve ai configs skills based on evals
devin-ai-integration[bot]