mirror of
https://github.com/workos/skills.git
synced 2026-09-14 20:07:08 +08:00
feat(workos): add terminology reference for docs URL lookups (#21)
* feat(workos): add terminology reference for docs URL lookups
New `references/workos-terms.md` maps WorkOS-specific terms
(Sign-in endpoint, initiate_login_uri, Redirect URI, Organization ID,
Admin Portal intent, JWKS, Sealed session, WORKOS_COOKIE_PASSWORD, etc.)
to canonical docs URLs. 10 seed rows, all URLs verified via WebFetch.
Router gains **Rule 0** for lookup-shaped queries ("what is X",
"docs URL for X") with explicit anti-triggers against setup phrasing
("set up Vault", "enable Admin Portal") so feature-setup requests
still route to Rule 3.
SKILL.md tightening:
- Frontmatter description shortened to ~485 chars, terminology-first
- How-to-Use rewritten as imperative MUST, not advisory
- llms.txt prohibition scoped to Rule 0 (Rules 7/8 still use it
for vague/no-match cases)
- "suggest open a PR" rather than "add a row" (agents can't persist
edits across sessions reliably)
New eval cases (`scripts/eval/cases/terminology.yaml`):
- terms-initiate-login-uri
- terms-sign-in-endpoint
Fixes the failure mode where agents fetched llms.txt blind on
"what is X" queries instead of using curated references.
* test(workos): add skill-creator eval harness + prompt/assertion spec
Adds `plugins/workos/skills/workos/evals/evals.json` — four realistic
terminology lookup prompts with assertion-level grading (content-contains-any/all/none
checks including explicit anti-pattern URLs the agent should NOT invent).
Complements the existing substring-based `scripts/eval/cases/terminology.yaml`:
the assertion format catches failure modes that flowStep substring matching
misses (e.g., "does the answer mention 'Dashboard' AND 'Redirects'",
"does it avoid inventing `/docs/initiate-login-uri`").
Workspace outputs under plugins/**/*-workspace/ are gitignored — iteration
results are ephemeral, only the spec is committed.
Iteration 1 result (4 evals × 2 arms, graded via grade.py):
With skill: 100% (17/17 assertions)
Without skill: 77.5% (13/17)
Delta: +22.5%
* chore: gitignore skill-creator eval workspaces
This commit is contained in:
Nick Nisi
@@ -7,3 +7,6 @@ scripts/eval/labels.jsonl
|
||||
|
||||
# Keep ideation docs untracked
|
||||
/docs/ideation/
|
||||
|
||||
# Skill-creator eval workspaces (iteration outputs, not source)
|
||||
plugins/**/*-workspace/
|
||||
|
||||
@@ -1,22 +1,24 @@
|
||||
---
|
||||
name: workos
|
||||
description: Use when the user is implementing, debugging, migrating to, or asking about WorkOS products — AuthKit, SSO/SAML, Directory Sync/SCIM, RBAC, FGA, MFA, Vault, Audit Logs, Admin Portal, webhooks/events, Custom Domains, backend SDKs (Node, Python, Go, Ruby, PHP, .NET, Kotlin, Elixir), or migrations from Auth0, Clerk, Cognito, Firebase, Supabase, Stytch, Descope, or Better Auth. Also use when code imports from @workos-inc/* packages or references workos.com docs. Routes the task to the correct reference file and surfaces non-obvious gotchas the model tends to get wrong from training data.
|
||||
description: Use when the user asks for a WorkOS docs URL, term, or dashboard field (Sign-in endpoint, initiate_login_uri, Redirect URI, `WORKOS_*` env vars), or is implementing, debugging, or migrating WorkOS — AuthKit, SSO/SAML, Directory Sync, RBAC, FGA, MFA, Vault, Audit Logs, Admin Portal, webhooks, Custom Domains, or migrating from Auth0, Clerk, Cognito, Firebase, Supabase, Stytch, Descope, or Better Auth. Also triggers on @workos-inc/* imports.
|
||||
---
|
||||
|
||||
# WorkOS Skill Router
|
||||
|
||||
## How to Use
|
||||
|
||||
When a user needs help with WorkOS, consult the tables below to route to the right reference.
|
||||
**This file is a router, NOT the answer.** Before responding to the user:
|
||||
|
||||
## Loading References
|
||||
|
||||
**All references** are topic files in the `references/` directory. Read the file and follow its instructions (fetch docs first, then use gotchas to avoid common traps).
|
||||
1. Match the request to a reference file using Rule 0 and the decision tree below.
|
||||
2. **You MUST Read the matched reference file with the Read tool before producing any answer, URL, or code.** If you have not Read a reference, you have not followed this skill.
|
||||
3. Follow the instructions inside the reference (it will tell you which live docs to fetch with WebFetch and which gotchas to avoid).
|
||||
|
||||
**Exception**: Widget requests use the `workos-widgets` skill via the Skill tool — it has its own multi-framework orchestration.
|
||||
|
||||
## Topic → Reference Map
|
||||
|
||||
> Terminology lookups — "what is X", "docs URL for X" — are handled by **Rule 0** below, not this topic map. They route to `references/workos-terms.md`.
|
||||
|
||||
### AuthKit Installation (Read `references/{name}.md`)
|
||||
|
||||
| User wants to... | Read file |
|
||||
@@ -94,6 +96,24 @@ Feature topic files above include endpoint tables for their respective APIs. Use
|
||||
|
||||
Apply these rules in order. First match wins.
|
||||
|
||||
### 0. Terminology / Docs URL Lookup
|
||||
|
||||
**Triggers**: Lookup-shaped phrasing — "what is X", "what does X mean", "docs URL for X", "where's the docs on X", "canonical link for X", "where do I configure X in the dashboard" — where X is a WorkOS-specific config field, endpoint, env var, or term. Examples: `initiate_login_uri`, "Sign-in endpoint", "Redirect URI", dashboard field names, `WORKOS_*` environment variables.
|
||||
|
||||
**Do NOT fire Rule 0** for setup-shaped phrasing like "set up Vault", "enable Admin Portal", "configure MFA" — those route to Rule 3 (Feature-Specific).
|
||||
|
||||
**Action**:
|
||||
|
||||
1. Read `references/workos-terms.md` — a curated table mapping WorkOS terms to canonical docs URLs.
|
||||
2. If the term is in the table, use the summary to answer; WebFetch the listed URL only if the user wants more detail.
|
||||
3. If the term is NOT in the table, follow the "Still not here?" fallback at the bottom of that file. When you find the canonical URL, answer the user and suggest they open a PR to add a row.
|
||||
|
||||
**For terminology lookups**, do NOT WebFetch `llms.txt` or guess `workos.com/docs/...` URLs before reading the terms file. (Rules 7 and 8 use `llms.txt` for different purposes — this prohibition is scoped to Rule 0 only.)
|
||||
|
||||
**Why this wins**: Terminology lookups happen independent of feature/framework/migration context. They need to short-circuit routing, not fall through to "Vague or General" (Rule 7).
|
||||
|
||||
---
|
||||
|
||||
### 1. Migration Context
|
||||
|
||||
**Triggers**: User mentions migrating FROM another provider (Auth0, Clerk, Cognito, Firebase, Supabase, Stytch, Descope, Better Auth, standalone SSO API).
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"skill_name": "workos",
|
||||
"evals": [
|
||||
{
|
||||
"id": 0,
|
||||
"name": "initiate-login-uri-lookup",
|
||||
"prompt": "I'm configuring an OIDC-style integration and saw the term `initiate_login_uri` mentioned. I'm using WorkOS AuthKit. What exactly is this and where do I actually configure it? Give me the canonical docs URL so I can read more. Keep the answer tight.",
|
||||
"files": [],
|
||||
"assertions": [
|
||||
{"text": "Mentions Sign-in endpoint as the WorkOS-side term for this concept", "kind": "content_contains_any", "needles": ["Sign-in endpoint", "sign-in endpoint", "Sign-In endpoint"]},
|
||||
{"text": "Points to Dashboard → Redirects as the config location", "kind": "content_contains_all", "needles": ["Dashboard", "Redirects"]},
|
||||
{"text": "Cites an actual workos.com/docs URL (not llms.txt)", "kind": "content_contains_any", "needles": ["workos.com/docs/authkit", "workos.com/docs/sso", "workos.com/docs/reference"]},
|
||||
{"text": "Does NOT invent a URL like /docs/initiate-login-uri", "kind": "content_contains_none", "needles": ["/docs/initiate-login-uri", "/docs/initiate_login_uri", "/docs/oidc/initiate-login-uri", "/docs/reference/initiate_login_uri"]},
|
||||
{"text": "Does NOT use llms.txt as the primary source", "kind": "content_contains_none", "needles": ["fetch llms.txt", "fetched llms.txt", "from llms.txt"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"name": "sign-in-endpoint-lookup",
|
||||
"prompt": "I'm staring at the WorkOS Dashboard and I see a field called 'Sign-in endpoint' in the Redirects section. What IS this exactly — why does WorkOS need it if I'm already passing a Redirect URI? Link me to the canonical docs page.",
|
||||
"files": [],
|
||||
"assertions": [
|
||||
{"text": "Distinguishes Sign-in endpoint from Redirect URI", "kind": "content_contains_all", "needles": ["Sign-in endpoint", "Redirect URI"]},
|
||||
{"text": "Explains the bookmark / password-reset use case", "kind": "content_contains_any", "needles": ["bookmark", "password reset", "password-reset", "did not originate", "didn't originate", "originate from your app"]},
|
||||
{"text": "Cites a workos.com/docs URL", "kind": "content_contains_any", "needles": ["workos.com/docs/authkit", "workos.com/docs/reference"]},
|
||||
{"text": "Does NOT invent /docs/sign-in-endpoint", "kind": "content_contains_none", "needles": ["/docs/sign-in-endpoint", "/docs/authkit/sign-in-endpoint", "/docs/reference/sign-in-endpoint"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"name": "cookie-password-lookup",
|
||||
"prompt": "My WorkOS AuthKit app is throwing an error about `WORKOS_COOKIE_PASSWORD` being invalid. What is this env var, what are the requirements (length, format), how do I generate a valid one, and where are the official docs?",
|
||||
"files": [],
|
||||
"assertions": [
|
||||
{"text": "States the 32+ character requirement", "kind": "content_contains_any", "needles": ["32+ character", "32+ chars", "32 characters", "at least 32", "minimum 32", "32-character"]},
|
||||
{"text": "Shows how to generate one with openssl", "kind": "content_contains_all", "needles": ["openssl", "rand"]},
|
||||
{"text": "Mentions session cookie encryption / sealing", "kind": "content_contains_any", "needles": ["seal", "encrypt", "session cookie"]},
|
||||
{"text": "Cites a workos.com/docs URL", "kind": "content_contains_any", "needles": ["workos.com/docs"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"name": "org-vs-connection-id",
|
||||
"prompt": "For my SSO integration I see both Organization ID and Connection ID as options when generating an authorization URL. What's the actual difference, and which one should I pass to getAuthorizationUrl in typical multi-tenant B2B setup? Docs link please.",
|
||||
"files": [],
|
||||
"assertions": [
|
||||
{"text": "Recommends Organization ID for multi-tenant B2B", "kind": "content_contains_any", "needles": ["Organization ID", "organizationId", "organization_id"]},
|
||||
{"text": "Explains what Connection ID is and when to use it", "kind": "content_contains_any", "needles": ["Connection ID", "connectionId", "connection_id"]},
|
||||
{"text": "Explains why Organization ID is preferred (connection routing)", "kind": "content_contains_any", "needles": ["group of users", "preferred", "typically", "let the org", "org can pick", "active connection", "specific connection"]},
|
||||
{"text": "Cites a workos.com/docs URL", "kind": "content_contains_any", "needles": ["workos.com/docs/sso", "workos.com/docs/authkit", "workos.com/docs/reference"]}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
# WorkOS Terminology → Canonical Docs URLs
|
||||
|
||||
If this file conflicts with fetched docs, follow the docs. URLs here are canonical at time of writing; if a user reports a broken link, WebFetch to confirm and update the row.
|
||||
|
||||
## How to Use
|
||||
|
||||
User asked about a WorkOS term, dashboard field, environment variable, or configuration concept? Look it up here first. The table gives you:
|
||||
|
||||
- **Term** — exact name as it appears in the WorkOS Dashboard, SDK, or docs
|
||||
- **What it is** — one-line definition (enough to answer simple "what is X" questions without a fetch)
|
||||
- **Canonical URL** — WebFetch this if the user wants the full reference
|
||||
- **See also** — deeper reference file to Read when the task goes beyond terminology
|
||||
|
||||
## Terms
|
||||
|
||||
| Term | What it is | Canonical URL | See also |
|
||||
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | ------------------------ |
|
||||
| Redirect URI | URL WorkOS redirects to after successful authentication; configured in Dashboard → Redirects. The `redirect_uri` request parameter must match one of the configured values EXACTLY (including trailing slash). | https://workos.com/docs/reference/authkit/authentication/get-authorization-url/redirect-uri | `workos-authkit-base.md` |
|
||||
| Sign-in endpoint | URL on YOUR app that AuthKit redirects to when a sign-in request did not originate from your app (e.g., bookmark of hosted sign-in, password-reset email). Configured in Dashboard → Redirects. | https://workos.com/docs/authkit/vanilla/nodejs#configure-sign-in-endpoint | `workos-authkit-base.md` |
|
||||
| `initiate_login_uri` | OIDC client metadata parameter an IdP reads to begin IdP-initiated login at the RP. For WorkOS configuration purposes this maps to the **Sign-in endpoint** in Dashboard → Redirects; no distinct WorkOS docs page exists under this name. | https://workos.com/docs/authkit/vanilla/nodejs#configure-sign-in-endpoint | `workos-authkit-base.md` |
|
||||
| Sign-out redirect | URL users are redirected to after logging out of AuthKit. Configured in Dashboard → Redirects and/or passed as the `return_to` query parameter on the logout URL. | https://workos.com/docs/reference/authkit/logout/get-logout-url | `workos-authkit-base.md` |
|
||||
| Organization ID | Identifier for a group of users (typically a tenant/customer org). Preferred parameter when initiating SAML/OIDC flows — pass this instead of Connection ID so the org can pick its active connection. | https://workos.com/docs/sso/overview | `workos-sso.md` |
|
||||
| Connection ID | Identifier for a specific SSO connection (auth method) belonging to an Organization. Use when you need to authenticate via a particular connection rather than letting the Org decide. | https://workos.com/docs/sso/overview | `workos-sso.md` |
|
||||
| Admin Portal `intent` | Query parameter on `generateLink` that selects which Admin Portal flow to open. Valid values: `sso`, `dsync`, `audit_logs`, `log_streams`, `domain_verification`, `certificate_renewal`, `bring_your_own_key`. | https://workos.com/docs/reference/admin-portal/portal-link/generate | `workos-admin-portal.md` |
|
||||
| JWKS endpoint | Public key set endpoint used to verify signatures on AuthKit-issued session access tokens. | https://workos.com/docs/reference/authkit/session-tokens/jwks | `workos-api-authkit.md` |
|
||||
| Sealed session | AuthKit session data encrypted and stored in a cookie. "Sealing" = encrypting with the cookie password at sign-in; "unsealing" = decrypting via `loadSealedSession()` / `authenticateWithSessionCookie()` on each request. | https://workos.com/docs/reference/authkit/session-helpers/load-sealed-session | `workos-node.md` |
|
||||
| `WORKOS_COOKIE_PASSWORD` | 32+ character password used to seal/unseal the AuthKit session cookie. Must be identical across all instances of your app. Generate with `openssl rand -base64 32`. | https://workos.com/docs/authkit/vanilla/nodejs | `workos-authkit-base.md` |
|
||||
|
||||
## Still not here?
|
||||
|
||||
1. Check the "See also" column for the closest feature reference and Read it — the term may be covered in context there.
|
||||
2. If still unclear, WebFetch https://workos.com/docs/llms.txt and search for the term.
|
||||
3. If you find a canonical URL for a term that wasn't in this table, answer the user, then **suggest they open a PR** adding a row to this file. (This file is human-maintained; you can't reliably persist edits from a user session.)
|
||||
|
||||
## Verification
|
||||
|
||||
URLs drift when WorkOS reorganizes docs. Verify a URL by WebFetch only when (a) the user reports a broken link, or (b) you're about to write the URL into a file the user will commit. Routine chat answers don't require verification — that wastes tokens.
|
||||
@@ -0,0 +1,49 @@
|
||||
- id: terms-initiate-login-uri
|
||||
product: authkit
|
||||
skill: workos-terms
|
||||
skillType: hand-crafted
|
||||
prompt: |
|
||||
What is `initiate_login_uri` in WorkOS? Give me the canonical docs URL
|
||||
where I can configure this for my AuthKit app.
|
||||
expected:
|
||||
methods: []
|
||||
envVars: []
|
||||
imports: []
|
||||
params: []
|
||||
flowSteps:
|
||||
- Sign-in endpoint
|
||||
- Dashboard
|
||||
- Redirects
|
||||
- workos.com/docs
|
||||
antiPatterns:
|
||||
- fetch llms.txt first
|
||||
- guess docs URL
|
||||
hallucinations:
|
||||
- /docs/initiate-login-uri
|
||||
- /docs/oidc/initiate-login-uri
|
||||
- /docs/reference/initiate_login_uri
|
||||
|
||||
- id: terms-sign-in-endpoint
|
||||
product: authkit
|
||||
skill: workos-terms
|
||||
skillType: hand-crafted
|
||||
prompt: |
|
||||
What is the "Sign-in endpoint" in WorkOS AuthKit, and where do I configure it?
|
||||
Give me the docs URL.
|
||||
expected:
|
||||
methods: []
|
||||
envVars: []
|
||||
imports: []
|
||||
params: []
|
||||
flowSteps:
|
||||
- Dashboard
|
||||
- Redirects
|
||||
- did not originate
|
||||
- workos.com/docs
|
||||
antiPatterns:
|
||||
- guess the URL
|
||||
- fetch llms.txt blind
|
||||
hallucinations:
|
||||
- /docs/sign-in-endpoint
|
||||
- /docs/authkit/sign-in-endpoint
|
||||
- /docs/reference/sign-in-endpoint
|
||||
Reference in New Issue
Block a user