skills/firecrawl__skills
1
0
Fork 0
mirror of https://github.com/firecrawl/skills.git synced 2026-09-14 18:22:20 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat: make the catalog a pure read-only distribution repo

Browse Source 
Every category is now a CI-synced mirror: workflow skills stay
authored in firecrawl/firecrawl-workflows (sync direction reversed),
and the research/developer index skills move to firecrawl/cli as CLI
skills backing the firecrawl research/developer commands, retiring the
reference category. Add repo-wide lockdown config and bundle the index
skills in the editor plugins via skills/cli/.
This commit is contained in:
Chenxin Yan
2026-08-21 02:51:45 -04:00
parent 9446658544
commit 2ed84b7c69
11 changed files with 329 additions and 238 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude-plugin/plugin.json
+2
View File
@@ -22,6 +22,8 @@
"./skills/cli/firecrawl-monitor",
"./skills/cli/firecrawl-parse",
"./skills/cli/firecrawl-download",
"./skills/cli/firecrawl-research-index",
"./skills/cli/firecrawl-developer-index",
"./skills/build/firecrawl-build",
"./skills/build/firecrawl-build-onboarding",
"./skills/build/firecrawl-build-scrape",
.github/lockdown.yml
+24
View File
@@ -0,0 +1,24 @@
# Configuration for the Repo Lockdown app: https://github.com/apps/repo-lockdown
# This repo is a read-only distribution catalog; every directory is CI-synced.
issues:
comment: >
This catalog is read-only — every skill is synced from its source repo.
Please open this issue against the source: CLI + index skills →
[firecrawl/cli](https://github.com/firecrawl/cli), build/SDK skills →
[firecrawl/firecrawl](https://github.com/firecrawl/firecrawl) (`skills/`),
workflow skills →
[firecrawl/firecrawl-workflows](https://github.com/firecrawl/firecrawl-workflows).
close: true
lock: true
pulls:
comment: >
This catalog is read-only and CI-synced; direct changes are overwritten on
the next sync. Please open this PR against the source repo: CLI + index
skills → [firecrawl/cli](https://github.com/firecrawl/cli), build/SDK
skills → [firecrawl/firecrawl](https://github.com/firecrawl/firecrawl)
(`skills/`), workflow skills →
[firecrawl/firecrawl-workflows](https://github.com/firecrawl/firecrawl-workflows).
close: true
lock: true
.github/workflows/sync-workflows-mirror.yml
-62
View File
@@ -1,62 +0,0 @@
name: Sync workflow skills to legacy mirror
# Mirrors skills/workflows/ into firecrawl/firecrawl-workflows so the legacy
# install path `npx skills add firecrawl/firecrawl-workflows` keeps working
# and keeps meaning "the 16 workflow skills", not the whole catalog.
on:
push:
branches: [main]
paths:
- "skills/workflows/**"
- ".github/workflows/sync-workflows-mirror.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: workflows-mirror-sync
cancel-in-progress: false
jobs:
sync:
runs-on: ubuntu-latest
steps:
- name: Checkout catalog
uses: actions/checkout@v6
with:
path: catalog
- name: Checkout legacy mirror
uses: actions/checkout@v6
with:
repository: firecrawl/firecrawl-workflows
ssh-key: ${{ secrets.WORKFLOWS_MIRROR_DEPLOY_KEY }}
path: mirror
- name: Sync skills
run: |
cd mirror
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
rm -rf skills
mkdir -p skills
cp -R ../catalog/skills/workflows/. skills/
# The catalog category README does not belong in the flat mirror.
rm -f skills/README.md
git add -A
if git diff --cached --quiet; then
echo "No changes to sync"
exit 0
fi
SHORT_SHA="${GITHUB_SHA:0:7}"
git commit -m "Sync from firecrawl/skills@${SHORT_SHA}" \
-m "Source: https://github.com/${GITHUB_REPOSITORY}/commit/${GITHUB_SHA}"
# Retry once if another sync landed first; never force-push.
git push origin main || { git pull --rebase origin main && git push origin main; }
AGENTS.md
+8 -23
View File
@@ -1,38 +1,23 @@
# Firecrawl Skills Catalog
Full catalog of Firecrawl agent skills, organized by category under `skills/`.
Read-only distribution catalog for all Firecrawl agent skills. **Every skill directory here is a CI-synced mirror — do not edit skills in this repo.**
## Layout and edit rules
## Layout and sources
- `skills/cli/` — **mirror** of `firecrawl/cli` `skills/`. Do not edit here; CI overwrites on every sync. PR against `firecrawl/cli`.
- `skills/build/` — **mirror** of the `firecrawl` monorepo `skills/`. Do not edit here; CI overwrites on every sync. PR against `firecrawl/firecrawl`.
- `skills/workflows/` — **source of truth**, authored here. Safe to edit.
- `skills/reference/` — **source of truth**, authored here. Safe to edit.
- `skills/cli/` — mirror of `firecrawl/cli` `skills/` (CLI skills + the research/developer index skills)
- `skills/build/` — mirror of the `firecrawl` monorepo `skills/`
- `skills/workflows/` — mirror of `firecrawl/firecrawl-workflows` `skills/`
Plugin metadata and top-level docs are safe to edit:
- `.cursor-plugin/`, `.claude-plugin/`, `.codex-plugin/`
- `README.md`, `AGENTS.md`, `.mcp.json`, `mcp.json`
- `.github/workflows/` (sync automation)
Only repo metadata is authored here: `.cursor-plugin/`, `.claude-plugin/`, `.codex-plugin/`, `README.md`, `AGENTS.md`, `.mcp.json`, `mcp.json`, `.github/`.
## Routing rule
CLI skills → PR `firecrawl/cli`. Build/SDK skills → PR `firecrawl` (monorepo, `skills/`). Everything else (workflows, reference) → PR this repo. New skills always land in an existing repo — when in doubt, this repo under `skills/workflows/`.
CLI skills (including the research/developer index skills) → PR `firecrawl/cli`. Build/SDK skills → PR `firecrawl` (monorepo, `skills/`). Workflow skills → PR `firecrawl/firecrawl-workflows`. Never PR skill content against this catalog — CI overwrites it on the next sync.
## Intent
Use the skills here when the task is:
- live web work during a session (CLI skills)
- live web work during a session, or querying the research paper / developer indexes (CLI skills)
- adding Firecrawl to a codebase, choosing between `/scrape`, `/search`, and `/interact`, getting `FIRECRAWL_API_KEY` into `.env` (build skills)
- end-to-end recipes like lead gen, deep research, SEO audits (workflow skills)
- querying the research paper index or developer index, which `/search` does not query (reference skills)
## Authoring Rules (workflows + reference)
- Keep each `SKILL.md` concise and trigger-oriented.
- Lead with "use this when..." guidance.
- Favor endpoint names in slash notation: `/scrape`, `/search`, `/interact`.
- Keep CLI references short and defer to `firecrawl/cli` instead of duplicating command manuals.
- Treat [`https://www.firecrawl.dev/agent-onboarding/SKILL.md`](https://www.firecrawl.dev/agent-onboarding/SKILL.md) as the canonical source for the two-path framing.
- Skill layout: `skills/<category>/<skill-name>/SKILL.md`, deeper docs in `skills/<category>/<skill-name>/references/`, one level deep.
CLAUDE.md
+5 -6
View File
@@ -1,10 +1,9 @@
# Firecrawl Skills
Full catalog of Firecrawl agent skills, organized under `skills/` by category:
Read-only distribution catalog of Firecrawl agent skills. Every directory under `skills/` is CI-synced from its source repo:
- `skills/cli/` — CLI skills for live web work (mirror of `firecrawl/cli`; do not edit here)
- `skills/build/` — app-integration skills (mirror of the `firecrawl` monorepo; do not edit here)
- `skills/workflows/` — end-to-end session recipes (authored here)
- `skills/reference/` — research paper index and developer index skills (authored here)
- `skills/cli/` — CLI skills + research/developer index skills (source: `firecrawl/cli`)
- `skills/build/` — app-integration skills (source: the `firecrawl` monorepo)
- `skills/workflows/` — end-to-end session recipes (source: `firecrawl/firecrawl-workflows`)
See `AGENTS.md` for the routing rule and authoring guidance.
Never edit skills here — PR the source repo. See `AGENTS.md` for the routing rule.
README.md
+16 -19
View File
@@ -2,6 +2,8 @@
The full catalog of Firecrawl skills for AI coding agents, following the [Agent Skills](https://agentskills.io) format. Available as a plugin for Claude Code, Cursor, and OpenAI Codex.
> **This repo is read-only.** Every skill here is CI-synced from its source repo — see [Contributing](#contributing-where-does-my-pr-go). Install from here; contribute at the source.
## Install
```bash
@@ -24,24 +26,26 @@ npx -y firecrawl-cli@latest init
| Category | Path | What it's for | Source of truth |
|---|---|---|---|
| CLI | [`skills/cli/`](./skills/cli) | Live web work during an agent session: search, scrape, crawl, interact from the terminal | **Mirror** of [`firecrawl/cli`](https://github.com/firecrawl/cli/tree/main/skills) — do not edit here |
| Build | [`skills/build/`](./skills/build) | Integrating Firecrawl APIs into product code: SDKs, REST, endpoint selection, API keys | **Mirror** of the [`firecrawl` monorepo `skills/`](https://github.com/firecrawl/firecrawl/tree/main/skills) — do not edit here |
| Workflows | [`skills/workflows/`](./skills/workflows) | End-to-end session recipes: lead gen, deep research, SEO audit, knowledge bases, and more | **Authored here** — PRs welcome |
| Reference | [`skills/reference/`](./skills/reference) | Query Firecrawl's research paper index and developer index | **Authored here** — PRs welcome |
| CLI | [`skills/cli/`](./skills/cli) | Live web work during an agent session (search, scrape, crawl, interact from the terminal), plus the research paper index and developer index skills | [`firecrawl/cli`](https://github.com/firecrawl/cli/tree/main/skills) |
| Build | [`skills/build/`](./skills/build) | Integrating Firecrawl APIs into product code: SDKs, REST, endpoint selection, API keys | [`firecrawl` monorepo `skills/`](https://github.com/firecrawl/firecrawl/tree/main/skills) |
| Workflows | [`skills/workflows/`](./skills/workflows) | End-to-end session recipes: lead gen, deep research, SEO audit, knowledge bases, and more | [`firecrawl/firecrawl-workflows`](https://github.com/firecrawl/firecrawl-workflows) |
## Contributing: where does my PR go?
> **CLI skills → PR [`firecrawl/cli`](https://github.com/firecrawl/cli). Build/SDK skills → PR the [`firecrawl`](https://github.com/firecrawl/firecrawl) monorepo (`skills/`). Everything else (workflows, reference) → PR this repo. New skills always land in an existing repo — when in doubt, this repo under `skills/workflows/`.**
Mirrored directories (`skills/cli/`, `skills/build/`) are overwritten by CI on every upstream sync; PRs against them will be redirected to the source repo.
> **CLI skills (including the research/developer index skills) → PR [`firecrawl/cli`](https://github.com/firecrawl/cli). Build/SDK skills → PR the [`firecrawl`](https://github.com/firecrawl/firecrawl) monorepo (`skills/`). Workflow skills → PR [`firecrawl/firecrawl-workflows`](https://github.com/firecrawl/firecrawl-workflows). Never PR this catalog — CI overwrites every directory on the next sync.**
## Skills
### CLI (`skills/cli/`, mirror)
### CLI (`skills/cli/`)
Skills that teach agents to use the [Firecrawl CLI](https://github.com/firecrawl/cli) for live web work: `firecrawl`, `firecrawl-scrape`, `firecrawl-search`, `firecrawl-crawl`, `firecrawl-map`, `firecrawl-interact`, `firecrawl-agent`, `firecrawl-monitor`, `firecrawl-parse`, `firecrawl-download`.
Skills that teach agents the [Firecrawl CLI](https://github.com/firecrawl/cli) for live web work — `firecrawl`, `firecrawl-scrape`, `firecrawl-search`, `firecrawl-crawl`, `firecrawl-map`, `firecrawl-interact`, `firecrawl-agent`, `firecrawl-monitor`, `firecrawl-parse`, `firecrawl-download` — plus the index skills backing the `firecrawl research` and `firecrawl developer` commands:
### Build (`skills/build/`, mirror)
| Skill | Description |
|---|---|
| [`firecrawl-research-index`](./skills/cli/firecrawl-research-index) | Find papers in the research paper index — biomedical and life-science literature (PubMed, bioRxiv, medRxiv) plus arXiv preprints |
| [`firecrawl-developer-index`](./skills/cli/firecrawl-developer-index) | Answer developer questions from issues, pull requests, READMEs, and documentation pages |
### Build (`skills/build/`)
| Skill | Description |
|---|---|
@@ -51,17 +55,10 @@ Skills that teach agents to use the [Firecrawl CLI](https://github.com/firecrawl
| [`firecrawl-build-search`](./skills/build/firecrawl-build-search) | Integrate `/search` for discovery-first workflows |
| [`firecrawl-build-interact`](./skills/build/firecrawl-build-interact) | Integrate `/interact` for clicks, forms, and dynamic flows after scrape |
### Workflows (`skills/workflows/`, authored here)
### Workflows (`skills/workflows/`)
End-to-end recipes such as [`firecrawl-lead-gen`](./skills/workflows/firecrawl-lead-gen), [`firecrawl-deep-research`](./skills/workflows/firecrawl-deep-research), [`firecrawl-seo-audit`](./skills/workflows/firecrawl-seo-audit), [`firecrawl-competitive-intel`](./skills/workflows/firecrawl-competitive-intel), and more — see [`skills/workflows/`](./skills/workflows) for the full set, and [`firecrawl-workflows`](./skills/workflows/firecrawl-workflows) for the umbrella skill and authoring guide.
### Reference (`skills/reference/`, authored here)
| Skill | Description |
|---|---|
| [`firecrawl-research-index`](./skills/reference/firecrawl-research-index) | Find papers in the research paper index — biomedical and life-science literature (PubMed, bioRxiv, medRxiv) plus arXiv preprints |
| [`firecrawl-developer-index`](./skills/reference/firecrawl-developer-index) | Answer developer questions from issues, pull requests, READMEs, and documentation pages |
## MCP Server
The plugin includes Firecrawl MCP configuration for the official [Firecrawl MCP server](https://github.com/firecrawl/firecrawl-mcp-server), so editors that support bundled MCP metadata can wire Firecrawl tools with `FIRECRAWL_API_KEY`.
@@ -84,9 +81,9 @@ Get your API key at [firecrawl.dev/app](https://www.firecrawl.dev/app). If you d
## Which skills do I need?
- **"Search/scrape the web for me right now, during this session"** → CLI skills (`skills/cli/`)
- **"Query published research papers or developer docs indexes"** → index skills (`skills/cli/`)
- **"Add Firecrawl to this codebase"** → build skills (`skills/build/`)
- **"Run an end-to-end task like lead gen or deep research"** → workflow skills (`skills/workflows/`)
- **"Query published research papers or developer docs indexes"** → reference skills (`skills/reference/`)
Both usage paths follow Firecrawl's onboarding skill (same install, different use cases): [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md).
docs/setup-sync-keys-prompt.md
+60
View File
@@ -0,0 +1,60 @@
# Task: set up deploy keys for the Firecrawl skills sync workflows
You need admin access to these GitHub repos: `firecrawl/skills`, `firecrawl/cli`, `firecrawl/firecrawl`, `firecrawl/firecrawl-workflows`, and the `gh` CLI authenticated (`gh auth status`).
Three CI sync workflows push commits across repos. Each needs its own SSH deploy keypair: the **public** key goes on the repo being pushed to (deploy key, write access), the **private** key goes on the repo running the workflow (Actions secret). Do not swap these.
| # | Source (private key as secret) | Secret name | Target (public key as deploy key) |
|---|---|---|---|
| A | `firecrawl/cli` | `CATALOG_DEPLOY_KEY` | `firecrawl/skills` |
| B | `firecrawl/firecrawl` | `CATALOG_DEPLOY_KEY` | `firecrawl/skills` |
| C | `firecrawl/skills` | `WORKFLOWS_MIRROR_DEPLOY_KEY` | `firecrawl/firecrawl-workflows` |
A deploy key attaches to exactly one repo, so A and B need separate keypairs even though both target `firecrawl/skills`.
Run exactly this:
```bash
cd "$(mktemp -d)"
ssh-keygen -t ed25519 -N "" -f cli-to-catalog -C "sync: firecrawl/cli -> firecrawl/skills"
ssh-keygen -t ed25519 -N "" -f monorepo-to-catalog -C "sync: firecrawl/firecrawl -> firecrawl/skills"
ssh-keygen -t ed25519 -N "" -f catalog-to-workflows -C "sync: firecrawl/skills -> firecrawl/firecrawl-workflows"
# Public keys -> deploy keys on targets (write access required)
gh repo deploy-key add cli-to-catalog.pub -R firecrawl/skills --allow-write -t "sync from firecrawl/cli"
gh repo deploy-key add monorepo-to-catalog.pub -R firecrawl/skills --allow-write -t "sync from firecrawl/firecrawl"
gh repo deploy-key add catalog-to-workflows.pub -R firecrawl/firecrawl-workflows --allow-write -t "sync from firecrawl/skills"
# Private keys -> Actions secrets on sources (names must match exactly)
gh secret set CATALOG_DEPLOY_KEY -R firecrawl/cli < cli-to-catalog
gh secret set CATALOG_DEPLOY_KEY -R firecrawl/firecrawl < monorepo-to-catalog
gh secret set WORKFLOWS_MIRROR_DEPLOY_KEY -R firecrawl/skills < catalog-to-workflows
# Destroy local copies; keys now live only in GitHub
cd - && rm -rf "$OLDPWD"
```
## Verify
```bash
gh repo deploy-key list -R firecrawl/skills # expect 2 keys, both read/write
gh repo deploy-key list -R firecrawl/firecrawl-workflows # expect 1 key, read/write
gh secret list -R firecrawl/cli # expect CATALOG_DEPLOY_KEY
gh secret list -R firecrawl/firecrawl # expect CATALOG_DEPLOY_KEY
gh secret list -R firecrawl/skills # expect WORKFLOWS_MIRROR_DEPLOY_KEY
```
If the sync workflows already exist on `main`, also trigger one end-to-end run:
```bash
gh workflow run sync-catalog.yml -R firecrawl/cli && gh run watch -R firecrawl/cli
```
Success: a `Sync from firecrawl/cli@<sha>` commit lands on `firecrawl/skills` `main`, or the run logs "No changes to sync". Failure `Permission denied (publickey)` means a public/private half was swapped — redo that pair.
## Report back
- Output of the five verify commands
- Whether the test run succeeded
- Anything you had to deviate from
docs/skills-structure-recommendation.md
+210
View File
@@ -0,0 +1,210 @@
# Skills Structure: Recommendation & Migration Plan
Response to the August 2026 skills-repos handoff. Additive-only per the rule:
every install command that works today keeps working.
## Recommendation in one paragraph
Make `firecrawl/skills` the **full catalog** — the one repo users install from
and the one path we promote everywhere. Keep each skill's **source of truth
co-located with the code it teaches**: CLI skills stay in `firecrawl/cli`,
build skills move to the `firecrawl` monorepo next to the SDKs, and workflow +
reference skills are authored directly in the catalog. CI (never humans) syncs
copies from each source into the catalog, and from the catalog out to the
legacy `firecrawl-workflows` mirror. No install path moves or breaks; changes
are additive.
## Why this works: install paths and authoring are decoupled
skills.sh keys install counts to `owner/repo/skill-name` — the repo users
install from, not where the skill is authored, and not the directory path
inside the repo. Two consequences:
1. Moving *authorship* of the build skills into the monorepo costs zero
installs, because their install path (`firecrawl/skills`) never changes.
2. Reorganizing the catalog into category subdirectories does not reset any
counter, because the key is the skill name, not the path.
We only start new counters where we change the *promoted* install path: CLI
skills and workflows get new catalog counters starting at zero, while their
old paths keep working (and keep counting) for anyone still using them.
| Family | Source of truth (humans edit) | Install path (promoted) | Legacy path | Count effect |
|---|---|---|---|---|
| CLI (10) | `firecrawl/cli` `skills/` | catalog `skills/cli/` | `firecrawl/cli` keeps working | new catalog counters; old 725K freezes as legacy installs taper |
| Build (5) | `firecrawl` monorepo `skills/` (moved) | catalog `skills/build/` | n/a — same repo as today | **none** — 256K keeps accruing uninterrupted |
| Workflows (16) | catalog `skills/workflows/` (moved) | catalog `skills/workflows/` | `firecrawl-workflows` mirror keeps working | new catalog counters; old 498K keeps counting for legacy installers |
| Reference/index (2) | catalog `skills/reference/` | catalog `skills/reference/` | n/a — same repo as today | none |
## Target catalog layout
```
firecrawl/skills
├── skills/
│ ├── cli/ # MIRROR — synced from firecrawl/cli, do not edit here
│ │ └── firecrawl, firecrawl-scrape, … (10)
│ ├── build/ # MIRROR — synced from firecrawl monorepo, do not edit here
│ │ └── firecrawl-build, firecrawl-build-scrape, … (5)
│ ├── workflows/ # SOURCE — PRs welcome here
│ │ └── firecrawl-lead-gen, firecrawl-deep-research, … (16)
│ └── reference/ # SOURCE — PRs welcome here
│ └── firecrawl-developer-index, firecrawl-research-index
├── .claude-plugin/ .cursor-plugin/ .codex-plugin/ # consolidated plugins
├── .mcp.json
└── README.md # routing rule + mirror notice per category
```
The `npx skills` CLI walks skill containers up to three levels deep, so
`skills/<category>/<name>/SKILL.md` is discovered natively. `--skill <name>`
and `owner/repo@skill` are name-based, so
`npx skills add firecrawl/skills --skill firecrawl-build` works identically
before and after categorization.
Bare `npx skills add firecrawl/skills` changes meaning from "5 build skills"
to "pick from the full catalog" — accepted deliberately. Without `-y` the CLI
shows an interactive skill multi-select, so this is a picker over ~33 skills,
not a blind bulk install.
## Sync design
Four push-based workflows, modeled on Anthropic's battle-tested
claude-code-action → claude-code-base-action mirror (461+ automated sync
commits). Pattern per workflow: trigger `on: push` with a `paths:` filter +
`workflow_dispatch`, wipe **only the owned subdirectory** in the target,
`cp -r` fresh content, skip commit when `git diff --quiet`, commit as
`Sync from <repo>@<shortsha>` with the full source SHA in the body, push with
a **repo-scoped deploy key** (no PAT expiry, no `workflow` scope wall).
```
1. firecrawl/cli push(skills/**) ─▶ catalog skills/cli/
2. firecrawl (monorepo) push(skills/**) ─▶ catalog skills/build/
3. catalog push(skills/workflows/**) ─▶ firecrawl-workflows skills/
4. repo-lockdown on firecrawl-workflows (auto-close + lock PRs/issues with redirect comment)
```
Details:
- Workflows 1 and 2 write disjoint catalog paths; add
`concurrency: { group: catalog-sync }` per source repo and a
`git pull --rebase && git push` retry — never force-push.
- Workflow 3 copies **only the workflows subset**, so
`npx skills add firecrawl/firecrawl-workflows` keeps meaning "the 16
workflow skills," not the whole catalog.
- Mirrored directories get a `README.md` banner prepended on every sync:
"Mirror of `<source>` — do not edit here, PR against `<source>`." The
legacy mirror additionally gets issues disabled and branch protection so
only the deploy key writes.
- Not chosen: git subtree (slow, re-splits history every push, no benefit for
a snapshot mirror), pull-based cron (staleness + GitHub auto-disables
scheduled workflows in quiet public repos after 60 days), symlinks/submodules
(the skills CLI follows symlinks fragilely and does not clone submodules).
## `firecrawl init` revamp
Keep init — it is the only install surface whose UX we control — but fix what
it installs:
- Installs the **CLI skills by default** (a coding-agent CLI user's actual need).
- Shows a **multi-select for workflow skills** as optional extras.
- **Stops installing build skills** — they are for people integrating the SDK
into product code, not CLI users. Build skills remain one command away and
are what the editor plugins bundle.
- Implemented as a thin wrapper over
`npx skills add firecrawl/skills --skill … -y`, so init's install volume
accrues to catalog counters from day one and we maintain no install plumbing.
## Plugins (and the Codex fix)
Consolidate Cursor/Claude/Codex plugin metadata in the catalog repo, bundling
**CLI usage + build skills** (`skills/cli/` + `skills/build/`), not the 16
workflows — workflows are session recipes, not editor-integration material.
This fixes "Codex has the wrong skills" as a side effect: today the Codex
plugin in `firecrawl/skills` points at `./skills/`, which contains only build
skills. After the catalog sync lands, the same repo contains the CLI skills
too, and the plugin manifest is updated to reference the two bundled
categories. Bonus: the skills CLI also reads `.claude-plugin/marketplace.json`,
so plugin manifests in the catalog do double duty for discovery.
Plugin metadata in `firecrawl-workflows` stays as-is (backward compat) but is
no longer promoted.
## Contributor routing rule (goes in every README)
> **CLI skills → PR `firecrawl/cli`. Build/SDK skills → PR `firecrawl`
> (monorepo, `skills/`). Everything else (workflows, reference) → PR
> `firecrawl/skills`. New skills always land in an existing repo — when in
> doubt, `firecrawl/skills` under `skills/workflows/`.**
## Backward-compatibility matrix
| Existing surface | After migration |
|---|---|
| `npx skills add firecrawl/cli --skill firecrawl` | unchanged — cli repo is still the CLI source of truth |
| `npx skills add firecrawl/skills` | still works; now offers the full catalog via interactive picker |
| `npx skills add firecrawl/skills --skill <build-skill>` | unchanged — name-based selection is path-independent |
| `npx skills add firecrawl/firecrawl-workflows` | unchanged — served by the CI-synced mirror |
| `firecrawl init` | still works; installs CLI skills + optional workflows instead of build skills |
| Editor plugin installs | unchanged manifests keep resolving; bundles gain the CLI skills |
| skills.sh counts | no existing counter breaks; build counters continue; cli/workflows old counters keep counting legacy installs |
## Migration action plan
Additive-only; each phase leaves every surface working. Sequencing rule:
**populate and sync the catalog before any README or command points at it.**
### Phase 0 — prep (no user-visible change)
1. Generate deploy keys: cli→catalog, monorepo→catalog, catalog→workflows-mirror.
2. Archive `web-agent` and `opencode-firecrawl` (GitHub archive keeps installs
working); point their READMEs at the catalog first.
### Phase 1 — build the catalog (additive inside `firecrawl/skills`)
3. Restructure: move the 5 build skills to `skills/build/`, the 2 index skills
to `skills/reference/` (safe — counts are name-keyed, `--skill` is name-based).
4. Copy the 16 workflow skills into `skills/workflows/` — this is the new
source of truth for workflows. `firecrawl-workflows` is untouched and still
canonical until Phase 2 flips it.
5. Add sync workflow 1 to `firecrawl/cli` → catalog `skills/cli/` lands via CI.
6. Add the routing rule + per-category mirror notices to the catalog README.
### Phase 2 — flip sources of truth
7. Move build skills' authorship into the `firecrawl` monorepo root `skills/`
(replacing today's pointer-README stubs); add sync workflow 2
(monorepo → catalog `skills/build/`). Mark catalog `skills/build/` as mirror.
8. Add sync workflow 3 (catalog `skills/workflows/` → `firecrawl-workflows`);
convert `firecrawl-workflows` to mirror: banner README, issues off,
repo-lockdown, branch protection. Routing rule in its README points PRs at
the catalog.
9. Freeze direct edits to mirrored dirs (banner + lockdown; social enforcement
is enough given CI overwrites drift on next sync).
### Phase 3 — fix the integrations
10. Update the catalog's Codex/Cursor/Claude plugin manifests to bundle
`skills/cli/` + `skills/build/`. This closes the Codex bug.
11. Revamp `firecrawl init`: CLI skills by default, workflow multi-select,
drop build skills, delegate to `npx skills add firecrawl/skills`.
### Phase 4 — flip promotion
12. Update every README, doc, and marketing surface to promote exactly one
command: `npx skills add firecrawl/skills` (plus `firecrawl init` for CLI
onboarding). Old commands are never mentioned as removed — they simply
stop being promoted.
13. Add the routing rule to `firecrawl/cli` and monorepo READMEs.
### Phase 5 — steady state
14. New skills land per the routing rule; the growth lever (keep shipping,
especially workflows) now has exactly one obvious destination per skill type.
15. Revisit the parked A/B question (granular vs consolidated packaging) once
catalog activation data accumulates — the catalog's category layout makes
adding experimental consolidated bundles additive and cheap.
## Risks & open items
- **Catalog counters start at zero** for CLI and workflow skills. Mitigated by
`firecrawl init` volume flowing to the catalog immediately, and by never
breaking the old paths.
- **Drive-by PRs against mirrored catalog dirs** will happen. CI overwrite +
banner + a maintainer redirect comment is sufficient; repo-lockdown on the
catalog itself is not an option since two of its categories accept PRs.
- **skills.sh stale entries** (~41 shown vs 7 real in this repo) have no
self-service cleanup mechanism today; they stay visible on the org page.
skills/reference/firecrawl-developer-index/SKILL.md
-59
View File
@@ -1,59 +0,0 @@
---
name: firecrawl-developer-index
description: Search issues, merged pull requests, READMEs, and documentation. Use when the question is how a library or API behaves, what an error means, or whether a bug was fixed; prefer this over a general web page.
---
# Firecrawl Developer Index
Answer a developer question from the primary source: the issue where the bug was reported, the merged pull request that fixed it, the README or documentation page that states the contract. A blog post that describes a behaviour is a weaker answer than the passage that defines it, so reach for the index first and the open web second.
There is **no fixed recipe**. Read the question, decide what kind it is, and choose the approach below. A literal error string wants a different move than "how do I do X". Don't run machinery a question doesn't call for.
## The tools, and what each is uniquely good at
- HTTP: **`GET|POST https://api.firecrawl.dev/v2/search/developer`**
MCP: **`firecrawl_developer_search(query, k?, skills?)`**
CLI: **`firecrawl developer <query> [--limit <n>] [--skills-only]`**
Ranked results over the whole index. Each carries `id` (`issue:owner/repo#123`), `url`, and the **matched passages in markdown**, so tables and code blocks survive. The artifact kind is the `id` prefix: `doc:`, `issue:`, `pull_request:`, or `readme:`.
The default first move for a developer question. It is the only surface that returns the passages, which is what lets you answer instead of pointing at a page.
`k` / `--limit` is 1–100 and defaults to 10. `skills="only"` / `--skills-only` restricts the search to agent-skill files.
Keyless; send `Authorization: Bearer $FIRECRAWL_API_KEY` for higher rate limits.
- MCP: **`firecrawl_search(query, categories: ["developer"])`**
CLI: **`firecrawl search <query> --categories developer`**
Developer hits in a `developer` group beside `web`, each with `url`, `title`, `description` (the matched passage), `position`, and `category: "developer"` — web results carry no `category`, so that is the field to key on when merging.
Use this when you are **already** running a web search and want developer sources weighed in the same call. It exposes none of the filters and no passage control.
- MCP: **`firecrawl_scrape(url)` / `firecrawl_search(query)`**
CLI: **`firecrawl scrape <url>` / `firecrawl search <query>`**
General web fetch and search, for what no primary source states: a comparison between two libraries, an outage, a migration write-up, a project with no public repository or indexed docs.
Also the follow-through when a hit is the right page but you need all of it — `scrape` the result's `url`.
## Filters, and what each one costs you
Only the HTTP surface takes these. On `GET`, pass `types=issue,pull_request` or repeat the parameter; on `POST`, pass arrays. All are optional.
- `types` — which of `doc`, `issue`, `pull_request`, `readme` to search. Defaults to all four. Narrowing here is the cheapest way to sharpen a query.
- `repos` (`owner/name`) scopes the repository half, meaning `issue`, `pull_request`, and `readme`; `sources` (documentation source ids, at most 20) scopes the documentation half, meaning `doc`. Passing both **unions** the halves rather than intersecting them. Both echo back in the response with `indexed: true|false` — that is how you tell "not in the index" from "found nothing".
- A filter that cannot match any requested `type` is a `400`, not an empty list: `repos` with no repository type in `types`, or `sources` without `doc`.
- `passages` (1–5, default 1) is the *maximum* passages per result, not a guarantee. Raise it when one page is clearly the right page but the first passage is the wrong part of it.
- `language`, `topic`, `license`, `min_stars`, `max_stars`, `archived`, `fork` describe a **repository**. Most documentation pages in the index have no repository behind them, so no repository fact can admit or exclude one. Send any of these without a `sources` scope and the response holds repository evidence only — `issue`, `pull_request`, `readme`. That is the design, not an index fault: do not retry it and do not report the index broken. To keep documentation, drop the repository filters, or scope the documentation half with `sources` and read the `sources` echo to confirm the id is indexed.
## Match the approach to the question
- **Literal error message or stack-trace string** → search the string itself plus the library name, with `types=["issue","pull_request"]`. Whoever hit it filed it. If nothing matches, strip the volatile parts (paths, line numbers, ids, addresses) and retry — the invariant middle of the message is what is indexed.
- **Conceptual "how do I do X"** → the full question in natural language, all four types. The answer is usually a `doc` or a `readme`; raise `passages` before raising `k`.
- **Known bug** → the issue reports it, the merged pull request *fixes* it, and the fix is what you want. Search `types=["issue","pull_request"]`, then re-query the issue's own terms scoped to its repo with `types=["pull_request"]`. A merged PR's passages tell you what changed and in which direction.
- **API contract** ("what does X return", "is Y required", "what is the default") → `readme` and `doc` are authoritative and a blog post is not. Use `types=["readme","doc"]`. If the contract looks like it moved, follow up with `pull_request` for the change that moved it.
- **Version-specific behaviour** → an issue's opening report describes the broken version; its resolution supersedes it. Raise `passages` to see further into the thread, and read the resolution and the linked pull request before answering. Never answer from an opening report alone.
- **Scoped to one library** → `repos=["owner/name"]` when you know the slug, plus `sources` if you want its docs in the same call. If a scoped search comes back empty, read the echoed `indexed` flag first: `false` means nothing from that repo or source can ever match and no rephrasing will help — drop the scope and search the whole index, or go to the web.
- **Ecosystem-wide** ("which libraries do X", "who else hit this") → no scope. Use `language` / `topic` / `min_stars` to keep to maintained repositories, accepting that this gives up all `doc` results.
- **Agent skills and tooling conventions** → `skills="only"` / `--skills-only`.
- **Comparison, opinion, news, or an unindexed project** → the open web. `firecrawl_search`, then `firecrawl_scrape` whatever deserves a full read. Combining is often right: take the contract from the index and the trade-off from the web.
## Principles
- **Quote the passage, cite the `url`.** The passages are the evidence; hand them over rather than paraphrasing them into a claim the reader can't check. `title` is frequently absent on `doc` results — fall back to `url`.
- **A merge supersedes a report.** When an issue and a pull request disagree, the merged pull request is the current behaviour. Say which one you read.
- **Scope last, not first.** Search the whole index, then narrow with `types`, `repos`, or `sources` once you know what the hits look like. Scoping first hides the result that would have told you where to look.
- **Go to the web when the index has nothing to say.** Trade-offs, ecosystem opinion, and anything about an unindexed project are web questions. Don't force them through the index, and don't dress a general web page up as a primary source.
skills/reference/firecrawl-research-index/SKILL.md
-69
View File
@@ -1,69 +0,0 @@
---
name: firecrawl-research-index
description: Find the papers that answer a research query in Firecrawl's research paper index — a corpus of paper abstracts whose largest share is biomedical and life-science literature (PubMed, bioRxiv, medRxiv), alongside arXiv preprints in CS, physics, and math — using semantic search, semantic and structural expansion, and in-body verification. Use this skill for literature-finding and paper-retrieval tasks of any kind, including clinical, biomedical, drug, gene, disease, and other life-science questions, whether the answer is a single paper or a full multi-paper set. The index is reached only through the `firecrawl_research_*` MCP tools or the `firecrawl research` CLI subcommands. Calling `firecrawl_search` with its `categories` option set to `["research"]` is a different feature — it filters ordinary web search to research-affiliated websites (the list includes PubMed, bioRxiv, medRxiv, arXiv, and publisher sites) and returns page results from them, without querying the paper records in this index.
---
# Firecrawl Research Index
Find the research papers that answer a research query. Some questions have a single answer; many have several — and when in doubt, lean toward returning the fuller relevant set (most relevant first) rather than narrowing to one. A reader is better served seeing the neighboring methods and papers than having them silently dropped.
## What is in the index
Paper abstracts, with full text reachable per paper. The largest share of the corpus is **biomedical and life-science** literature — **PubMed** journal articles plus **bioRxiv** and **medRxiv** preprints — so clinical, drug, gene, disease, epidemiology, and public-health questions are in scope. **arXiv** preprints cover computer science, physics, and mathematics. Coverage outside those sources is thinner: a paper that exists only behind a publisher paywall or in a niche venue may not be indexed, and the general web tools below are the fallback when it isn't.
There is **no fixed recipe**. Read the query, decide what kind it is, and choose the approach below. Some queries need a single search; others need heavy sturctural/semantic expansion. Don't run machinery a query doesn't call for.
## The tools, and what each is uniquely good at
- MCP: **`firecrawl_research_search_papers(query, k?)`**
CLI: **`firecrawl research search-papers <query> [--k <number>]`**
Semantic (HyDE) search over **abstracts**. The natural first move for almost any query.
If results look thin or all-alike, re-run with a different framing (sibling domain, rival method, dataset/benchmark name) rather than giving up.
- MCP: **`firecrawl_research_related_papers(seed_ids, intent, mode?, k?)`**
CLI: **`firecrawl research related-papers <seedIds...> --intent <intent> [--mode <similar|citers|references>] [--k <number>]`**
Semantic and structural expansion, ranked to your `intent`.
This reaches papers semantic search *cannot*, and it's how you turn one good hit into the rest of a set.
`mode=similar` → niche siblings; `citers` → who uses/builds on the seeds; `references` → what they build on / compare against.
- MCP: **`firecrawl_research_inspect_paper(id)`**
CLI: **`firecrawl research inspect-paper <id>`**
Canonical metadata for **one** paper: title, abstract, authors, categories, source ids, and dates.
Use it after `search_papers` or `related_papers` when you need the complete citation/metadata for a candidate, or when you have an id from elsewhere and need to confirm what paper it resolves to.
This does **not** read the paper body; use `read_paper` for specific full-text questions.
- MCP: **`firecrawl_research_read_paper(id, question)`**
CLI: **`firecrawl research read-paper <id> --question <question>`**
In-body passages of **one** paper, to verify a load-bearing constraint (a method actually used, a score actually reported, an affiliation, what a paper compares to).
Use it to settle a specific doubt, not on everything.
- MCP: **`firecrawl_search(query, categories: ["research"])`**
CLI: **`firecrawl search <query> --categories research`**
**Not this index.** This is a *website* filter: it restricts a normal web search to a short list of research-affiliated domains — the list does include `pubmed.ncbi.nlm.nih.gov`, `biorxiv.org`, `medrxiv.org`, and `arxiv.org` alongside publisher sites — and returns page results in a `research` group beside `web`, each with `url`, `title`, `description` (the matched passage), `position`, and `category: "research"` — web results carry no `category`, so that is the field to key on when merging.
So it reaches those sites' **web pages**; what it does not do is query their **paper records** in this index — no semantic search over abstracts, no citation-graph or related-paper expansion, no canonical paper metadata, and no in-body passages. The results are ordinary web results.
Use it when you are **already** running a web search and want those sites weighed in the same call. For anything that is actually a paper-finding task, use `firecrawl_research_search_papers` and its siblings above.
- MCP: **`firecrawl_search(query)` / `firecrawl_scrape(url)`**
CLI: **`firecrawl search <query>` / `firecrawl scrape <url>`**
General **web** search and page fetch, for facts that don't live in paper abstracts: benchmark **leaderboards**, rankings, "who scores best / is largest / is most used."
Find the ranking on the web, then map the top entries back to papers with `search_papers`.
Reach for these only when the corpus can't answer the question on its own.
## Match the approach to the query
- **Single *named* paper** ("the Qwen3 report") → one `search_papers`, done. This is the only case that truly wants exactly one paper.
- **Paper by description / by method or technique** ("the paper that introduced X", "training-free N-gram detection of AI text") → find the best match, then assume there's a *family*: expand with `related_papers` and **include the closely-related methods/papers too**. Even when one paper is the exact literal match, surface and keep its neighbors — don't narrow to the single best hit and reason the rest out. Only treat it as one-answer if the query names a specific paper.
- **Enumeration / method-family** ("papers that do X", "alternatives to Adam", "benchmarks for Y") → the answer is a *set*, and this is where `related_papers` earns its keep: expand several strong anchors with `mode=similar`, re-seed from new strong hits. One search is never enough here.
- **Exhibiting** ("papers that *use* / exhibit property P") → the relevant papers apply P but their abstracts may not describe it. Go from P's defining paper outward via `citers`/`references`, and use `read_paper` to confirm a candidate actually uses P.
- **Superlative / leaderboard** ("best on benchmark X", "largest", "most popular") → the ranking lives on **leaderboards / the web**, not in any single abstract. Use `firecrawl_search` / `firecrawl_scrape` to find the benchmark's leaderboard or rankings, read off the top models/papers, then `search_papers` each to get its paper. As a fallback, search the benchmark and `read_paper` candidates for reported numbers. The hardest kind — cast wide.
- **Org / author filtered** ("from \<org\>", "by \<author\>") → topical match isn't enough; verify the affiliation/authorship (metadata or `read_paper`) before keeping a paper.
- **Compare-against** ("what does paper X benchmark against / build on") → the answer is *inside* paper X: `read_paper(X, ...)` or `related_papers([X], ..., mode="references")`.
## Principles
- **Two different features share the word "research."** The paper index is `firecrawl_research_*` / `firecrawl research`. The `categories: ["research"]` option on `firecrawl_search` is a website filter — it does point web search at PubMed, bioRxiv, medRxiv, arXiv, and publisher sites, but what comes back is their web pages, not paper records. If a task is about finding papers, the tools in this skill are the ones that read the corpus; reaching for `categories: ["research"]` will quietly answer a different question.
- **Query shape and subject field are separate.** A clinical-trial question and a machine-learning question take the same shapes above; what differs is only which source the hits come from. Don't send a biomedical or life-science query to the open web on the assumption the corpus is arXiv-only — PubMed, bioRxiv, and medRxiv are the largest part of what `search_papers` reads.
- **When in doubt, include.** For any topic / method / comparison question, return the relevant *family*, not just the single best match — err toward keeping a plausibly-relevant paper rather than dropping it. The neighboring methods are part of a good answer; don't reason close work out just because one paper is the most exact match.
- **Follow the literature, and keep what you find.** The seminal source, the competing methods, the close neighbors are usually a hop away — use `related_papers`, and *include* them, not just the first hit. Stopping at one good result is the most common way to leave the reader with half an answer.
- **Verify to exclude, not to gatekeep.** Use `read_paper` to rule a paper *out* when a hard constraint clearly fails (wrong org/author, doesn't actually report the score). When a paper is plausibly relevant, lean toward keeping it rather than demanding proof.
- **Only drop the clearly off-topic.** Don't pad with papers you're confident are unrelated — but that's a high bar; most plausibly-relevant work should make the cut.
skills/workflows/README.md
+4
View File
@@ -0,0 +1,4 @@
# Workflow skills (mirror)
Mirror of [`firecrawl/firecrawl-workflows`](https://github.com/firecrawl/firecrawl-workflows) — do not edit here.
PR changes against `firecrawl/firecrawl-workflows`; CI overwrites this directory on every sync.