From e53786037277b0dd30659f5c87868a834493aac3 Mon Sep 17 00:00:00 2001 From: Musab Gultekin Date: Fri, 18 Sep 2026 08:57:18 +0300 Subject: [PATCH] Add Serping API as an optional SERP provider (serp-analyzer, keyword-research, competitor-analysis) (#14) Co-authored-by: Musab Gultekin <3749407+musabgultekin@users.noreply.github.com> Co-authored-by: Quanlai Li Co-authored-by: Claude Fable 5 --- README.md | 3 ++ cli/README.md | 3 +- skills/competitor-analysis/SKILL.md | 21 ++++++++++++- skills/keyword-research/SKILL.md | 44 +++++++++++++++++++++++++++ skills/serp-analyzer/SKILL.md | 46 +++++++++++++++++++++++++++++ 5 files changed, 115 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6b19377..d0f01c9 100644 --- a/README.md +++ b/README.md @@ -229,6 +229,9 @@ SERPAPI_API_KEY=your_key_here DATAFORSEO_LOGIN=your_login DATAFORSEO_PASSWORD=your_password +# Serping API (serp-analyzer, keyword-research, competitor-analysis) +SERPINGAPI_API_KEY=your_key_here + # Glasser (glasser-research; interactive users can run `glasser login` instead) GLASSER_API_KEY=your_key_here diff --git a/cli/README.md b/cli/README.md index 8d6eb90..e5dd881 100644 --- a/cli/README.md +++ b/cli/README.md @@ -36,7 +36,7 @@ After installing, open Claude Code and use any skill as a slash command: | Skill | Description | |-------|-------------| | `seo-audit` | Full technical + on-page SEO audit with actionable fixes | -| `keyword-research` | Keyword research via SemRush, DataForSEO, SerpAPI | +| `keyword-research` | Keyword research via SemRush, DataForSEO, SerpAPI, Serping API | | `serp-analyzer` | Analyze SERP results and ranking factors | | `backlink-audit` | Audit backlink profile via SemRush or Ahrefs | | `schema-markup` | Generate and validate Schema.org structured data | @@ -108,6 +108,7 @@ RESEND_API_KEY= # Send emails (email-sequence, email-subject-lines) SEMRUSH_API_KEY= # SEO data (keyword-research, backlink-audit, competitor-analysis) AHREFS_API_KEY= # Backlink data (backlink-audit) SERPAPI_API_KEY= # SERP data (serp-analyzer, keyword-research) +SERPINGAPI_API_KEY= # SERP data, free tier (serp-analyzer, keyword-research, competitor-analysis) BRANDDEV_API_KEY= # Brand monitoring (brand-monitor) UNSPLASH_CLIENT_ID= # Stock images (write-blog, social-content) GOOGLE_CLIENT_ID= # Google APIs (analytics, search-console, ads) diff --git a/skills/competitor-analysis/SKILL.md b/skills/competitor-analysis/SKILL.md index 7bced2c..008f657 100644 --- a/skills/competitor-analysis/SKILL.md +++ b/skills/competitor-analysis/SKILL.md @@ -13,6 +13,7 @@ The following API keys enable richer data collection. All are optional -- the fr - `SEMRUSH_API_KEY` - Domain overview, organic keywords, competitor discovery, traffic estimates - `SERPAPI_API_KEY` - Real-time SERP competitive analysis, ad copy extraction +- `SERPINGAPI_API_KEY` - Real-time organic SERP positions and SERP features (free tier: 1,000 searches/month, no card) - `SCRAPINGBEE_API_KEY` - Scrape competitor pages that block direct fetching ### SemRush API (if SEMRUSH_API_KEY available) @@ -64,6 +65,24 @@ curl -s "https://serpapi.com/search.json?q={commercial_keyword}&api_key=${SERPAP ``` The response `ads` array contains: `position`, `title`, `link`, `displayed_link`, `tracking_link`, `description`, `sitelinks`. This reveals competitor ad copy, landing pages, and messaging. +### Serping API (if SERPINGAPI_API_KEY available) + +**SERP Competitive Analysis** - Same use as SerpAPI above, for organic results and SERP features (web search only — no `ads` or `shopping_results`). Free tier is 1,000 searches/month with no card, so it works when no SerpAPI key is configured: +```bash +# Real-time SERP for competitive keywords +curl -s -X POST "https://api.serpingapi.com/v1/search" \ + -H "X-API-Key: ${SERPINGAPI_API_KEY}" \ + -H "Content-Type: application/json" \ + -d '{"q": "{keyword}", "gl": "us", "hl": "en", "num": 20}' +``` +Use this to: +- Identify which competitors dominate organic results for target keywords (parse `organic`: `position`, `title`, `link`, `snippet`) +- Discover related competitor keywords from `relatedSearches` (`.query`) +- See competitor presence in SERP features: `knowledgeGraph`, `answerBox`, `peopleAlsoAsk` +- Add `"tbs": "qdr:m"` to see who ranked in the last month, or `"location"` for a local SERP + +Errors are `{"error": {"code", "message"}}`; on `429 quota_exceeded` tell the user the monthly quota is spent and fall back to WebSearch. + ### ScrapingBee (if SCRAPINGBEE_API_KEY available) Use ScrapingBee to scrape competitor pages that block direct fetching via WebFetch (e.g., JavaScript-heavy pages, bot-protected sites, pricing pages): @@ -97,7 +116,7 @@ Google "[category]" (ads + organic top 10), G2/Capterra "Compare" pages, Reddit/ ## Step 3: SEO Analysis -If `SEMRUSH_API_KEY` is available, use the Domain Overview and Organic Keywords endpoints (see Optional API Integrations above) to populate the profile below with real data. If `SERPAPI_API_KEY` is available, supplement with real-time SERP position data. Otherwise, use WebSearch and public tools to estimate. +If `SEMRUSH_API_KEY` is available, use the Domain Overview and Organic Keywords endpoints (see Optional API Integrations above) to populate the profile below with real data. If `SERPAPI_API_KEY` or `SERPINGAPI_API_KEY` is available, supplement with real-time SERP position data. Otherwise, use WebSearch and public tools to estimate. For each competitor: diff --git a/skills/keyword-research/SKILL.md b/skills/keyword-research/SKILL.md index 535819c..20c1a72 100644 --- a/skills/keyword-research/SKILL.md +++ b/skills/keyword-research/SKILL.md @@ -351,3 +351,47 @@ curl -s "https://serpapi.com/search.json?q={keyword}&api_key=${SERPAPI_API_KEY}& - `device=desktop` or `device=mobile` - Desktop vs mobile SERPs (mobile may show different PAA questions) **Note:** SerpAPI charges per search. Use it strategically for your highest-priority keywords rather than for bulk research. Pair it with SemRush for volume data and DataForSEO for bulk lookups. + +### Serping API (People Also Ask & Related Searches — free tier) + +If `SERPINGAPI_API_KEY` is available, use Serping API (https://serpingapi.com) for the same "People Also Ask" and related-searches data from live Google SERPs. The free Reader plan includes 1,000 searches per month with no credit card, so it is a good default when no SerpAPI key is set. Paid plans are flat monthly (from $49 for 20,000 searches). + +**Search Endpoint:** + +```bash +# Get SERP data including People Also Ask and related searches +curl -s -X POST "https://api.serpingapi.com/v1/search" \ + -H "X-API-Key: ${SERPINGAPI_API_KEY}" \ + -H "Content-Type: application/json" \ + -d '{"q": "{keyword}", "gl": "us", "hl": "en", "num": 10}' +``` + +**Parsing People Also Ask:** + +```bash +# Extract People Also Ask questions +curl -s -X POST "https://api.serpingapi.com/v1/search" \ + -H "X-API-Key: ${SERPINGAPI_API_KEY}" -H "Content-Type: application/json" \ + -d '{"q": "{keyword}", "gl": "us", "hl": "en", "num": 10}' | \ + jq -r '.peopleAlsoAsk[] | {question, snippet, link, title}' +``` + +Key response sections (present only when Google shows them): +- **`peopleAlsoAsk`** - Array of "People Also Ask" questions with snippets and source URLs +- **`relatedSearches`** - Array of `{ "query": ... }` objects that Google suggests +- **`organic`** - Organic results with `position`, `title`, `link`, `snippet` +- **`answerBox`** / **`knowledgeGraph`** - Featured snippet and knowledge panel, when present + +**Parsing Related Searches:** + +```bash +# Extract related searches for content ideation +curl -s -X POST "https://api.serpingapi.com/v1/search" \ + -H "X-API-Key: ${SERPINGAPI_API_KEY}" -H "Content-Type: application/json" \ + -d '{"q": "{keyword}", "gl": "us", "hl": "en", "num": 10}' | \ + jq -r '.relatedSearches[] | .query' +``` + +Use the data exactly as described for SerpAPI above (FAQ headings, gap discovery, keyword expansion, intent validation, clustering). + +**Additional Serping API parameters:** `location` (e.g. `"Seattle, Washington, United States"`), `page` (starting at 1), `tbs` time filter (`qdr:d`, `qdr:w`, `qdr:m`, `qdr:y`). Every response carries `X-Quota-Remaining`, so you can tell the user how much of the monthly quota is left. Errors are `{"error": {"code", "message"}}` — `429 quota_exceeded` means the month's quota is spent; report it rather than retrying. diff --git a/skills/serp-analyzer/SKILL.md b/skills/serp-analyzer/SKILL.md index 657d372..9a4a8e9 100644 --- a/skills/serp-analyzer/SKILL.md +++ b/skills/serp-analyzer/SKILL.md @@ -13,6 +13,7 @@ Optional API keys for enriched data (the skill can work without any of them usin - `SEMRUSH_API_KEY` - for keyword and organic results data - `SERPAPI_API_KEY` - for real-time Google SERP data including SERP features - `DATAFORSEO_LOGIN` and `DATAFORSEO_PASSWORD` - for advanced SERP data +- `SERPINGAPI_API_KEY` - for real-time Google SERP data (free tier: 1,000 searches/month, no card) ## Analysis Process @@ -95,6 +96,51 @@ The response provides: Location codes: 2840 = US, 2826 = UK, 2124 = Canada, 2036 = Australia. Change `location_code` for geo-targeted analysis. +**Method F: Serping API (if SERPINGAPI_API_KEY available)** + +Real-time Google SERP data as Serper-style JSON from a single endpoint. The free Reader plan includes 1,000 searches per month with no credit card, so this works for a first run without a deposit. Get a key at https://serpingapi.com and set `SERPINGAPI_API_KEY`. + +```bash +# Real-time Google SERP data via Serping API (POST, JSON body) +curl -s -X POST "https://api.serpingapi.com/v1/search" \ + -H "X-API-Key: ${SERPINGAPI_API_KEY}" \ + -H "Content-Type: application/json" \ + -d '{"q": "{keyword}", "gl": "us", "hl": "en", "num": 20}' +``` + +The JSON response includes (sections appear only when Google returns them): +- `organic` - Array of organic listings with `position`, `title`, `link`, `snippet` (sometimes `sitelinks`, `date`, `rating`) +- `peopleAlsoAsk` - People Also Ask questions with `question`, `snippet`, `title`, `link` +- `answerBox` - Featured snippet / direct answer +- `knowledgeGraph` - Knowledge panel data with `title`, `type`, `description`, `attributes` +- `relatedSearches` - Related search queries as `{ "query": ... }` +- `searchParameters` - Echo of the parameters the search ran with + +Parse example: +```bash +# Extract organic results +curl -s -X POST "https://api.serpingapi.com/v1/search" \ + -H "X-API-Key: ${SERPINGAPI_API_KEY}" -H "Content-Type: application/json" \ + -d '{"q": "{keyword}", "gl": "us", "hl": "en", "num": 20}' | \ + jq '.organic[] | {position, title, link, snippet}' + +# Extract People Also Ask questions +curl -s -X POST "https://api.serpingapi.com/v1/search" \ + -H "X-API-Key: ${SERPINGAPI_API_KEY}" -H "Content-Type: application/json" \ + -d '{"q": "{keyword}", "gl": "us", "hl": "en", "num": 20}' | \ + jq '.peopleAlsoAsk[] | {question, snippet}' + +# Check for featured snippet / knowledge graph +curl -s -X POST "https://api.serpingapi.com/v1/search" \ + -H "X-API-Key: ${SERPINGAPI_API_KEY}" -H "Content-Type: application/json" \ + -d '{"q": "{keyword}", "gl": "us", "hl": "en", "num": 20}' | \ + jq '{answerBox, knowledgeGraph: (.knowledgeGraph | {title, type, description})}' +``` + +Optional parameters: `location` (e.g. `"Seattle, Washington, United States"`), `page` (starting at 1), `tbs` for a time filter (`qdr:d` day, `qdr:w` week, `qdr:m` month, `qdr:y` year). Web search only — no ads, shopping, or local pack sections. + +Errors come back as `{"error": {"code": "...", "message": "..."}}`: `401 invalid_api_key` means the key is wrong or revoked; `429 quota_exceeded` means the monthly quota is used up (resets on the 1st, UTC). In either case tell the user the specific error and fall back to Method B. + ### Step 2: Map SERP Features Document every SERP feature present for this keyword: