skills/kostja94__marketing-skills
1
0
Fork 0
mirror of https://github.com/kostja94/marketing-skills.git synced 2026-09-19 00:51:29 +08:00
Code Issues Packages Projects Releases Wiki Activity

chore: robots path-format rules and Claude-User crawler update

Browse Source 
Add Disallow/Allow vs Sitemap path format table, wildcard precedence, Claude-User entry, and anthropic-ai deprecation note.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
zyjst
2026-06-09 13:13:06 +08:00
parent b6f1202116
commit 70987bad4e
2 changed files with 21 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CHANGELOG.md
+6
View File
@@ -17,6 +17,12 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
---
## [2026-06-08] v2.3.22
### Changed
- **robots** (v1.2.0) - Path format rules (Disallow/Allow vs Sitemap), wildcard and precedence notes; Claude-User crawler entry; deprecated anthropic-ai user-agent.
## [2026-06-08] v2.3.21
### Changed
skills/seo/technical/robots/SKILL.md
+15
View File
@@ -72,6 +72,19 @@ Identify:
| `Sitemap:` | Declare sitemap absolute URL | `Sitemap: https://example.com/sitemap.xml` |
| `Clean-param:` | Strip query params (Yandex) | See below |
### Path Format Rules (Critical)
Different directives use different path formats — a common source of errors:
| Directive | Format | Correct | Wrong |
|-----------|--------|---------|-------|
| Disallow / Allow | **Root-relative path only** (starts with `/`) | `Disallow: /admin/` | `Disallow: https://example.com/admin/` |
| Sitemap | **Absolute URL only** | `Sitemap: https://example.com/sitemap.xml` | `Sitemap: /sitemap.xml` |
**Wildcards**: `*` matches any character sequence (`Disallow: /tmp/*`). `$` marks exact URL ending (`Allow: /news/.html$`).
**Priority**: More specific paths take precedence. `Allow: /shop/shoes/` overrides `Disallow: /shop/`. Path matching is case-sensitive: `Disallow: /PDF/` does not match `/pdf/`.
### Critical: Do Not Block
| Do not block | Reason |
@@ -92,7 +105,9 @@ robots.txt is effective for all measured AI crawlers. Set rules per user-agent;
| **GPTBot** | OpenAI training | Disallow | Respects robots.txt; shares crawl data with OAI-SearchBot if both allowed |
| **ChatGPT-User** | User-initiated browsing | N/A | **No longer respects robots.txt** (Dec 2025); use server-side controls instead |
| **Claude-SearchBot** | Claude search | Allow | Respects robots.txt |
| **Claude-User** | Anthropic user-initiated browsing | Allow | Respects robots.txt (unlike ChatGPT-User) |
| **ClaudeBot** | Anthropic training | Disallow | Respects robots.txt |
> **Deprecated**: ~~anthropic-ai~~ — retired by Anthropic, replaced by ClaudeBot / Claude-User / Claude-SearchBot. References to `anthropic-ai` in robots.txt have no effect.
| **PerplexityBot** | Perplexity search | Allow | Respects robots.txt |
| **Google-Extended** | Gemini training | Disallow | Respects robots.txt |
| **CCBot** | Common Crawl (LLM training) | Disallow | Respects robots.txt |