civitai__civitai/docs/seo-sitemap-migration.md

SEO Sitemap Migration — Per-Host Sitemaps + robots.txt

This doc covers the move from next-sitemap (build-time, single-host) to fully SSR sitemaps and robots.txt that emit per-color URLs for civitai.com / civitai.red. It includes the rollout checklist and a follow-up proposal for historical (per-month) sitemap partitioning.

What changed

• /sitemap.xml — was a static file generated by next-sitemap post-build, pinned to NEXTAUTH_URL (i.e. next.civitai.com). Now an SSR route at src/pages/sitemap.xml/index.tsx that resolves the request's color domain and emits sub-sitemaps with the matching base URL.
• /sitemap-models.xml, /sitemap-articles.xml — were already SSR but used getBaseUrl() (which falls back to NEXTAUTH_URL) and had no per-color content filter. Now resolve color from the request, emit per-host URLs, and filter content by canonical-host rules that mirror Gated. See "Sitemap inclusion rules" below for the full predicates — the model query additionally factors in Model.nsfw (force-NSFW boolean override).
• /sitemap-tools.xml — removed entirely. Tools have no NSFW level so per-color filtering is N/A; the tools landing page (/tools) is still in the green allowlist of sitemap-pages.xml so the section is discoverable.
• /sitemap-pages.xml — new. Curated list of high-value static routes (/models, /articles, /safety, etc.). Replaces next-sitemap's auto-discovered /sitemap-0.xml, which included noisy routes like /login, /dev/onboarding, /region-blocked, and the sitemap routes themselves.
• /robots.txt — was generated by next-sitemap at build time with a single hardcoded host. Now SSR at src/pages/robots.txt/index.tsx with the same disallow rules plus per-host Sitemap: and Host: lines.
• next-sitemap package and postbuild script removed. Static public/sitemap*.xml and public/robots.txt are no longer generated.

Helpers live at src/server/utils/sitemap.ts (respondWithSitemap, respondWithSitemapIndex, respondWithText). Per-color browsing level filtering uses publicBrowsingLevelsFlag and sfwBrowsingLevelsFlag in src/shared/constants/browsingLevel.constants.ts.

Sitemap inclusion rules

The sitemap filters mirror what Gated decides on the detail page so the sitemap and the per-page noindex signal don't disagree.

nsfwLevel is bitwise. A model can have multiple bits set (e.g. nsfwLevel=7 = PG | PG-13 | R), which means the model has produced outputs spanning those levels. Gated decides canonical-host membership using hasSafeBrowsingLevel(level) — true when any safe bit (PG or PG-13) is set. So a multi-level model is canonical on green if it has any safe bit, even if it also has NSFW bits.

The sitemap rules below match that semantic exactly.

Models

Model.nsfw (boolean) is treated as a force-NSFW override on top of nsfwLevel — the same way Gated treats it on the model detail page. This means a model with nsfw=true and a PG nsfwLevel is canonically NSFW and belongs only on the red sitemap, not the green one.

Color	Inclusion rule
green	nsfw = false AND (nsfwLevel & publicBrowsingLevelsFlag) != 0
blue / red	nsfwLevel != 0 AND (nsfw = true OR (nsfwLevel & sfwBrowsingLevelsFlag) = 0)

A model with nsfwLevel=7 (PG | PG-13 | R) and nsfw=false is included only on green, even though it has the R bit — the PG bit makes green canonical.

PG-13-only (nsfwLevel=2) without nsfw=true appears in neither sitemap. That matches the indexability decision: PG-13 requires login on green and isn't the canonical on red.

Articles

Article.nsfw exists in the schema but is not passed to Gated, so the boolean has no effect on article indexability decisions. The article sitemap filters on nsfwLevel only, with the same hasSafeBrowsingLevel semantic:

Color	Inclusion rule
green	status = 'Published' AND (nsfwLevel & publicBrowsingLevelsFlag) != 0
blue / red	status = 'Published' AND nsfwLevel != 0 AND (nsfwLevel & sfwBrowsingLevelsFlag) = 0

Keeping Article.nsfw out of the selector also means article webhook payloads and the Meilisearch article index don't gain a new field — no downstream side effect from this change for articles.

Pre-deploy checklist

1. Lockfile sync. next-sitemap was removed from dependencies in package.json. Run pnpm install and commit the updated pnpm-lock.yaml so CI builds against the same dep tree.
2. Cloudflare audit. Previously, robots.txt was served by a Cloudflare Worker / Page Rule. If that's still active on civitai.com or civitai.red, it will intercept /robots.txt before reaching Next.js and the new SSR route never runs. Disable / remove the Cloudflare-side serving as part of this deploy. (Suggested check: search the CF dashboard for robots.txt rules and any Workers attached to a route matching */robots.txt.)

Deploy steps

1. Ship the change.

2. Purge Cloudflare cache for the following paths on both civitai.com and civitai.red zones:

   • /sitemap.xml
   • /sitemap-pages.xml
   • /sitemap-models.xml
   • /sitemap-articles.xml
   • /sitemap-tools.xml (was previously generated; now 404 — purge so the stale cached copy stops being served)
   • /robots.txt

   Without a purge, CF will keep serving the previously-cached static next-sitemap output for hours.

Post-deploy verification

Hit each URL on each host and eyeball the output:

URL	Expectation
https://civitai.com/sitemap.xml	Index lists 3 sub-sitemaps (pages, models, articles) with civitai.com URLs
https://civitai.red/sitemap.xml	Index lists 3 sub-sitemaps with civitai.red URLs
https://civitai.com/sitemap-models.xml	Models with nsfw=false AND PG bit set (incl. multi-level like PG|PG-13|R), all URLs on civitai.com
https://civitai.red/sitemap-models.xml	Models with nsfw=true OR no safe bits set (R-only, X-only, R|X, etc.), all URLs on civitai.red. Multi-level with any safe bit must NOT appear here
https://civitai.com/sitemap-articles.xml	Articles with PG bit set (any combo) on civitai.com
https://civitai.red/sitemap-articles.xml	Articles with no safe bits set on civitai.red
https://civitai.com/robots.txt	Full disallow list, Host: https://civitai.com, all Sitemap: lines on civitai.com
https://civitai.red/robots.txt	Same but civitai.red
https://civitai.com/sitemap-tools.xml	404 (route removed)
https://civitai.red/sitemap-tools.xml	404 (route removed)

No URL should contain next.civitai.com. If any does, CF cache wasn't fully purged or a Cloudflare rule is still intercepting the path.

Google Search Console

Old GSC submissions almost certainly point at next.civitai.com/sitemap.xml. For each verified property:

1. Remove stale next.civitai.com sitemap submissions.
2. Submit https://civitai.com/sitemap.xml to the civitai.com property.
3. Submit https://civitai.red/sitemap.xml to the civitai.red property.
4. Watch the "Sitemaps" tab over the next 1–2 weeks for crawl errors. Most common cause of errors post-migration is missing per-color URL handling on a downstream layer (e.g. a model page that 404s on civitai.red but is in the red sitemap because it has an R+ image).

Deferred follow-ups

These are intentionally out of scope for the initial migration; tackle in separate PRs if/when the metrics or performance pressure justifies them.

1. Edge caching for SSR sitemap/robots routes

Each SSR sitemap is currently fetched from origin on every request. CPU cost is negligible, but high-volume crawlers (Bing, Yandex, AI scrapers) can issue hundreds of fetches per day per host. Add Cache-Control: public, s-maxage=3600 to the five SSR routes via ctx.res.setHeader('Cache-Control', ...). Cloudflare will cache at the edge and origin sees a fraction of the traffic.

Suggested TTLs:

• /sitemap.xml (index) — 1h. Small, content-driven only by which sub-sitemaps exist, which changes rarely.
• /sitemap-pages.xml — 24h. Curated list, hand-edited; updates ship with code.
• /sitemap-models.xml, /sitemap-articles.xml — 1h. New content needs to be discoverable reasonably quickly.
• /robots.txt — 24h. Almost never changes.

2. EXPLAIN ANALYZE on the model + article queries

The new model and article queries use bitwise predicates plus m.nsfw = true / nsfwLevel != 0 filters and split per color (see sitemap-models.xml and sitemap-articles.xml). Both are bounded by LIMIT 1000 and ordered by indexed columns (thumbsUpCount/downloadCount for models, publishedAt for articles), so the predicate change should be near-free — but worth one EXPLAIN run on prod post-deploy to confirm.

3. Article sort change

The article sitemap previously used getArticles({ sort: MostBookmarks }). The new red rule ("no safe bits set") can't be expressed through getArticles's inclusion-style browsingLevel param, so the route was rewritten as raw SQL — losing the metric-based sort along the way. It now sorts ORDER BY publishedAt DESC, which is more standard for sitemaps anyway (gives Google fresh content first). If you want most-bookmarked articles surfaced specifically, that's a refactor to either extend getArticles with an excludeBrowsingLevel param or to join ArticleStat in the raw query.

4. Coverage gaps on red

The red sitemap excludes any content with a safe bit set (PG or PG-13). If business decides red should also surface PG content (e.g. for users who land directly on civitai.red), the cleanest change is to drop the nsfwLevel != 0 AND (nsfwLevel & sfwBrowsingLevelsFlag) = 0 clause from the red branch of sqlByColor.nsfw in sitemap-models.xml and the matching clause in sitemap-articles.xml. Note this would also require dropping the Gated default-deindex on civitai.red for SFW content (otherwise sitemap and noindex would contradict).

Future enhancement: historical / monthly sitemaps

Right now sitemap-models.xml and sitemap-articles.xml cap at 1000 entries each (LIMIT 1000 on the underlying query) sorted by popularity. That means the long tail of older content never appears in any sitemap and only gets indexed via internal linking. For a content site of Civitai's size, that's a meaningful coverage gap.

The standard fix is to partition each content sitemap by time, with the sitemap index referencing every period's sub-sitemap. Google reads the index, crawls each sub-sitemap, and tracks the <lastmod> per sub-sitemap so it can skip re-fetching periods that haven't changed.

Proposed structure

/sitemap.xml                           ← index, lists every period below
  /sitemap-models/2024-01.xml          ← models with publishedAt in 2024-01
  /sitemap-models/2024-02.xml
  …
  /sitemap-models/2026-05.xml          ← current month, mutates as content publishes
  /sitemap-articles/2024-01.xml
  /sitemap-articles/2024-02.xml
  …
  /sitemap-pages.xml                   ← unchanged, no time dimension

Sitemap protocol caps each sub-sitemap at 50,000 URLs / 50MB. Civitai's per-month publish rate fits comfortably under that. The sitemap index can hold up to 50,000 sub-sitemaps, so even decades of monthly partitioning is fine.

Routing in Next.js

A dynamic route pages/sitemap-models/[period].xml/index.tsx matches paths like /sitemap-models/2024-01.xml. Inside, parse period as YYYY-MM, validate, and run a date-range query that uses the same canonical-host predicates as the all-time sitemap:

-- green
SELECT id, name, COALESCE("lastVersionAt", "publishedAt") AS "updatedAt"
FROM "Model"
WHERE status = 'Published'
  AND nsfw = false
  AND ("nsfwLevel" & $1) != 0       -- $1 = publicBrowsingLevelsFlag
  AND "publishedAt" >= $2            -- start of period
  AND "publishedAt" <  $3            -- start of next period
ORDER BY id
LIMIT 50000;

-- red / blue
SELECT id, name, COALESCE("lastVersionAt", "publishedAt") AS "updatedAt"
FROM "Model"
WHERE status = 'Published'
  AND "nsfwLevel" != 0
  AND (nsfw = true OR ("nsfwLevel" & $1) = 0)  -- $1 = sfwBrowsingLevelsFlag
  AND "publishedAt" >= $2
  AND "publishedAt" <  $3
ORDER BY id
LIMIT 50000;

The publishedAt-bounded query is selective enough that it should run fast without a dedicated index, but if perf becomes an issue, a partial index on (publishedAt) WHERE status = 'Published' would help.

Computing the index

The sitemap.xml index needs to know which periods exist. Options:

• Hardcoded start date. Pick the launch month (e.g. 2022-11) and emit every month from there to today. Simple and zero-cost. New months are implicit (nothing to update each month).
• Query distinct months. SELECT DISTINCT date_trunc('month', "publishedAt") FROM "Model" WHERE status = 'Published'. More accurate but adds DB load to the index endpoint. With caching this is fine.

Recommend the hardcoded-start approach. The index is small (one entry per month × N content types), no DB query needed, and missing months that have zero content just emit empty sitemaps (valid per spec, harmless).

Caching strategy

• Past months are effectively immutable. Once a month closes, its content set rarely changes (only edits to existing items move the per-entry lastmod). Cache aggressively: Cache-Control: public, s-maxage=604800 (1 week).
• Current month is mutable. New content publishes throughout the month. Short TTL: Cache-Control: public, s-maxage=900 (15 min).
• Index. Mostly stable. s-maxage=3600 (1 hour).

The route handler can decide which TTL to apply by comparing the requested period to the current YYYY-MM.

Per-color content split still applies

Each monthly route emits per-color URLs and applies the same canonical-host rules from "Sitemap inclusion rules" above. So civitai.com/sitemap-models/2024-01.xml lists models published in Jan 2024 that are canonical on green (PG bit set, not force-NSFW) with civitai.com URLs; civitai.red/sitemap-models/2024-01.xml lists models from the same period that are canonical on red (no safe bits, or force-NSFW) with civitai.red URLs.

Migration plan (if/when this is built)

1. Implement pages/sitemap-models/[period].xml/index.tsx and matching article route, parallel to the existing sitemap-models.xml.
2. Update sitemap.xml index to emit monthly entries instead of (or in addition to) the all-time sub-sitemaps.
3. Verify a few specific months in dev, then prod-deploy.
4. After Google fully ingests the new structure (typically 2–4 weeks), delete the old sitemap-models.xml / sitemap-articles.xml routes.

Tradeoffs

For monthly partitioning:

• Long-tail content gets indexed
• Crawlers can skip unchanged periods → less origin load
• Better aligns with Google's incremental-crawl model

Against:

• Marginally more code (one new dynamic route per content type)
• Marginally more complex caching
• Doesn't help for static pages (no time dimension makes sense)

Net: worth doing for models and articles, not worth it for curated pages.

Companion policy: civitai.red default-deindex

The sitemap migration emits per-color URLs but does not, on its own, prevent Google from indexing pages on civitai.red that aren't in the red sitemap. The sitemap is a "please crawl these" hint — it does not stop Google from crawling and indexing other paths it discovers via internal links, external backlinks, or search-result-driven traffic.

To avoid duplicate-content competition between civitai.com and civitai.red, the cleanest posture is:

• civitai.red is canonical only for NSFW content. Anything PG/PG-13 or utility/static lives canonical on civitai.com.
• Default noindex on red, opt-in per page. Detail pages whose content is R+ explicitly opt back in. Browse/landing pages that present NSFW content (e.g. /models, /articles) opt back in. Everything else stays deindexed.

Where the deindex signal comes from

<meta name="robots" content="noindex,nofollow"> is emitted by src/components/Meta/Meta.tsx:124 when either:

• the request host isn't a known production color (computed via canIndex in src/pages/_app.tsx:306), or
• the calling page passes deIndex={true} to <Meta>.

The default-deindex policy is implemented by populating that second hook correctly, not by flipping canIndex host-wide.

Implementation: Gated drives detail-page deindex

The seven entity-detail surfaces (model, article, image, post, etc.) already flow through Gated for their NSFW gating. That makes Gated the natural choke point for the deindex decision, since it already has the four signals it needs: the gate state, the contentNsfwLevel, the allowMatureContent host flag (from AppContext), and the bot-bypass isPaywalled flag.

Gated defaults meta.deIndex to true in any of these states:

State	Why
redirect	civitai.com showing the "mature content lives on red" card. No real content visible.
login	Login wall. Crawlers see no content.
unrated	"Pending review" card. No real content visible.
page AND allowMatureContent AND isSafeForCanonical	civitai.red rendering SFW content normally. Green is canonical for that URL; red shouldn't compete.

isSafeForCanonical resolves the SFW question with the entity's coarse nsfw boolean acting as a force-NSFW override on top of the bitwise level:

• nsfw === true → treat as NSFW regardless of nsfwLevel
• otherwise → fall back to hasSafeBrowsingLevel(contentNsfwLevel)

Some entities (Model, Article, Post, Collection, Bounty) carry the coarse boolean alongside the bitwise level — pages should pass nsfw to Gated whenever the underlying record has it. A false value is treated the same as undefined (no override); only true flips the calculation.

The bot-bypass case (state === 'page' && isPaywalled) only fires for non-safe content (R+) per Gated.tsx:118, so isSafeForCanonical is false there and the URL stays indexable, with paywall structured data attached. That's exactly the case we built the bot bypass for, and bots see the same signal humans would have if they hit that URL on red — noindex for SFW, indexable for NSFW.

Caller and default combine via meta.deIndex || shouldDefaultDeIndex — if either says "deindex", the page is deindexed. The || (not ??) is deliberate: most callers pass deIndex: <boolean expression> that resolves to false for the common "published & searchable" case, and ?? would let that false mask the default. Pages cannot force-index past a Gated default — if Gated says deindex, the page deindexes. (Force-indexing past a Gated default is rarely the right thing anyway; Gated's defaults are designed to match the canonical-domain decision.)

When Gated started owning the deIndex decision, the inline <Head><meta robots noindex></Head> blocks inside MatureContentRedirect and LoginRequiredCard became redundant and were removed — single source of truth via Gated → Meta.

Implementation: static / utility pages

Gated only wraps content-detail pages. Static and utility pages on red (/safety, /support, /tos, /content-policies, /tools, /train, /shop, /buzz/marketplace, /user/*, etc.) need the same default-deindex posture but don't flow through Gated. They get it via a host check in their own Meta calls — either inline or via a shared useDefaultDeIndexOnRed() hook for ergonomics.

Pages that stay indexable on red

The red sitemap-pages.xml allowlist is the source of truth for what stays indexable on red:

• Content-browse pages: /, /models, /articles, /images, /posts, /videos, /comics, /comics/browse, /collections. These show NSFW content on red and are the red canonical for those URLs.
• /pricing: red has its own pricing tier specifics that differ from green's. Indexable on both as separate canonicals.
• /generate: same URL on both, but red's generator covers SFW + NSFW while green's is SFW-only — different value prop, different canonical.

These pages either don't pass deIndex (browse pages — Meta defaults to indexable) or explicitly pass deIndex={false} (the two exceptions).

Tricky cases

Surface	Indexable on red?	Reasoning
Model detail with nsfwLevel having no safe bits (e.g. R-only, X-only, R|X)	Yes	hasSafeBrowsingLevel false → isSafeForCanonical false → no auto-deindex. Red is the canonical.
Model detail with nsfwLevel having any safe bit (e.g. PG, PG|PG-13, PG|PG-13|R)	No	hasSafeBrowsingLevel true → isSafeForCanonical true → auto-deindex. Green is canonical.
Model detail with nsfw=true (any nsfwLevel)	Yes	Force-NSFW override flips isSafeForCanonical to false regardless of level.
Article detail with nsfwLevel having no safe bits	Yes	Articles don't pass nsfw to Gated, so the bitwise rule alone applies.
Article detail with nsfwLevel having any safe bit	No	Same; hasSafeBrowsingLevel true → auto-deindex on red.
/models, /articles, /images, etc. (browse pages)	Yes	Listed in red allowlist; different content per host.
/pricing	Yes	Red exception. Distinct content.
/generate	Yes	Red exception. Different value prop.
/user/[username]	No (recommended)	Mixed content; let green be canonical.
/tag/[tag], /search/*	No	Low SEO value; thin content.
/safety, /support, /tos, /content-policies	No	Same content as green; not NSFW-specific.
/tools, /train, /shop, /buzz/marketplace	No	Utility/commercial; canonical on green.

User profile pages (/user/[username]) are the trickiest: a user can have both PG and NSFW content. The simplest call is to deindex all user profiles on red and let them live canonical on green. Users who want their NSFW gallery indexed separately would need a per-user opt-in flag — not worth building until there's demand.

Audit + verification

After deploy, verify in Google Search Console (red property) that indexed-page count drops to NSFW detail pages + the allowlisted browse/exception pages over the following 1–2 weeks.

Two consistency checks worth adding to CI or a one-off audit script:

1. Every URL emitted by civitai.red/sitemap-models.xml resolves to a page that does not emit noindex.
2. Every URL emitted by civitai.red/sitemap-pages.xml resolves to a page that does not emit noindex.

Interaction with sitemap content

This policy is a complement to the sitemap split, not a substitute. The sitemap still controls what we ask Google to crawl; the noindex directives control what Google is allowed to index after crawling.

• A URL in red's sitemap should be indexable on red (otherwise we're contradicting ourselves).
• A URL not in red's sitemap should generally be noindex on red — that's what this policy enforces.