civitai__civitai/docs/article-ingestion-status-proposal.md

Article Ingestion Status — Design Proposal

Status: Draft, pending implementation decision Context: Surfaced during review of article-scanning-rollout.md on 2026-04-10, updated 2026-04-12 Related: civitai/civitai#1879 (feature/scan-article-images), article-content-scanning.md

---

TL;DR

Add an ArticleIngestionStatus enum + Article.ingestion column that mirrors how Image.ingestion already gates image serving. Non-owner, non-moderator article reads filter on ingestion = 'Scanned'. This:

1. Closes the legacy-backfill Green exposure window — mitigated by deployment strategy (see §"Gap 1" below). Backfills run against prod DB from staging before the production deploy, so legacy articles are already scanned when Green access turns on.
2. Supersedes the accepted async text-scan window decision from 2026-04-10 — not by reversing it, but by offering a gating mechanism the earlier discussion didn't consider.
3. Unifies image and text scan gating under a single predicate.
4. Separates scan state (system) from publication state (user intent), which removes most of the awkwardness around ArticleStatus.Processing.
5. Adds proper scan-lifecycle date tracking (scanRequestedAt, scannedAt) to Article, mirroring Image. Fixes the contentScannedAt misnomer and closes the gap where text-only articles have no scan timestamp at all.

Decision pending: ship this as a followup (recommended) vs. absorb into the current PR (bigger scope, cleaner landing).

---

The problem this solves

Gap 1 — Legacy backfill exposes unscanned content to Civitai Green (MITIGATED)

2026-04-12 update: This gap is closed by the deployment strategy. Both backfills will run against the prod DB from a staging environment before the production deploy ships articles: ['public']. By the time Green access turns on, every legacy article already has correct ImageConnection rows, recomputed nsfwLevel masks, and EntityModeration results. There is no window where unscanned legacy content is visible to Green.

The original analysis below is preserved for context on why the gap existed in the first rollout sequence, but it is no longer a blocker.

Original analysis (preserved for context)

The current rollout sequence (§4 of article-scanning-rollout.md) is:

1. Deploy code — feature flags already at articles: ['public'] → Civitai Green starts serving the entire legacy article corpus immediately.
2. Run image backfill (migrate-article-images.ts).
3. Run text moderation backfill (migrate-article-text-moderation.ts).

Between steps 1 and 2, a legacy article's nsfwLevel is computed from just cover image + userNsfwLevel — because ImageConnection rows for content images don't exist yet, so GREATEST(max(cover), max(content)) in updateArticleNsfwLevels has nothing to max over on the content side. Text moderation hasn't run, so no floor: NsfwLevel.R has been applied either. (Pre-2026-04-17: same effective outcome, just expressed as "article.nsfw is still false".)

So a legacy article with a PG cover, NSFW content images embedded in the body, and NSFW text currently has nsfwLevel = PG. After step 1 deploys, Civitai Green filters by (nsfwLevel & publicBrowsingLevelsFlag) != 0 and happily serves that article. The backfills eventually fix it, but the entire legacy corpus is visible to Green at its pre-scan level for the duration of the backfill window.

The commit message for 28b38c7dd acknowledges that legacy masks are wrong and will be updated by the backfill. article-scanning-rollout.md §2.3 calls this out directly: "running updateArticleNsfwLevels over legacy articles (which happens implicitly when backfills complete) will change existing masks for any article that mixed PG + NSFW content images under the old code." Any article whose mask changes during the backfill was being served at the wrong level before the backfill ran. That's the leak window.

Gap 2 — The live text-scan window is still there

Per the 2026-04-10 project decision, new text-only articles publish before their xGuard scan completes. That decision was made because the alternative (Option A: force Processing on all new articles) added visible UX cost — authors would see a "Processing" dialog on what used to be instant-publish.

This proposal is not Option A. It closes the window without the UX cost, because the gate is serving-side, not publication-side. Authors still publish instantly from their POV. Green just doesn't see the article until both scans complete.

---

The proposal

Schema

enum ArticleIngestionStatus {
  Pending   // at least one scan (image or text) hasn't completed
  Scanned   // both image extraction+scan and text moderation completed successfully
  Blocked   // text moderation blocked OR an image scan blocked
  Error     // text moderation failed/expired OR image scan errored terminally
  Rescan    // content changed, needs re-scanning (mirrors ImageIngestionStatus.Rescan)
}

model Article {
  // ... existing fields
  contentScannedAt DateTime?              // repurposed: set when BOTH scans complete (was: set at image-link time)
  ingestion        ArticleIngestionStatus @default(Pending)
  scanRequestedAt  DateTime?              // when scan was first requested (image link + text submit)

  @@index([status, ingestion, nsfwLevel])  // supports the Green query shape
}

These fields mirror Image.scanRequestedAt and Image.scannedAt. They give us:

• Filterability: query for articles that were never scanned (contentScannedAt IS NULL), or that have been waiting too long (scanRequestedAt < NOW() - INTERVAL '1 hour' AND ingestion = 'Pending').
• Observability: measure scan latency (contentScannedAt - scanRequestedAt), find stuck articles, build dashboards.
• Parity with Image: same semantics, same query patterns.

contentScannedAt repurposed (not deprecated)

The existing contentScannedAt field was previously set when linkArticleContentImages ran (i.e., when image links were created), not when scanning completed. Text-only articles never got it set at all.

New semantics: contentScannedAt is now set only inside recomputeArticleIngestion when ingestion transitions to Scanned — meaning both image scans and text moderation have completed successfully. The name is now accurate: "when was this article's content actually scanned."

scanRequestedAt replaces the old contentScannedAt write timing — it's set when linkArticleContentImages + submitTextModeration fire, tracking "when did we kick off scanning."

Migration: adds two new fields (ingestion, scanRequestedAt). ingestion defaults to Pending. contentScannedAt keeps its column but its write sites are redirected. Every existing article is Pending on schema apply — every legacy article is immediately invisible to Civitai Green the moment the migration runs (if the serving gates are deployed with the migration). scanRequestedAt defaults to NULL for legacy articles and gets populated as backfills run.

Serving-path gate

Every public/Green read path already filters on status = 'Published' and nsfwLevel & browsingLevelsFlag != 0. Add one more predicate for non-owner, non-moderator callers:

AND a.ingestion = 'Scanned'

Call sites identified in the initial gotchas review:

• getArticles (src/server/services/article.service.ts:257)
• getArticleById (src/server/services/article.service.ts:639)
• getCivitaiNews, getCivitaiEvents
• sitemap-articles.xml
• /api/og
• articles.search-index.ts:175 — Meilisearch pull filter (scoped to status = Published today; add ingestion = 'Scanned')
• Any feed/query that goes through getArticles-style gates

Mods and owners see any ingestion state so authors don't lose visibility of their own in-flight articles.

State transitions — Option A (derive from ground truth on events, recommended)

No new bookkeeping fields. Implement a helper that recomputes Article.ingestion from the tables that are already the source of truth:

// src/server/services/article.service.ts
export async function recomputeArticleIngestion(articleId: number): Promise<void> {
  const [imageAggregate, textModeration] = await Promise.all([
    dbWrite.$queryRaw<{
      pending: bigint;
      blocked: bigint;
      error: bigint;
      scanned: bigint;
      total: bigint;
    }[]>`
      SELECT
        COUNT(*) FILTER (WHERE i.ingestion = 'Pending') as pending,
        COUNT(*) FILTER (WHERE i.ingestion = 'Blocked') as blocked,
        COUNT(*) FILTER (WHERE i.ingestion = 'Error') as error,
        COUNT(*) FILTER (WHERE i.ingestion = 'Scanned') as scanned,
        COUNT(*) as total
      FROM "ImageConnection" ic
      JOIN "Image" i ON i.id = ic."imageId"
      WHERE ic."entityType" = 'Article' AND ic."entityId" = ${articleId}
    `,
    dbWrite.entityModeration.findUnique({
      where: { entityType_entityId: { entityType: 'Article', entityId: articleId } },
      select: { status: true, blocked: true },
    }),
  ]);

  const agg = imageAggregate[0];
  const noImages = agg.total === 0n;
  const imageDone = noImages || agg.pending === 0n;
  const textDone = textModeration?.status === 'Succeeded';
  const anyBlocked = agg.blocked > 0n || textModeration?.blocked === true;
  const anyError =
    agg.error > 0n ||
    (textModeration && ['Failed', 'Expired', 'Canceled'].includes(textModeration.status));

  const next: ArticleIngestionStatus = anyBlocked
    ? 'Blocked'
    : anyError
    ? 'Error'
    : imageDone && textDone
    ? 'Scanned'
    : 'Pending';

  const now = new Date();
  await dbWrite.article.update({
    where: { id: articleId },
    data: {
      ingestion: next,
      // Set scannedAt only on the transition to Scanned, and only if not already set
      // (mirrors Image.scannedAt logic — preserves original scan date on rescans)
      ...(next === 'Scanned' ? { scannedAt: now } : {}),
    },
  });
}

Important edge case: a brand-new article with no content images and text moderation pending is imageDone=true (no images to wait on) but textDone=false, so it correctly stays Pending. Don't let the noImages shortcut mark it Scanned.

Call sites for recomputeArticleIngestion (sets ingestion + scannedAt):

• src/pages/api/webhooks/image-scan-result.ts — after each image scan result is processed (it already queries which articles the image belongs to for the debounce logic)
• src/pages/api/webhooks/text-moderation-result.ts — inside the Article entity handler, after recordEntityModerationSuccess/Failure
• src/server/services/article.service.ts upsert path — after linkArticleContentImages + submitTextModeration (handles the "new article with no images, text pending" base case — writes ingestion: Pending explicitly)
• src/pages/api/admin/temp/migrate-article-images.ts — at the end of each processed article
• src/pages/api/admin/temp/migrate-article-text-moderation.ts — not strictly needed because the text backfill just submits to xGuard; the webhook handler is what updates state when xGuard responds

Call sites for scanRequestedAt (set once when scanning is first kicked off):

• src/server/services/article.service.ts upsert path — set scanRequestedAt = new Date() alongside the linkArticleContentImages + submitTextModeration calls. This is the "we started scanning" timestamp. Covers both image-bearing and text-only articles because text moderation always fires.
• src/pages/api/admin/temp/migrate-article-images.ts — set scanRequestedAt for each legacy article as it's processed by the backfill (if not already set).
• src/pages/api/admin/temp/migrate-article-text-moderation.ts — set scanRequestedAt for text-only legacy articles that have no content images and were missed by the image backfill.

Option B — Two booleans (rejected)

Add imageScanComplete: Boolean and textScanComplete: Boolean to Article. Each webhook sets its own boolean; a helper derives ingestion.

Rejected because: two more columns, two more denormalized states that can drift from the ground truth in Image.ingestion and EntityModeration.status. Option A has the same number of SQL round-trips but no drift risk.

---

Relationship to ArticleStatus

Currently ArticleStatus.Processing conflates scan state with publication state. With Article.ingestion, we keep both but let them mean different things:

Axis	Column	Values
User publication intent	ArticleStatus	Draft, Published, Unpublished, UnpublishedViolation
System scan state	ArticleIngestionStatus	Pending, Scanned, Blocked, Error, Rescan

ArticleStatus.Processing becomes mostly redundant. The auto-publish flow in updateArticleImageScanStatus (article.service.ts:1678-1733) currently does "when all images scanned, flip status: Processing → Published". Under the new model, it does "when all images scanned, call recomputeArticleIngestion(articleId), don't touch status". If the user's publication intent is Published, they're already Published from their POV — now they become visible to the public when ingestion transitions to Scanned.

Two paths for removing Processing:

1. Keep Processing as a no-op transitional state — don't remove it from the enum, just stop using it. Lowest-risk, minimizes schema churn. Downsides: dead state value, new enum value (Processing) from this branch becomes vestigial.
2. Remove Processing from the enum — cleaner but requires a second migration, and any external consumer (ClickHouse/Metabase/etc. — see rollout doc §5.7.2) that saw Processing during the branch's lifetime has to handle both the new and removed states.

Recommend Path 1 for this followup. The vestigial enum value can be removed in a later cleanup PR.

Re-scan on edit (currently flips status = Processing in article.service.ts:953): under the new model, flip ingestion = Rescan (or back to Pending) instead. Keep status = Published. Green can't see the article during the re-scan window, but the author still sees it as Published in their dashboard.

---

Legacy backfill story

2026-04-12 update: Since Gap 1 is mitigated by running backfills from staging against prod DB before deploy, the ingestion column is no longer the primary safety gate for legacy content. However, the ingestion column + date fields still provide long-term value: filterability, observability, and a self-gating mechanism for any future backfill or re-scan scenario.

The migration adds ingestion (default Pending), scanRequestedAt (default NULL), and scannedAt (default NULL) to every existing article.

Backfill flow (run from staging against prod DB before production deploy):

1. Image backfill runs — for each article: extract media, create ImageConnection rows, kick off image scans, set scanRequestedAt = NOW(). At the end of each article, call recomputeArticleIngestion(id). Articles with no content media transition toward Scanned if text side is also done; articles with content media stay Pending until the async image scans complete.
2. Image scan webhooks — as each content image completes scanning, the webhook calls recomputeArticleIngestion(articleId). When the last scan finishes (and text is already Succeeded), ingestion flips to Scanned and scannedAt is set.
3. Text backfill runs in parallel — submits each article to xGuard, sets scanRequestedAt if not already set (covers text-only articles). The webhook handler calls recomputeArticleIngestion after processing the xGuard response.
4. Articles light up for Green one at a time, as each finishes both sides. No cliff-edge "all of prod becomes visible at once" moment.
5. Production deploy ships with articles: ['public']. Legacy corpus is already scanned and has correct ingestion, scannedAt, and nsfwLevel values.

---

Live path for new articles

1. User creates article → ingestion = Pending (default).
2. Upsert path runs linkArticleContentImages + submitTextModeration → explicitly calls recomputeArticleIngestion to lock in Pending (or Scanned if there's no content and somehow text is already done, which shouldn't happen synchronously).
3. Content images scan asynchronously via the existing image pipeline. Each scan result webhook calls the helper.
4. Text moderation returns asynchronously via xGuard. The text-moderation webhook handler calls the helper.
5. When both are done, ingestion = Scanned, article is visible to Green.

Authors don't see a Processing dialog. They don't experience any UX regression from the status quo on main. The 2026-04-10 concern about "visible UX cost on instant-publish articles" doesn't apply because the gate is invisible to authors.

---

Scope estimate

Area	Files	Rough lines
Schema migration	prisma/schema.full.prisma, generated migration SQL	~25
Helper function	src/server/services/article.service.ts	~55
Webhook wiring	image-scan-result.ts, text-moderation-result.ts	~20
Serving gates	~6 read sites + Meilisearch index filter	~50
Remove Processing from auto-publish	article.service.ts updateArticleImageScanStatus	~15
Upsert path wiring + scanRequestedAt	article.service.ts create + update branches	~20
Re-scan path	article.service.ts upsert status-flip logic	~10
Backfill wiring + scanRequestedAt	migrate-article-images.ts	~20
Repurpose contentScannedAt writes	article.service.ts (old writes → scanRequestedAt, new writes via helper)	~5
Tests	new tests for recomputeArticleIngestion transitions + date fields	~170

A few hundred lines across ~15 files + one DB migration. Doable in a session or two of focused work. The existing parity test from the current PR (src/utils/__tests__/article-extraction-parity.test.ts) is unaffected.

---

Open questions to resolve before writing code

1. Author-facing UI impact — the ArticleScanStatus component in ArticleUpsertForm.tsx currently reads article.status === Processing to show scan progress. Does it need to switch to reading article.ingestion === Pending? If so, what does the author see while text is still scanning but images are done? (Current behavior on text-only articles: nothing — they publish instantly. New behavior: they see a "text scan pending" indicator?) This is the one place where the 2026-04-10 UX concern could resurface. The solution is probably to keep the author-facing indicator quiet for text-only articles and only surface it when the delay exceeds some threshold, or when scan fails.

2. article.nsfw text-moderation flag — deprecated 2026-04-17. Text moderation no longer writes the nsfw boolean. Instead, the webhook just calls updateArticleNsfwLevels([id]); the service reads the EntityModeration row (persisted moments earlier by recordEntityModerationSuccess) via an EXISTS subquery and applies an R floor when the text was flagged. The field stays in the DB schema for backward compatibility but is no longer read or written anywhere. ingestion = Blocked still tracks the orthogonal "text violated ToS" axis via status = UnpublishedViolation.

3. Rescan visibility — when an author edits a Published article and adds new content, should the article stay visible at the old nsfwLevel until the new scan completes, or should it immediately disappear from Green? The safe answer is "disappear" (flip ingestion = Rescan, serving gate hides it). The status-quo-preserving answer is "stay visible at old level" (don't flip ingestion until the new scan completes, at which point update in place). Safe is better for a rollout; revisit later if authors complain.

4. Processing auto-publish notification — the "Your article has been published successfully!" message fires on Processing → Published transition. Under the new model: does it fire on ingestion = Scanned? Or drop it entirely because the author never saw a "processing" state to begin with? Probably drop it — we're making the scan gate invisible, so a "finished scanning" notification is noise. Except for text-only articles that went through the scan window: those users never expected to see a notification, so dropping is consistent.

5. Moderation tooling — any admin UI that filters articles by status. Does it need an ingestion filter too? Probably not — mods should see everything by default, but a "show me articles stuck in Pending" filter might be useful for debugging scan pipeline issues. Add in a followup if requested.

6. Index choice — @@index([status, ingestion, nsfwLevel]) is one option. The actual best index depends on query patterns. Check getArticles query plans before committing to a specific shape; a partial index on (ingestion, nsfwLevel) WHERE status = 'Published' might be better if most Green queries also filter to Published.

7. contentScannedAt on rescan — when an author edits a Published article and triggers a rescan, recomputeArticleIngestion will overwrite contentScannedAt when the new scan completes. The old timestamp is lost. If we need history, consider keeping the old value and only overwriting on Scanned transition (current implementation). This means contentScannedAt always reflects "last time all scans were green."

---

Two paths forward

2026-04-12 decision: Path B was chosen — the ingestion refactor is absorbed into the scanning PR on feature/scan-article-images. One clean landing with ingestion + scanRequestedAt + repurposed contentScannedAt available from the start.

Implementation (Path B — absorbed into this PR)

1. Schema migration adds ArticleIngestionStatus enum, ingestion (default Pending), scanRequestedAt columns, and composite index.
2. recomputeArticleIngestion helper derives ingestion state from ImageConnection + Image.ingestion and EntityModeration ground truth.
3. Helper is called from: image scan webhook (updateArticleImageScanStatus), text moderation webhook (success + failure), and backfill (migrate-article-images.ts).
4. Upsert paths set scanRequestedAt + ingestion: Pending instead of the old premature contentScannedAt write.
5. Serving gates (ingestion = 'Scanned') added to: getArticles, getArticleById, getCivitaiNews, Meilisearch index, OG endpoint. Owners and mods bypass.
6. Run backfills from staging against prod DB. Articles light up to Scanned as both scan sides complete.
7. Deploy. New articles are gated by ingestion = 'Scanned' from day one.

---

Next-session checklist

Implementation status (2026-04-12):

• Path B chosen and implemented — ingestion refactor absorbed into the scanning PR.
• Schema: ArticleIngestionStatus enum, ingestion + scanRequestedAt fields, composite index, migration SQL.
• recomputeArticleIngestion helper — derives state from ImageConnection + EntityModeration ground truth.
• Webhook wiring — image scan (updateArticleImageScanStatus) + text moderation (success + failure paths).
• Upsert paths — scanRequestedAt + ingestion: Pending replaces old contentScannedAt writes.
• Serving gates — getArticles, getArticleById, getCivitaiNews, Meilisearch index, OG endpoint.
• Backfill wiring — migrate-article-images.ts uses scanRequestedAt + calls recomputeArticleIngestion.

Still open:

• Confirm the historical Green state — did articles: ['public'] on main already mean Green sees articles? Check git blame on src/server/services/feature-flags.service.ts.
• The §5.6 parity test (src/utils/__tests__/article-extraction-parity.test.ts) and §5.2 stale Succeeded re-enforcement fix are still on the table from the original gotchas review.
• ArticleScanStatus UI component — may need updating to read ingestion instead of (or in addition to) status === Processing.
• Processing auto-publish notification — currently fires on Processing → Published; consider whether it should also fire on ingestion → Scanned transition, or be dropped.
• Run pnpm run typecheck to verify all type references resolve.

---

User-triggered rescan

With Article.ingestion in place, we can give authors a "request rescan" action for their own articles. This is useful when:

• An article is stuck in Pending or Error due to a transient scan pipeline issue.
• The author edited their article but scanning didn't re-trigger (e.g., content hash dedupe blocked re-moderation after an xGuard recalibration).
• A moderator restored an article from UnpublishedViolation and the author wants to confirm it scans clean.

Proposed UX: a button in the article edit form (visible when ingestion is not Scanned) that:

1. Resets ingestion = Rescan, clears scannedAt (keeps scanRequestedAt for history), and sets a new scanRequestedAt.
2. Re-runs linkArticleContentImages to pick up any image changes.
3. Re-submits text moderation to xGuard (requires clearing the contentHash on EntityModeration for this article so the dedupe check doesn't skip it).
4. The normal webhook flow calls recomputeArticleIngestion as results come back.

Rate limiting: cap at N rescans per article per 24h to prevent abuse. Store count in ArticleMetadata or a simple Redis key.

Scope: this is a followup feature, not part of the initial ingestion refactor. But the ingestion column and date fields make it trivial to implement later — without them, there's no clean way to track "rescan requested" vs "never scanned" vs "scan complete".

---

Files referenced in this proposal

For quick navigation in the next session:

• Schema: prisma/schema.full.prisma (never edit schema.prisma, it's generated)
• Article service: src/server/services/article.service.ts
  • Upsert create path: ~line 800
  • Upsert update path: ~line 930-1100
  • updateArticleImageScanStatus auto-publish: ~line 1678-1733
  • linkArticleContentImages: ~line 1428
• NSFW level math: src/server/services/nsfwLevels.service.ts:280-325
• Webhooks:
  • src/pages/api/webhooks/image-scan-result.ts
  • src/pages/api/webhooks/text-moderation-result.ts (entity handler: lines 25-66)
• Backfills:
  • src/pages/api/admin/temp/migrate-article-images.ts
  • src/pages/api/admin/temp/migrate-article-text-moderation.ts
• Feature flags: src/server/services/feature-flags.service.ts (the articles entry)
• Extraction (already has parity test locked in): src/server/services/article-content-cleanup.service.ts, src/utils/article-helpers.ts
• Existing image ingestion pattern to mirror: Image.ingestion in schema, src/server/services/image.service.ts query sites (line 1535, 4509, 3991, etc.)