feat(moderation): block 17 additional character names in prompts (#4765)

Adds 17 lowercase character names to the POI wordlist, so that generation prompts containing them are refused.

## Mechanism (verified on this base, not inherited from a handoff)

`generateFromGraph` → `auditPromptServer` (`src/server/services/orchestrator/orchestration-new.service.ts:1588`, `:1641`) → `promptAuditing.ts:226` → `audit.ts:179` / `:195` `includesPoi` → `audit.ts:764` `checkable(poiWords)` → the JSON at `audit.ts:10`. Both prompt and negative prompt are checked. That procedure serves onsite and bearer-token API callers.

Not the mechanism, in case anyone looks there: the DB `Blocklist` has no POI type and can only *un*-block via `stripBenignPhrases`; XGuard on generation runs inside the catch after the legacy gate has already blocked; `ModerationRules` gates posting images and uploading models, not generation.

## 🔴 Two deploys are required, and merging is neither

The JSON is a static ESM import compiled into a module-level const (`audit.ts:759-770`) — no runtime reload, no Redis mirror.

1. **The main app** must deploy before the site blocks these prompts.
2. **`apps/moderator` must be released separately.** `@civitai/mod-utils` is a `workspace:*` dependency, so the moderator's copy of the list ships on its own release train. Until that ships, the moderator prompt-tester shows these prompts unhighlighted while production blocks them.

## Why the duplicate list is in this diff

`packages/civitai-mod-utils/src/prompt-audit/lists/words-poi.json` is a byte-identical hand-maintained copy of the same list, read by `apps/moderator`'s prompt-tester and prohibited-prompts pages. The first commit here updated only the app copy, which put the two out of sync — so the second commit re-syncs it. Both files are byte-identical at this head (verified by blob hash, both 1590 entries).

Nothing in the repo compares them, which is how the drift happened and why it was silent. A parity guard is filed as follow-up rather than built here.

## Side effect: post-hoc image flagging

The same list drives image scan results (`src/server/services/image-scan-result.service.ts:863`, `:1169`; `src/pages/api/webhooks/image-scan-result.ts:638`, `:1138`), setting `flags.poi`. Already-uploaded images whose stored prompt contains one of these names will be flagged. Likely desirable, but it adds moderation-queue volume after the deploy rather than nothing happening.

## POI is a hard block with no user override

`SOFT_BLOCK_CATEGORIES` (`audit.ts:76`) excludes `poi`, so a false positive is unappealable. Checked on this base rather than inherited: all 17 are pure ASCII, none already present in any list under `src/utils/metadata/lists/`, and zero substring collisions in either direction against the pre-existing 1573 entries. A reviewer additionally tried and failed to construct an innocent prompt that the preprocessor fuses into any of the 17.

Matching is word-boundary anchored, so `diddley` does not hit `diddl` — that is the half of the test that can actually fail.

## Why 17 and not 16

The preprocessor (`audit.ts:764-767`) deletes characters outside `[\w\s|:[\],]` rather than splitting on them, so a hyphenated spelling arrives as one fused token that the base name cannot match. The 17th entry is that fused form. Verified both directions: before it, `includesPoi('diddl-maus plush toy')` returned `false`; after, it returns the entry.

## Coverage limits

The block is not unconditional, and the prompt audit lives in this web app rather than in the orchestrator service — anything reaching the orchestrator without traversing this repo is not covered. Both are pre-existing and out of scope here; the specifics went to the owners privately rather than into a public repo.

## Verification

- `pnpm run typecheck` — 0 errors
- `pnpm run lint` — byte-identical totals to the pre-change run (4583 problems / 358 errors, all pre-existing); no finding in any changed file
- `pnpm exec prettier --check` on all changed files — clean
- `pnpm run test:packages:run` — `Test Files  89 passed | 3 skipped (92)`, exit 0
- Full unit suite — `Test Files  1 failed | 1752 passed | 3 skipped (1756)`, exit 1. The one failure is `src/server/integrations/__tests__/moderation-cache-probe.test.ts`, which **fails identically on clean `main` at the merge base** (controlled for: a full-suite run in a separate checkout at `1fbca93dba` fails the same test in the same place). It imports nothing this PR touches.
- `blocks.router.workflow.test.ts` collected 380 tests, so the submodule was initialised.
- Every assertion added here has a positive control that was run, not assumed. Reverting the JSON gives `AssertionError: diddl: expected false to be 'diddl'`; deleting only the 17th entry gives `expected false to be 'diddlmaus'`; stripping the word-boundary lookaheads from `audit.ts` reddens the benign controls with `diddley bow, blues guitar: expected 'diddl' to be false`; and changing the preprocessor to split on `-` instead of deleting it reddens the hyphen test alone with `expected 'diddl' to be 'diddlmaus'`.

Reviewed across five lanes (reuse, correctness, perf, tests, intent), three passes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AD7s3W1zvMR1TZRRwNA5Bq
This commit is contained in:
Justin Maier
2026-09-11 14:00:27 -06:00
committed by GitHub
parent 9ee050e5aa
commit ab24dc437c
3 changed files with 90 additions and 3 deletions
@@ -1571,5 +1571,22 @@
"trump jr",
"trump",
"Alexandria Ocasio-Cortez",
"Putin"
"Putin",
"diddl",
"diddlina",
"pimboli",
"loupsily",
"galupy",
"wollywell",
"simsaly",
"lollilovebear",
"mimihopps",
"ackaturbo",
"vanillivi",
"bibombl",
"milimits",
"tiplitaps",
"diddldaddl",
"blubberpeng",
"diddlmaus"
]
+54 -1
View File
@@ -1,5 +1,10 @@
import { describe, it, expect } from 'vitest';
import { includesInappropriate, includesMinor, includesMinorAge } from '~/utils/metadata/audit';
import {
includesInappropriate,
includesMinor,
includesMinorAge,
includesPoi,
} from '~/utils/metadata/audit';
describe('includesMinorAge', () => {
describe('danbooru/pony tag false positives', () => {
@@ -121,3 +126,51 @@ describe('young-word anchoring (minor-review queue)', () => {
}
});
});
// Do not delete these entries or this test without checking with a maintainer first.
describe('POI — Diddl character names', () => {
const diddlNames = [
'diddl',
'diddlina',
'pimboli',
'loupsily',
'galupy',
'wollywell',
'simsaly',
'lollilovebear',
'mimihopps',
'ackaturbo',
'vanillivi',
'bibombl',
'milimits',
'tiplitaps',
'diddldaddl',
'blubberpeng',
// Not redundant with `diddl`: the poi preprocessor DELETES `-` rather than splitting on
// it, so `diddl-maus` arrives as one token that `diddl` cannot match.
'diddlmaus',
];
it('blocks each name', () => {
for (const name of diddlNames) {
expect(includesPoi(`a drawing of ${name}, pastel colours`), name).toBe(name);
}
});
it('blocks the hyphenated spelling, which the preprocessor fuses into one token', () => {
expect(includesPoi('a drawing of diddl-maus')).toBe('diddlmaus');
});
// POI is a hard block with no user override, so a name matching inside a longer word is an
// unappealable false positive. Every control below really does contain `diddl`; only that
// entry is short enough to occur inside English words, so the others would assert nothing.
it('does not block longer real words that contain a name', () => {
for (const prompt of [
'diddley bow, blues guitar',
'diddling with the exposure slider',
'a paradiddle drum pattern',
]) {
expect(includesPoi(prompt), prompt).toBe(false);
}
});
});
+18 -1
View File
@@ -1571,5 +1571,22 @@
"trump jr",
"trump",
"Alexandria Ocasio-Cortez",
"Putin"
"Putin",
"diddl",
"diddlina",
"pimboli",
"loupsily",
"galupy",
"wollywell",
"simsaly",
"lollilovebear",
"mimihopps",
"ackaturbo",
"vanillivi",
"bibombl",
"milimits",
"tiplitaps",
"diddldaddl",
"blubberpeng",
"diddlmaus"
]