mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
feat(moderation): block 17 additional character names in prompts (#4765)
Adds 17 lowercase character names to the POI wordlist, so that generation prompts containing them are refused. ## Mechanism (verified on this base, not inherited from a handoff) `generateFromGraph` → `auditPromptServer` (`src/server/services/orchestrator/orchestration-new.service.ts:1588`, `:1641`) → `promptAuditing.ts:226` → `audit.ts:179` / `:195` `includesPoi` → `audit.ts:764` `checkable(poiWords)` → the JSON at `audit.ts:10`. Both prompt and negative prompt are checked. That procedure serves onsite and bearer-token API callers. Not the mechanism, in case anyone looks there: the DB `Blocklist` has no POI type and can only *un*-block via `stripBenignPhrases`; XGuard on generation runs inside the catch after the legacy gate has already blocked; `ModerationRules` gates posting images and uploading models, not generation. ## 🔴 Two deploys are required, and merging is neither The JSON is a static ESM import compiled into a module-level const (`audit.ts:759-770`) — no runtime reload, no Redis mirror. 1. **The main app** must deploy before the site blocks these prompts. 2. **`apps/moderator` must be released separately.** `@civitai/mod-utils` is a `workspace:*` dependency, so the moderator's copy of the list ships on its own release train. Until that ships, the moderator prompt-tester shows these prompts unhighlighted while production blocks them. ## Why the duplicate list is in this diff `packages/civitai-mod-utils/src/prompt-audit/lists/words-poi.json` is a byte-identical hand-maintained copy of the same list, read by `apps/moderator`'s prompt-tester and prohibited-prompts pages. The first commit here updated only the app copy, which put the two out of sync — so the second commit re-syncs it. Both files are byte-identical at this head (verified by blob hash, both 1590 entries). Nothing in the repo compares them, which is how the drift happened and why it was silent. A parity guard is filed as follow-up rather than built here. ## Side effect: post-hoc image flagging The same list drives image scan results (`src/server/services/image-scan-result.service.ts:863`, `:1169`; `src/pages/api/webhooks/image-scan-result.ts:638`, `:1138`), setting `flags.poi`. Already-uploaded images whose stored prompt contains one of these names will be flagged. Likely desirable, but it adds moderation-queue volume after the deploy rather than nothing happening. ## POI is a hard block with no user override `SOFT_BLOCK_CATEGORIES` (`audit.ts:76`) excludes `poi`, so a false positive is unappealable. Checked on this base rather than inherited: all 17 are pure ASCII, none already present in any list under `src/utils/metadata/lists/`, and zero substring collisions in either direction against the pre-existing 1573 entries. A reviewer additionally tried and failed to construct an innocent prompt that the preprocessor fuses into any of the 17. Matching is word-boundary anchored, so `diddley` does not hit `diddl` — that is the half of the test that can actually fail. ## Why 17 and not 16 The preprocessor (`audit.ts:764-767`) deletes characters outside `[\w\s|:[\],]` rather than splitting on them, so a hyphenated spelling arrives as one fused token that the base name cannot match. The 17th entry is that fused form. Verified both directions: before it, `includesPoi('diddl-maus plush toy')` returned `false`; after, it returns the entry. ## Coverage limits The block is not unconditional, and the prompt audit lives in this web app rather than in the orchestrator service — anything reaching the orchestrator without traversing this repo is not covered. Both are pre-existing and out of scope here; the specifics went to the owners privately rather than into a public repo. ## Verification - `pnpm run typecheck` — 0 errors - `pnpm run lint` — byte-identical totals to the pre-change run (4583 problems / 358 errors, all pre-existing); no finding in any changed file - `pnpm exec prettier --check` on all changed files — clean - `pnpm run test:packages:run` — `Test Files 89 passed | 3 skipped (92)`, exit 0 - Full unit suite — `Test Files 1 failed | 1752 passed | 3 skipped (1756)`, exit 1. The one failure is `src/server/integrations/__tests__/moderation-cache-probe.test.ts`, which **fails identically on clean `main` at the merge base** (controlled for: a full-suite run in a separate checkout at `1fbca93dba` fails the same test in the same place). It imports nothing this PR touches. - `blocks.router.workflow.test.ts` collected 380 tests, so the submodule was initialised. - Every assertion added here has a positive control that was run, not assumed. Reverting the JSON gives `AssertionError: diddl: expected false to be 'diddl'`; deleting only the 17th entry gives `expected false to be 'diddlmaus'`; stripping the word-boundary lookaheads from `audit.ts` reddens the benign controls with `diddley bow, blues guitar: expected 'diddl' to be false`; and changing the preprocessor to split on `-` instead of deleting it reddens the hyphen test alone with `expected 'diddl' to be 'diddlmaus'`. Reviewed across five lanes (reuse, correctness, perf, tests, intent), three passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AD7s3W1zvMR1TZRRwNA5Bq
This commit is contained in:
@@ -1571,5 +1571,22 @@
|
||||
"trump jr",
|
||||
"trump",
|
||||
"Alexandria Ocasio-Cortez",
|
||||
"Putin"
|
||||
"Putin",
|
||||
"diddl",
|
||||
"diddlina",
|
||||
"pimboli",
|
||||
"loupsily",
|
||||
"galupy",
|
||||
"wollywell",
|
||||
"simsaly",
|
||||
"lollilovebear",
|
||||
"mimihopps",
|
||||
"ackaturbo",
|
||||
"vanillivi",
|
||||
"bibombl",
|
||||
"milimits",
|
||||
"tiplitaps",
|
||||
"diddldaddl",
|
||||
"blubberpeng",
|
||||
"diddlmaus"
|
||||
]
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { includesInappropriate, includesMinor, includesMinorAge } from '~/utils/metadata/audit';
|
||||
import {
|
||||
includesInappropriate,
|
||||
includesMinor,
|
||||
includesMinorAge,
|
||||
includesPoi,
|
||||
} from '~/utils/metadata/audit';
|
||||
|
||||
describe('includesMinorAge', () => {
|
||||
describe('danbooru/pony tag false positives', () => {
|
||||
@@ -121,3 +126,51 @@ describe('young-word anchoring (minor-review queue)', () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// Do not delete these entries or this test without checking with a maintainer first.
|
||||
describe('POI — Diddl character names', () => {
|
||||
const diddlNames = [
|
||||
'diddl',
|
||||
'diddlina',
|
||||
'pimboli',
|
||||
'loupsily',
|
||||
'galupy',
|
||||
'wollywell',
|
||||
'simsaly',
|
||||
'lollilovebear',
|
||||
'mimihopps',
|
||||
'ackaturbo',
|
||||
'vanillivi',
|
||||
'bibombl',
|
||||
'milimits',
|
||||
'tiplitaps',
|
||||
'diddldaddl',
|
||||
'blubberpeng',
|
||||
// Not redundant with `diddl`: the poi preprocessor DELETES `-` rather than splitting on
|
||||
// it, so `diddl-maus` arrives as one token that `diddl` cannot match.
|
||||
'diddlmaus',
|
||||
];
|
||||
|
||||
it('blocks each name', () => {
|
||||
for (const name of diddlNames) {
|
||||
expect(includesPoi(`a drawing of ${name}, pastel colours`), name).toBe(name);
|
||||
}
|
||||
});
|
||||
|
||||
it('blocks the hyphenated spelling, which the preprocessor fuses into one token', () => {
|
||||
expect(includesPoi('a drawing of diddl-maus')).toBe('diddlmaus');
|
||||
});
|
||||
|
||||
// POI is a hard block with no user override, so a name matching inside a longer word is an
|
||||
// unappealable false positive. Every control below really does contain `diddl`; only that
|
||||
// entry is short enough to occur inside English words, so the others would assert nothing.
|
||||
it('does not block longer real words that contain a name', () => {
|
||||
for (const prompt of [
|
||||
'diddley bow, blues guitar',
|
||||
'diddling with the exposure slider',
|
||||
'a paradiddle drum pattern',
|
||||
]) {
|
||||
expect(includesPoi(prompt), prompt).toBe(false);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1571,5 +1571,22 @@
|
||||
"trump jr",
|
||||
"trump",
|
||||
"Alexandria Ocasio-Cortez",
|
||||
"Putin"
|
||||
"Putin",
|
||||
"diddl",
|
||||
"diddlina",
|
||||
"pimboli",
|
||||
"loupsily",
|
||||
"galupy",
|
||||
"wollywell",
|
||||
"simsaly",
|
||||
"lollilovebear",
|
||||
"mimihopps",
|
||||
"ackaturbo",
|
||||
"vanillivi",
|
||||
"bibombl",
|
||||
"milimits",
|
||||
"tiplitaps",
|
||||
"diddldaddl",
|
||||
"blubberpeng",
|
||||
"diddlmaus"
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user