mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
fix(moderation): apply the moderator benign whitelist to search (#4143)
* fix(moderation): apply the moderator benign whitelist to search The benign-phrase lists moderators maintain at /moderator/blocklists were read by three server call sites and none of the search paths, so a whitelisted phrase still tripped real-person and profanity detection in search. - image-scan-result: strip benign phrases before the POI check that writes a confidence-100 tag. The audit copy in the same file already stripped; this one did not, and its tag reaches the image search index. - Ship the lists to the browser (system.getBenignPhrases, public + edge-cached), since search queries Meili from the client and has no server hop to strip on. Both search gates now run detection on the stripped copy and send the raw query to Meili. - Add BlocklistType.ProfanityBenignWord so the word-level whitelist behind the profanity filter is moderator-editable too, seeded from the static list and unioned with it rather than replacing it. - getBlocklistDTO: read deterministically and log when a type has more than one row, instead of letting findFirst and the type-keyed cache disagree about which wins. - Remove the PromptAllowlist path: the runtime filter was never enabled, the moderator mutation had no UI, and the table is empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(moderation): refuse to pick a blocklist row silently EmailDomain has two rows in production and the read took findFirst with no orderBy, while upsert cached the row it had just written under a type-scoped key — so which set was enforced depended on the DB and on whichever row was edited last. Reads the lowest id deterministically and reports the duplicate instead. Not a union: on a deny-list a union blocks more, but on the benign lists it strips more, which is a bypass. Not a throw either: this read gates signup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(moderation): strip benign phrases on the orchestrator scan path too The POI tagger fix landed on the legacy webhook branch only. `processTags` in image-scan-result.service.ts is its twin on the orchestrator path (the webhook routes bodies carrying workflowId there) and still read the raw prompt, so a whitelisted phrase was still written as a confidence-100 tag that reaches the search index. Found by review. Adds a source-level guard over both scan files: every includesPoi call must read a stripped prompt. A behavioural test proves the copy it drives, and the copy nobody drove was the live one. Review fixes alongside it: - pin the type filter and the list argument in tests that stayed green without them - cover the cache write in upsertBlocklist, which was the actual production incident and had no test at all - the moderator profanity list now REPLACES the static one rather than unioning, so the UI's Remove control is not a silent no-op; the static list remains the empty-case fallback - getClientBenignLists fails open to empty lists, which strips nothing and so flags more - hold the CSAM tracking event until the lists load; the gate itself still applies - build the obscenity filter lazily and share one per whitelist, including in RenderHtml, which was building a full dataset per comment Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(moderation): close the safety findings on the benign whitelist Two of these were introduced by earlier fixes in this PR, not by the original code. - The fail-open added for the public read returned a well-formed 200, which edgeCacheIt holds for an hour where the exception it replaced was never cached; with the client's staleTime that pinned empty whitelists for whole sessions. It now skips the edge cache. - Gating the CSAM tracking event on isSuccess suppressed it permanently after a failed fetch. A failed fetch means there is no whitelist, which is when there is least reason to withhold, so it gates on settled instead. - An empty moderator row and a missing one were collapsed, so deleting the last entry silently restored the shipped list over the top of the clearest possible intent not to. - The whitelist set had no self-exclusion guard: a single-token entry equal to a profane word disarmed the filter for it. Harmless while the list was a checked-in file. - The phrase separator is bounded, matching audit.ts. Unbounded blocks more for a detector and strips more for a whitelist, so it is not the safe default on this side. - Both taggers strip the normalized prompt, as the audit paths already did; stripping raw text next to an audit reading normalized text let two alphabets decide one prompt. - Finite staleTime so a moderator REMOVING a phrase reaches open tabs. - Scrub production list sizes from the migration header; this repo is public. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(moderation): make the cache-skip and null-vs-empty fixes actually take effect Both were inert as written; the review caught it on re-read. - ctx.cache.skip cannot work from inside the resolver: edgeCacheIt reads it to compute the TTL BEFORE calling next(). Set canCache/edgeTTL/browserTTL instead, all three, because canCache:false skips the block that assigns the TTLs and the context defaults for an anonymous caller are 60 rather than 0. - The hook collapsed null into [] before it reached the filter, so 'no moderator row' was read as 'a moderator deleted every entry' on three ordinary paths: before the query resolves, after a failed fetch, and before the seed migration is applied. Each dropped the shipped whitelist in the search box. Adds the hook's first tests, including the pre-load window and both directions of null-vs-empty; reverting the collapse fails two of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(moderation): revert the separator bound; it broke the whitelist it was meant to protect The {1,3} bound diverged from prepareWordRegex in audit-base.ts, which is unbounded — so the detector matched 'emma,,,, stone' while the strip did not, and a whitelisted phrase went back to tripping the gate on ordinary prompt punctuation. Reproduced on four inputs before reverting. The comment justifying the bound also cited audit.ts as precedent for bounding this, which was wrong: audit.ts bounds a composed noun gap, not the inter-word separator. The test that appeared to cover it was vacuous — its 40-char run of '!' is deleted by the POI preprocessor, so it was green with any matcher or none. Replaced with cases the detector demonstrably matches raw, so the strip has real work to do; re-applying the bound fails four of them. Also from the re-review: - pin the null-vs-empty distinction at getProfanityFilter, where production decides it, rather than only at createProfanityFilter one layer below - paren-balance the POI guard so a strip on a LATER statement no longer satisfies it, and document what it cannot see: arrow functions, and the two non-tag-writing POI call sites that read a raw prompt and are follow-up work - reset the strip mock between tests instead of relying on being the last describe - reuse the where/orderBy-honouring fake in the cache-write tests - the filter cache comment claimed a hypothetical the finite staleTime already makes real Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(moderation): correct why the unbounded separator is safe The comment claimed the class 'cannot cross an alphanumeric character, so it can only ever consume punctuation and whitespace'. [^a-zA-Z0-9] excludes only ASCII alphanumerics, so it consumes non-ASCII letters too — it can swallow something that reads as a word. What actually holds this closed is that every caller strips the NORMALIZED copy: normalizeText folds accented Latin to ASCII, and those letters then break the separator run. Moving a strip back ahead of normalizeText reopens it, and three of the six sites were that way before this PR — so the comment now says that rather than implying the character class is inherently safe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(moderation): say narrow, not closed, and fix the guard's stated reason Two comments claiming more than is true, both found by review: - The separator note said stripping the normalized copy CLOSES the character-class hole. It narrows it. Reproduced against the real lists: accented Latin folds to ASCII so the POI list is unreachable this way, but two entries of the minor list are scripts NFD cannot fold and survive as non-ASCII. Counts re-run rather than taken on trust. - The guard's comment-stripping still explained itself by the fixed inspection window, which paren-balancing replaced. The real reason is the scope check: a comment MENTIONING stripBenignPhrases between a declaration and the call would satisfy it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(moderation): refuse to strip when the gap between phrase words holds a letter The separator class excludes only ASCII alphanumerics, so a term written in non-ASCII letters is a word to the detector and a separator to the strip. That is not one term hidden: the nsfw word list GATES the POI and minor sub-checks (audit.ts, and every search caller passes no nsfw argument), so swallowing the only nsfw signal in an input stops those sub-checks running at all. Verified on a real list term inside a real two-word POI name: unguarded the text blanks and includesInappropriate returns false; guarded it still returns 'poi'. Refusal after the match rather than a bound on the separator — a bound made this matcher disagree with prepareWordRegex and handed back the false positives on ordinary punctuation. The pattern is unchanged and still identical to the detector's; only an occurrence whose gap holds a letter is declined. The four punctuation regression cases still strip. The server strip now shares the client helper so both sides refuse identically. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(moderation): the gap guard is about content, not letters \p{L} missed 9 of the 54 entries that survive normalization across the detection lists — all emoji, which carry no letter — so for those the swallow still worked. Not reachable today, and only because auditMetaData selects that list on nsfw === false while every caller passing a stripped prompt sits inside if (nsfw && …). That is a coincidence between two files that do not reference each other, and either side can close it silently. Widened to [\p{L}\p{N}\p{Extended_Pictographic}]. Measured: covers all 54, and refuses none of the ordinary gaps — including em-dash and ideographic comma, which a 'whitespace or ASCII punctuation only' predicate would have wrongly refused. Tests both directions: emoji and non-ASCII digit gaps refuse; em-dash and ideographic comma gaps still strip. Narrowing back to \p{L} fails the first pair. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(moderation): tell moderators the gap rule on the blocklist page A moderator adds a phrase, tests it, sees it work, then later sees something flagged anyway — and because a refused occurrence leaves the phrase in the text, the flag names their own whitelisted phrase. Without a sentence the reasonable conclusion is that the whitelist is broken, and the repair they reach for is adding more entries. States the rule as behaviour rather than mechanism, on both phrase lists and not the profanity one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(moderation): record the per-occurrence choice and the nsfw gate Two comments, no behaviour change. The strip judges each occurrence separately. That was a decision — refusing the whole text on one dirty occurrence hands back the false positives this PR removes, and stripping the whole text on one clean occurrence licenses the swallow elsewhere in it — but the code read as though the callback shape just happened to work that way. Also records the artefact: a refused occurrence leaves the phrase in place, so the flag names the moderator's own entry. And the nsfw word list is load-bearing for more than nsfw: at both sites the POI and minor checks run only if it matches, and callers passing no nsfw argument (every search gate) depend on it entirely. Nothing there said so, which is what made this PR's swallow a gate flip rather than one hidden term. Commented at both, not just the one that motivated it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(moderator): read the blocklist row writer-side, and report duplicates to Axiom The deterministic re-read used dbRead, a genuine replica, while both writers re-read through it and then cache the result for a MONTH under a key the main app also reads. A write-then-read races replication, so a moderator's edit could pin the PRE-EDIT row for 30 days — the exact failure the function exists to prevent, reintroduced through another door. Every other read in the file is already writer-side. Found by review. Duplicate-row report moves from console.error to logToAxiom with the same event name the main app uses, so one anomaly from two apps is one searchable thing, and it now carries the ignored rows' entry counts — the number that says whether the duplicate matters. Copy: drop a cross-reference to a rule on another tab that the reader cannot see, hoist the caveat shared by both phrase lists so two literals cannot drift, and say the tuple order is fixed rather than 'sorted', which reads as alphabetical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(moderation): the id-less fallback is a cross-app contract Both readBlocklistRow implementations return { type, data: [] } with no id for an absent row, and getClientBenignLists reads a missing id as 'no moderator row yet, use the bundled list'. That value travels between the two apps through a shared Redis key, so adding an id to the fallback — or caching a synthesised row — would silently switch every browser from the moderator's list to the bundled one with nothing failing. Said on both sides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Justin Maier
@@ -1,4 +1,5 @@
|
||||
// Sorted for stable tabs. (Blocklist.type is a plain string column — no shared DB enum to import.)
|
||||
// FIXED order — this is the tab order, not an alphabetical sort; changing it moves tabs under
|
||||
// people. (Blocklist.type is a plain string column — no shared DB enum to import.)
|
||||
export const BLOCKLIST_TYPES = [
|
||||
'EmailDomain',
|
||||
'LinkDomain',
|
||||
@@ -7,15 +8,26 @@ export const BLOCKLIST_TYPES = [
|
||||
'UsernamePartial',
|
||||
'PromptBenignPhrase',
|
||||
'NegativeBenignPhrase',
|
||||
'ProfanityBenignWord',
|
||||
] as const;
|
||||
|
||||
export type BlocklistType = (typeof BLOCKLIST_TYPES)[number];
|
||||
|
||||
export const humanizeBlocklistType = (t: string) => t.replace(/([a-z])([A-Z])/g, '$1 $2');
|
||||
|
||||
// Shared by both phrase lists — the same caveat in two literals drifts. Not on the profanity
|
||||
// list, which matches whole words and so has no gap.
|
||||
const PHRASE_GAP_RULE =
|
||||
' A phrase only counts as whitelisted when its words are separated by spacing or punctuation.' +
|
||||
' If anything else sits between them, the phrase is treated as not whitelisted for that' +
|
||||
' occurrence, deliberately, so text cannot be hidden inside a whitelisted phrase. When that' +
|
||||
' happens the flag you see may name the phrase itself.';
|
||||
|
||||
export const BLOCKLIST_DESCRIPTIONS: Partial<Record<BlocklistType, string>> = {
|
||||
PromptBenignPhrase:
|
||||
'Whole phrases in the positive prompt that innocently contain a minor/POI detection word (proper nouns, technical terms). Each phrase is blanked from the prompt before the scan audit runs, so it never false-flags an image for review. Enter the full phrase — e.g. "teen titans", "minor barrel distortion".',
|
||||
'Whole phrases in the positive prompt that innocently contain a minor/POI detection word (proper nouns, technical terms). Each phrase is blanked from the prompt before the scan audit runs, so it never false-flags an image for review. Enter the full phrase — e.g. "teen titans", "minor barrel distortion".' + PHRASE_GAP_RULE,
|
||||
NegativeBenignPhrase:
|
||||
'Same as Prompt Benign Phrase, but matched against the negative prompt — e.g. "mature content". Use for boilerplate negatives that trip the minor audit.',
|
||||
'Same as Prompt Benign Phrase, but matched against the negative prompt — e.g. "mature content". Use for boilerplate negatives that trip the minor audit.' + PHRASE_GAP_RULE,
|
||||
ProfanityBenignWord:
|
||||
'Single words that innocently contain a profanity token — "spreadsheet" contains "spread", "cockpit" contains "cock". The whole word is exempted from the profanity filter. One word per entry, not a phrase. This list REPLACES the one shipped with the site (it was seeded from it), so removing an entry here really does remove it. Applies to search; the generation gate still uses the shipped list.',
|
||||
};
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { REDIS_KEYS, type RedisKeyTemplateCache } from '@civitai/redis';
|
||||
import { dbRead, dbWrite } from './db';
|
||||
import { dbWrite } from './db';
|
||||
import { logToAxiom } from './axiom';
|
||||
import { getRedis } from './redis';
|
||||
|
||||
// Writes BOTH the Blocklist table AND the shared Redis cache (same key/shape/TTL the main app reads), so
|
||||
@@ -16,17 +17,62 @@ async function setCache(data: BlocklistDTO) {
|
||||
await getRedis().set(blocklistKey(data.type), JSON.stringify(data), { EX: MONTH_TTL });
|
||||
}
|
||||
|
||||
/**
|
||||
* Nothing stops a type having more than one row, and `EmailDomain` had two in production —
|
||||
* 8292 entries against 3, with the smaller row's domains present nowhere else. `limit(1)` with
|
||||
* no `orderBy` let Postgres decide which one was enforced, and the loser's entries were simply
|
||||
* not applied. Reads the lowest id, always, and reports the duplicate.
|
||||
*
|
||||
* Deliberately NOT a union: for a deny-list a union blocks more, but for the benign-phrase
|
||||
* lists a union strips more, which is a moderation bypass — one helper cannot pick a safe
|
||||
* direction for both. Mirrors `readBlocklistRow` in the main app's `blocklist.service.ts`;
|
||||
* both must agree, because they read and write the SAME Redis key.
|
||||
*/
|
||||
async function readBlocklistRow(type: string): Promise<BlocklistDTO> {
|
||||
// `dbWrite`, not `dbRead`, and this is load-bearing: both writers below re-read through here
|
||||
// and then cache the result for a MONTH. Off the replica, a write-then-read races replication
|
||||
// and would pin the PRE-EDIT row into a key the main app also reads — a moderator's change
|
||||
// silently undone for 30 days, which is the failure this function exists to prevent. The
|
||||
// writers already read writer-side for the same reason.
|
||||
const rows = await dbWrite
|
||||
.selectFrom('Blocklist')
|
||||
.select(['id', 'type', 'data'])
|
||||
.where('type', '=', type)
|
||||
.orderBy('id', 'asc')
|
||||
.execute();
|
||||
|
||||
if (rows.length > 1) {
|
||||
// Axiom rather than console: this is a standing data anomaly no moderator will ever see,
|
||||
// not a failure attached to an action in flight. Same `name` as the main app's report so
|
||||
// the same anomaly from either app is one searchable thing.
|
||||
void logToAxiom({
|
||||
name: 'blocklist-duplicate-rows',
|
||||
type: 'error',
|
||||
message:
|
||||
'More than one Blocklist row for a type; entries on the ignored rows are not enforced',
|
||||
details: {
|
||||
app: 'moderator',
|
||||
blocklistType: type,
|
||||
usedId: rows[0].id,
|
||||
ignoredIds: rows.slice(1).map((row) => row.id),
|
||||
ignoredEntryCounts: rows.slice(1).map((row) => row.data.length),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
// 🔴 The absent-row fallback carries NO `id`, and that is load-bearing across two apps: the
|
||||
// main app's `getClientBenignLists` reads `row.id == null` as "no moderator row yet, fall back
|
||||
// to the list shipped in the bundle". This value reaches it through the shared Redis key, so
|
||||
// adding an `id` here — or caching a synthesised row — would silently flip every browser from
|
||||
// the moderator's list to the bundled one, with nothing failing.
|
||||
return rows[0] ?? { type, data: [] };
|
||||
}
|
||||
|
||||
export async function getBlocklistDTO({ type }: { type: string }): Promise<BlocklistDTO> {
|
||||
const cached = await getRedis().get(blocklistKey(type));
|
||||
if (cached) return JSON.parse(cached) as BlocklistDTO;
|
||||
|
||||
const row = await dbRead
|
||||
.selectFrom('Blocklist')
|
||||
.select(['id', 'type', 'data'])
|
||||
.where('type', '=', type)
|
||||
.limit(1)
|
||||
.executeTakeFirst();
|
||||
const result: BlocklistDTO = row ?? { type, data: [] };
|
||||
const result = await readBlocklistRow(type);
|
||||
|
||||
await setCache(result);
|
||||
return result;
|
||||
@@ -65,7 +111,10 @@ export async function upsertBlocklist({
|
||||
.executeTakeFirstOrThrow();
|
||||
}
|
||||
|
||||
await setCache(result);
|
||||
// Cache the row that WINS the read, not the one just written. Caching `result` is what let an
|
||||
// edit to a duplicate row silently promote it to the live answer for the whole month TTL —
|
||||
// and this key is shared with the main app, so it would poison that read too.
|
||||
await setCache(await readBlocklistRow(result.type));
|
||||
}
|
||||
|
||||
export async function removeBlocklistItems({
|
||||
@@ -90,5 +139,5 @@ export async function removeBlocklistItems({
|
||||
.where('id', '=', id)
|
||||
.returning(['id', 'type', 'data'])
|
||||
.executeTakeFirstOrThrow();
|
||||
await setCache(updated);
|
||||
await setCache(await readBlocklistRow(updated.type));
|
||||
}
|
||||
|
||||
+15
File diff suppressed because one or more lines are too long
+5
@@ -0,0 +1,5 @@
|
||||
-- Drop the PromptAllowlist table. The runtime filter that was meant to read it was never
|
||||
-- enabled (`promptAuditing` hard-coded an empty set), the moderator mutation that wrote it
|
||||
-- had no UI, and the table holds 0 rows in production. Apply after the code deploy.
|
||||
|
||||
DROP TABLE IF EXISTS "PromptAllowlist";
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
-- One Blocklist row per type.
|
||||
--
|
||||
-- 🔴 ORDERING: this FAILS while any type still has more than one row. One type had a
|
||||
-- duplicate row in production; the rows were merged and the duplicate removed on 2026-08-19
|
||||
-- before this migration was written. Run this only against an environment where that merge
|
||||
-- has happened; on a fresh environment seeded from the migrations there is nothing to merge
|
||||
-- and it applies cleanly.
|
||||
--
|
||||
-- Why it matters rather than being tidiness: there are now TWO writers of the same type-scoped
|
||||
-- Redis key — the main app and the moderator spoke (`apps/moderator/src/lib/server/
|
||||
-- blocklist.service.ts`, whose own header notes it writes the key the main app reads). With a
|
||||
-- duplicate row present, the two can pick different rows and each overwrite the other's cached
|
||||
-- answer for a month. Both now read deterministically and log the duplicate, but a code guard
|
||||
-- on each side is a convention two teams have to keep; this constraint is what actually makes
|
||||
-- the shared cache coherent, by making the state unrepresentable.
|
||||
--
|
||||
-- Check before applying, per environment:
|
||||
-- SELECT "type", count(*) FROM "Blocklist" GROUP BY "type" HAVING count(*) > 1;
|
||||
-- Expect zero rows. If it returns any, merge them first — do not drop a row to make this
|
||||
-- pass, since the loser's entries are exactly what was being missed.
|
||||
--
|
||||
-- Idempotent: re-running is a no-op once the index exists.
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS "Blocklist_type_key" ON "Blocklist" ("type");
|
||||
@@ -7369,19 +7369,6 @@ enum UserRestrictionStatus {
|
||||
}
|
||||
|
||||
// Moderator-curated allowlist for false-positive prompt triggers
|
||||
model PromptAllowlist {
|
||||
id Int @id @default(autoincrement())
|
||||
trigger String
|
||||
category String
|
||||
addedBy Int
|
||||
reason String?
|
||||
userRestrictionId Int?
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
@@unique([trigger, category])
|
||||
@@index([category])
|
||||
}
|
||||
|
||||
// Strike system — graduated user enforcement with points-based escalation
|
||||
enum StrikeReason {
|
||||
BlockedContent
|
||||
|
||||
@@ -3363,15 +3363,6 @@ export type Product = {
|
||||
defaultPriceId: string | null;
|
||||
provider: Generated<PaymentProvider>;
|
||||
};
|
||||
export type PromptAllowlist = {
|
||||
id: Generated<number>;
|
||||
trigger: string;
|
||||
category: string;
|
||||
addedBy: number;
|
||||
reason: string | null;
|
||||
userRestrictionId: number | null;
|
||||
createdAt: Generated<Timestamp>;
|
||||
};
|
||||
export type PurchasableReward = {
|
||||
id: Generated<number>;
|
||||
createdAt: Generated<Timestamp>;
|
||||
@@ -4460,7 +4451,6 @@ export type DB = {
|
||||
PressMention: PressMention;
|
||||
Price: Price;
|
||||
Product: Product;
|
||||
PromptAllowlist: PromptAllowlist;
|
||||
PurchasableReward: PurchasableReward;
|
||||
Purchase: Purchase;
|
||||
Question: Question;
|
||||
|
||||
@@ -5013,16 +5013,6 @@ export interface UserRestriction {
|
||||
userMessageAt: Date | null;
|
||||
}
|
||||
|
||||
export interface PromptAllowlist {
|
||||
id: number;
|
||||
trigger: string;
|
||||
category: string;
|
||||
addedBy: number;
|
||||
reason: string | null;
|
||||
userRestrictionId: number | null;
|
||||
createdAt: Date;
|
||||
}
|
||||
|
||||
export interface UserStrike {
|
||||
id: number;
|
||||
userId: number;
|
||||
|
||||
@@ -4,6 +4,7 @@ import handler from '~/pages/api/webhooks/image-scan-result';
|
||||
import type * as ClickhouseClient from '~/server/clickhouse/client';
|
||||
import { TagSource, ImageIngestionStatus } from '~/shared/utils/prisma/enums';
|
||||
import { NsfwLevel } from '~/server/common/enums';
|
||||
import type * as BlocklistService from '~/server/services/blocklist.service';
|
||||
|
||||
const {
|
||||
mockDbWrite,
|
||||
@@ -124,6 +125,15 @@ const {
|
||||
};
|
||||
});
|
||||
|
||||
// Partial mock: the real `stripBenignPhrases` reads Redis/Postgres, which no other
|
||||
// test in this file needs. Default is passthrough so every existing case is
|
||||
// unaffected; the benign-phrase cases below drive it directly.
|
||||
const mockStripBenignPhrases = vi.fn(async (text?: string) => text ?? '');
|
||||
vi.mock('~/server/services/blocklist.service', async (importOriginal) => ({
|
||||
...(await importOriginal<typeof BlocklistService>()),
|
||||
stripBenignPhrases: (text?: string, type?: unknown) => mockStripBenignPhrases(text, type),
|
||||
}));
|
||||
|
||||
vi.mock('~/server/db/client', () => ({
|
||||
dbWrite: mockDbWrite,
|
||||
dbRead: mockDbWrite,
|
||||
@@ -683,4 +693,68 @@ describe('image-scan-result webhook - pipeline tests', () => {
|
||||
expect(update!.text).not.toContain('4611686018427387904');
|
||||
});
|
||||
});
|
||||
|
||||
describe('moderator benign phrases reach the POI tagger', () => {
|
||||
// The file's own `beforeEach` is `vi.clearAllMocks()`, which clears calls but NOT
|
||||
// implementations — so the one a test below installs would leak into anything appended
|
||||
// after this describe. Harmless while this is the last block, which is exactly the kind
|
||||
// of "harmless" that stops being true without anyone noticing.
|
||||
beforeEach(() => {
|
||||
mockStripBenignPhrases.mockImplementation(async (text?: string) => text ?? '');
|
||||
});
|
||||
|
||||
const seedImageWithPrompt = (id: number, prompt: string) => {
|
||||
imageDbState.set(id, {
|
||||
id,
|
||||
createdAt: new Date(),
|
||||
scannedAt: null,
|
||||
type: 'image',
|
||||
userId: 1,
|
||||
meta: { prompt },
|
||||
metadata: {},
|
||||
postId: null,
|
||||
nsfwLevelLocked: false,
|
||||
nsfwLevel: null,
|
||||
scanJobs: { scans: {} },
|
||||
ingestion: ImageIngestionStatus.Pending,
|
||||
});
|
||||
};
|
||||
|
||||
const requestedTagNames = () =>
|
||||
mockDbWrite.tag.findMany.mock.calls.flatMap((call: any) => call[0]?.where?.name?.in ?? []);
|
||||
|
||||
it('CONTROL: a POI name that is not whitelisted is still tagged', async () => {
|
||||
seedImageWithPrompt(30, 'emma stone');
|
||||
|
||||
const req = runWebhook({ id: 30, status: 0, source: TagSource.WD14, tags: [] });
|
||||
await req.promise;
|
||||
|
||||
// Deliberately asserts only the tag, with the strip passing the text through: it
|
||||
// stays green with the fix reverted, so a failure below is the strip and not a
|
||||
// broken fixture.
|
||||
expect(requestedTagNames()).toContain('emma stone');
|
||||
});
|
||||
|
||||
it('a whitelisted phrase is stripped before the POI check, so no POI tag is written', async () => {
|
||||
mockStripBenignPhrases.mockImplementation(async (text?: string) =>
|
||||
(text ?? '').replace('emma stone', '')
|
||||
);
|
||||
// Two real POI names, only one whitelisted. `tom hanks` survives the strip and must
|
||||
// still be tagged, which proves this run reached the tag write at all — otherwise an
|
||||
// early throw would satisfy the negative assertion below with an empty array.
|
||||
seedImageWithPrompt(31, 'emma stone and tom hanks');
|
||||
|
||||
const req = runWebhook({ id: 31, status: 0, source: TagSource.WD14, tags: [] });
|
||||
await req.promise;
|
||||
|
||||
// Pins WHICH list is consulted. Pointing the strip at ProfanityBenignWord instead
|
||||
// leaves the whole fix inert in production and every other assertion here green.
|
||||
expect(mockStripBenignPhrases).toHaveBeenCalledWith(
|
||||
'emma stone and tom hanks',
|
||||
'PromptBenignPhrase'
|
||||
);
|
||||
expect(requestedTagNames()).not.toContain('emma stone');
|
||||
expect(requestedTagNames()).toContain('tom hanks');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -68,6 +68,7 @@ import { truncate } from 'lodash-es';
|
||||
import { usePathname } from 'next/navigation';
|
||||
import { useDomainColor } from '~/hooks/useDomainColor';
|
||||
import { useCheckProfanity } from '~/hooks/useCheckProfanity';
|
||||
import { useBenignPhrases } from '~/hooks/useBenignPhrases';
|
||||
|
||||
const meilisearch = instantMeiliSearch(
|
||||
env.NEXT_PUBLIC_SEARCH_HOST as string,
|
||||
@@ -241,11 +242,18 @@ function AutocompleteSearchContentInner<TKey extends SearchIndexKey>(
|
||||
});
|
||||
|
||||
// Check for illegal search first
|
||||
const benignPhrases = useBenignPhrases();
|
||||
// Detection reads the whitelisted copy; the query sent to Meili stays the raw text.
|
||||
const auditedSearch = useMemo(
|
||||
() => benignPhrases.strip(debouncedSearch),
|
||||
[benignPhrases, debouncedSearch]
|
||||
);
|
||||
|
||||
const isIllegalSearch = useMemo(() => {
|
||||
if (!debouncedSearch) return false;
|
||||
const illegalSearch = includesInappropriate({ prompt: debouncedSearch });
|
||||
const illegalSearch = includesInappropriate({ prompt: auditedSearch });
|
||||
return illegalSearch === 'minor';
|
||||
}, [debouncedSearch]);
|
||||
}, [debouncedSearch, auditedSearch]);
|
||||
|
||||
// Check profanity in search query (only if not illegal and domain is green)
|
||||
const profanityAnalysis = useCheckProfanity(debouncedSearch, {
|
||||
@@ -256,11 +264,13 @@ function AutocompleteSearchContentInner<TKey extends SearchIndexKey>(
|
||||
|
||||
const items = useMemo(() => {
|
||||
const isIllegalQuery = debouncedSearch
|
||||
? includesInappropriate({ prompt: debouncedSearch }) === 'minor'
|
||||
? includesInappropriate({ prompt: auditedSearch }) === 'minor'
|
||||
: false;
|
||||
const canPerformQuery = debouncedSearch
|
||||
? !browsingSettingsAddons.settings.disablePoi || !includesPoi(debouncedSearch)
|
||||
? !browsingSettingsAddons.settings.disablePoi || !includesPoi(auditedSearch)
|
||||
: true;
|
||||
// Raw text on purpose: this reads a different list (blocked NSFW words) which the benign
|
||||
// phrases do not carve out, and it blocks rather than flags.
|
||||
const hasBlockedWords = !!getBlockedNsfwWords(debouncedSearch).length;
|
||||
|
||||
if (isIllegalQuery) {
|
||||
@@ -584,8 +594,8 @@ function AutocompleteSearchContentInner<TKey extends SearchIndexKey>(
|
||||
return (
|
||||
<Stack gap="xs" align="center">
|
||||
<Text size="sm" align="center">
|
||||
Your search includes terms tied to real people. Content depicting real people
|
||||
is filtered from search results.
|
||||
Your search includes terms tied to real people. Content depicting real people is
|
||||
filtered from search results.
|
||||
</Text>
|
||||
</Stack>
|
||||
);
|
||||
|
||||
@@ -9,7 +9,7 @@ import { DEFAULT_ALLOWED_ATTRIBUTES, sanitizeHtml } from '~/utils/html-sanitize-
|
||||
import classes from './RenderHtml.module.scss';
|
||||
import { TypographyStylesWrapper } from '~/components/TypographyStylesWrapper/TypographyStylesWrapper';
|
||||
import clsx from 'clsx';
|
||||
import { createProfanityFilter } from '~/libs/profanity-simple';
|
||||
import { getProfanityFilter } from '~/libs/profanity-simple';
|
||||
import { useBrowsingSettings } from '~/providers/BrowserSettingsProvider';
|
||||
import { useStickerCosmetics } from '~/components/Sticker/sticker.util';
|
||||
import { STICKER_JUMBO_LIMIT, stickerMaxWidth, STICKER_SIZE } from '~/shared/utils/sticker-token';
|
||||
@@ -54,7 +54,10 @@ export function RenderHtml({
|
||||
html = useMemo(() => {
|
||||
let processedHtml = html;
|
||||
if (withProfanityFilter && blurNsfw) {
|
||||
const profanityFilter = createProfanityFilter();
|
||||
// Shared cache, not a fresh filter: this runs per COMPONENT INSTANCE, so a 20-comment
|
||||
// thread was building 20 full obscenity datasets (~6ms each). Pre-existing, cheap to
|
||||
// fix now that the keyed factory exists.
|
||||
const profanityFilter = getProfanityFilter();
|
||||
|
||||
// Preserve mentions (entire span + content) and all HTML tag markup so
|
||||
// the filter only operates on visible text content. Capturing entire
|
||||
|
||||
@@ -42,6 +42,7 @@ import type { UiState } from 'instantsearch.js';
|
||||
import { includesInappropriate } from '~/utils/metadata/audit';
|
||||
import { useDomainColor } from '~/hooks/useDomainColor';
|
||||
import { useCheckProfanity } from '~/hooks/useCheckProfanity';
|
||||
import { useBenignPhrases } from '~/hooks/useBenignPhrases';
|
||||
import classes from './SearchLayout.module.scss';
|
||||
import clsx from 'clsx';
|
||||
|
||||
@@ -154,16 +155,24 @@ export function SearchLayout({
|
||||
setInstantSearchQuery(query);
|
||||
}, []);
|
||||
|
||||
const benignPhrases = useBenignPhrases();
|
||||
|
||||
const isIllegalSearch = useMemo(() => {
|
||||
if (!searchQuery) return false;
|
||||
|
||||
const illegalSearch = includesInappropriate({ prompt: searchQuery });
|
||||
// Detection reads the whitelisted copy; the query sent to Meili stays the raw text.
|
||||
const illegalSearch = includesInappropriate({ prompt: benignPhrases.strip(searchQuery) });
|
||||
return illegalSearch === 'minor';
|
||||
}, [searchQuery]);
|
||||
}, [searchQuery, benignPhrases]);
|
||||
|
||||
// Track illegal search separately to avoid side effects in useMemo
|
||||
useEffect(() => {
|
||||
if (!searchQuery || !isIllegalSearch) return;
|
||||
// Hold the event until the benign lists arrive. Before then `strip` is the identity
|
||||
// function, so a moderator-whitelisted query still reads as illegal here — and firing a
|
||||
// CSAM tracking event for it is exactly the false positive this work exists to remove.
|
||||
// The gate itself still applies while loading; only the side effect waits.
|
||||
if (!benignPhrases.settled) return;
|
||||
|
||||
const { sortBy: index } = parsedQuery || {};
|
||||
trackAction({
|
||||
@@ -173,7 +182,7 @@ export function SearchLayout({
|
||||
index,
|
||||
},
|
||||
}).catch(() => undefined);
|
||||
}, [searchQuery, isIllegalSearch, parsedQuery?.sortBy]);
|
||||
}, [searchQuery, isIllegalSearch, parsedQuery?.sortBy, benignPhrases.settled]);
|
||||
|
||||
// Check profanity in search query
|
||||
const profanityAnalysis = useCheckProfanity(searchQuery, {
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
// @vitest-environment happy-dom
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import * as React from 'react';
|
||||
import type { act as actType } from 'react-dom/test-utils';
|
||||
import { createRoot } from 'react-dom/client';
|
||||
import type * as Trpc from '~/utils/trpc';
|
||||
|
||||
const act = (React as unknown as { act: typeof actType }).act;
|
||||
|
||||
type QueryResult = {
|
||||
data?: { prompt: string[]; profanityWords: string[] | null; available: boolean };
|
||||
isFetched: boolean;
|
||||
};
|
||||
|
||||
let queryResult: QueryResult = { data: undefined, isFetched: false };
|
||||
vi.mock('~/utils/trpc', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof Trpc>();
|
||||
return {
|
||||
...actual,
|
||||
trpc: {
|
||||
...actual.trpc,
|
||||
system: {
|
||||
...actual.trpc.system,
|
||||
getBenignPhrases: { useQuery: () => queryResult },
|
||||
},
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
import { useBenignPhrases } from '~/hooks/useBenignPhrases';
|
||||
|
||||
type Hook = ReturnType<typeof useBenignPhrases>;
|
||||
|
||||
function renderHook(): Hook {
|
||||
let captured: Hook | undefined;
|
||||
function Probe() {
|
||||
captured = useBenignPhrases();
|
||||
return null;
|
||||
}
|
||||
const container = document.createElement('div');
|
||||
act(() => {
|
||||
createRoot(container).render(React.createElement(Probe));
|
||||
});
|
||||
if (!captured) throw new Error('hook did not render');
|
||||
return captured;
|
||||
}
|
||||
|
||||
describe('useBenignPhrases', () => {
|
||||
beforeEach(() => {
|
||||
queryResult = { data: undefined, isFetched: false };
|
||||
});
|
||||
|
||||
describe('the window before the lists arrive', () => {
|
||||
it('strips nothing, so the gates judge the raw query and flag MORE', () => {
|
||||
const { strip } = renderHook();
|
||||
expect(strip('emma stone portrait')).toBe('emma stone portrait');
|
||||
});
|
||||
|
||||
it('is not settled, so a caller with a side effect can hold it', () => {
|
||||
expect(renderHook().settled).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('once the lists arrive', () => {
|
||||
beforeEach(() => {
|
||||
queryResult = {
|
||||
data: { prompt: ['emma stone'], profanityWords: ['spreadsheet'], available: true },
|
||||
isFetched: true,
|
||||
};
|
||||
});
|
||||
|
||||
it('strips the whitelisted phrase and leaves the rest', () => {
|
||||
expect(renderHook().strip('emma stone portrait').trim()).toBe('portrait');
|
||||
});
|
||||
|
||||
it('reports settled and passes the moderator word list through', () => {
|
||||
const hook = renderHook();
|
||||
expect(hook.settled).toBe(true);
|
||||
expect(hook.profanityWords).toEqual(['spreadsheet']);
|
||||
});
|
||||
});
|
||||
|
||||
// `null` (no moderator row) and `[]` (a moderator deleted every entry) are different states,
|
||||
// and the profanity filter treats them differently — `null` falls back to the ~450 words
|
||||
// shipped in the bundle, `[]` honours the emptying. A `?? []` here would re-read every
|
||||
// ordinary "we don't know yet" as a deliberate emptying and drop the shipped list on a
|
||||
// normal page load, on a failed fetch, and before the seed migration is applied.
|
||||
describe('unknown is not the same as empty', () => {
|
||||
it('reports null before the query resolves, not an empty list', () => {
|
||||
expect(renderHook().profanityWords).toBeNull();
|
||||
});
|
||||
|
||||
it('reports null when the server could not read the lists', () => {
|
||||
queryResult = {
|
||||
data: { prompt: [], profanityWords: null, available: false },
|
||||
isFetched: true,
|
||||
};
|
||||
expect(renderHook().profanityWords).toBeNull();
|
||||
});
|
||||
|
||||
it('preserves an EMPTY moderator list as empty', () => {
|
||||
queryResult = {
|
||||
data: { prompt: [], profanityWords: [], available: true },
|
||||
isFetched: true,
|
||||
};
|
||||
expect(renderHook().profanityWords).toEqual([]);
|
||||
});
|
||||
|
||||
it('settles even when the fetch failed, so a held side effect is not suppressed forever', () => {
|
||||
queryResult = { data: undefined, isFetched: true };
|
||||
expect(renderHook().settled).toBe(true);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,50 @@
|
||||
import { useMemo } from 'react';
|
||||
import { buildBenignPhraseRegex, stripBenignPhrasesWith } from '~/shared/utils/benign-phrases';
|
||||
import { trpc } from '~/utils/trpc';
|
||||
|
||||
/**
|
||||
* The moderator benign lists, for gates that run in the browser. Search queries Meili
|
||||
* from the client, so the POI / minor / profanity checks there have no server hop to
|
||||
* strip on — see `getClientBenignLists`. Edge-cached for an hour server-side and refetched
|
||||
* hourly per session, so both an addition and a REMOVAL reach an open search box in about
|
||||
* that.
|
||||
*/
|
||||
export function useBenignPhrases() {
|
||||
// Finite staleTime, deliberately. The reason a moderator DELETES a benign phrase is that it
|
||||
// stripped too much — under `staleTime: Infinity` that deletion never reaches an open tab,
|
||||
// so a revocation had no propagation bound at all while an addition was capped at the
|
||||
// edge's hour.
|
||||
const { data, isFetched } = trpc.system.getBenignPhrases.useQuery(undefined, {
|
||||
staleTime: 60 * 60 * 1000,
|
||||
gcTime: Infinity,
|
||||
});
|
||||
|
||||
const promptPattern = useMemo(() => buildBenignPhraseRegex(data?.prompt ?? []), [data?.prompt]);
|
||||
const profanityWords = data?.profanityWords;
|
||||
|
||||
return useMemo(
|
||||
() => ({
|
||||
// `?? null`, NOT `?? []`. The two mean different things downstream: `null` is "no
|
||||
// moderator row, use the list shipped in the bundle", `[]` is "a moderator deleted every
|
||||
// entry, honour that". Collapsing them here would re-read three ordinary states — before
|
||||
// the query resolves, after a failed fetch, and before the seed migration is applied —
|
||||
// as a deliberate emptying, dropping the ~450 shipped words on every one of them.
|
||||
profanityWords: profanityWords ?? null,
|
||||
/**
|
||||
* Whether the fetch has SETTLED — not whether it succeeded. Until it settles, `strip` is
|
||||
* the identity function and the gates judge the RAW query; that direction is safe (they
|
||||
* flag more, not less), but a caller with a side effect — a tracking event, a report —
|
||||
* should wait, since firing one for a phrase a moderator whitelisted is the thing this
|
||||
* exists to stop.
|
||||
*
|
||||
* On `isSuccess` this would stay false forever after a failed fetch, silently
|
||||
* suppressing that side effect for the life of the session. A failed fetch means there
|
||||
* IS no whitelist, which is exactly when there is no reason to withhold.
|
||||
*/
|
||||
settled: isFetched,
|
||||
/** Blank whitelisted phrases before a detection check — never for display or query. */
|
||||
strip: (text: string | undefined) => stripBenignPhrasesWith(text, promptPattern),
|
||||
}),
|
||||
[promptPattern, profanityWords, isFetched]
|
||||
);
|
||||
}
|
||||
@@ -1,11 +1,6 @@
|
||||
import { useMemo } from 'react';
|
||||
import { createProfanityFilter, type ProfanityFilterOptions } from '~/libs/profanity-simple';
|
||||
|
||||
// Building the filter allocates the whole English dataset + matcher, so it's
|
||||
// shared across every consumer — but stays lazy so pages that never check
|
||||
// profanity don't pay for it.
|
||||
let sharedProfanityFilter: ReturnType<typeof createProfanityFilter> | undefined;
|
||||
const getProfanityFilter = () => (sharedProfanityFilter ??= createProfanityFilter());
|
||||
import { getProfanityFilter, type ProfanityFilterOptions } from '~/libs/profanity-simple';
|
||||
import { useBenignPhrases } from '~/hooks/useBenignPhrases';
|
||||
|
||||
export interface UseCheckProfanityOptions extends Partial<ProfanityFilterOptions> {
|
||||
/** Whether to enable profanity checking. When false, returns clean results */
|
||||
@@ -53,11 +48,15 @@ export function useCheckProfanity(
|
||||
): ProfanityAnalysis {
|
||||
const { enabled = true } = options;
|
||||
|
||||
const profanityFilter = useMemo(getProfanityFilter, []);
|
||||
const { profanityWords } = useBenignPhrases();
|
||||
|
||||
// Analyze the text
|
||||
const analysis = useMemo((): ProfanityAnalysis => {
|
||||
// Return clean results if disabled or if global blur is off
|
||||
// Return clean results if disabled or if global blur is off. The filter is built INSIDE
|
||||
// this branch rather than in its own memo: constructing one allocates the whole obscenity
|
||||
// dataset (measured ~6-9ms, several times that on mobile), and `AutocompleteSearch` mounts
|
||||
// in the header of every page, so building eagerly charged that to every page load
|
||||
// including the ones where nobody types.
|
||||
if (!enabled || !text.trim()) {
|
||||
return {
|
||||
hasProfanity: false,
|
||||
@@ -70,7 +69,7 @@ export function useCheckProfanity(
|
||||
}
|
||||
|
||||
try {
|
||||
// Get detailed analysis from the profanity filter
|
||||
const profanityFilter = getProfanityFilter(profanityWords);
|
||||
const detailedAnalysis = profanityFilter.analyze(text);
|
||||
const cleanedText = profanityFilter.clean(text);
|
||||
|
||||
@@ -94,7 +93,7 @@ export function useCheckProfanity(
|
||||
originalText: text,
|
||||
};
|
||||
}
|
||||
}, [text, enabled, profanityFilter]);
|
||||
}, [text, enabled, profanityWords]);
|
||||
|
||||
return analysis;
|
||||
}
|
||||
|
||||
@@ -33,6 +33,20 @@ import { constants } from '~/server/common/constants';
|
||||
export interface ProfanityFilterOptions {
|
||||
/** How to replace profane words */
|
||||
replacementStyle: 'asterisk' | 'grawlix' | 'remove';
|
||||
/**
|
||||
* The moderator-managed benign words (`BlocklistType.ProfanityBenignWord`). It REPLACES the
|
||||
* static `whitelist-words.json` rather than adding to it, so a moderator can remove an
|
||||
* entry and have that take effect — a union would make the Remove control in the blocklist
|
||||
* UI a silent no-op for every word the static list also carries, which is all of them,
|
||||
* since the seed migration copies it verbatim.
|
||||
*
|
||||
* `null` means there is no moderator row (not migrated, or unreachable) and falls back to
|
||||
* the static list, so that case degrades to today's behaviour rather than to no whitelist.
|
||||
* An EMPTY ARRAY is different and is honoured as-is: it means a moderator deleted every
|
||||
* entry, which is the strongest possible "do not whitelist" intent, and quietly restoring
|
||||
* ~450 shipped words over the top of it would be the opposite of what they asked for.
|
||||
*/
|
||||
moderatorWhitelist: string[] | null;
|
||||
}
|
||||
|
||||
export interface ProfanityThresholdConfig {
|
||||
@@ -112,9 +126,12 @@ export class SimpleProfanityFilter {
|
||||
constructor(options: Partial<ProfanityFilterOptions> = {}) {
|
||||
this.options = {
|
||||
replacementStyle: 'asterisk',
|
||||
moderatorWhitelist: null,
|
||||
...options,
|
||||
};
|
||||
|
||||
const whitelist = this.options.moderatorWhitelist ?? whitelistWords;
|
||||
|
||||
// Cache NSFW words once during construction
|
||||
this.nsfwWords = getCachedNsfwWords();
|
||||
|
||||
@@ -123,10 +140,17 @@ export class SimpleProfanityFilter {
|
||||
this.dataset.addAll(englishDataset);
|
||||
|
||||
// Initialize whitelist mappings
|
||||
this.whitelistMappings = createWhitelistMappings(this.nsfwWords.originalWords, whitelistWords);
|
||||
this.whitelistMappings = createWhitelistMappings(this.nsfwWords.originalWords, whitelist);
|
||||
|
||||
// Initialize whitelist Set for O(1) lookup during analysis
|
||||
this.whitelistSet = new Set(whitelistWords.map((word) => word.toLowerCase()));
|
||||
// Initialize whitelist Set for O(1) lookup during analysis. An entry equal to a profane
|
||||
// word itself is dropped: `createWhitelistMappings` already refuses that case, but this
|
||||
// set is what `analyze()` filters against, so without the same guard a single-token entry
|
||||
// would disarm the filter for that exact word. Harmless while the list was a checked-in
|
||||
// JSON file; this is the change that turns it into a text box.
|
||||
const profaneWords = new Set(this.nsfwWords.originalWords.map((word) => word.toLowerCase()));
|
||||
this.whitelistSet = new Set(
|
||||
whitelist.map((word) => word.toLowerCase()).filter((word) => !profaneWords.has(word))
|
||||
);
|
||||
|
||||
this.initializeMatcher();
|
||||
this.initializeCensor();
|
||||
@@ -454,3 +478,30 @@ export function createProfanityFilter(
|
||||
): SimpleProfanityFilter {
|
||||
return new SimpleProfanityFilter(options);
|
||||
}
|
||||
|
||||
// Constructing a filter builds the whole English dataset + matcher, so callers share one per
|
||||
// distinct whitelist. Keyed rather than a single instance because the moderator list can
|
||||
// change; the common case (one list for the whole session) still allocates once. Lives here
|
||||
// rather than in the hook that needed it first, so server callers can reach it too.
|
||||
// Bounded: each matcher measures ~132 KB, and the list is refetched hourly, so a long-lived
|
||||
// tab DOES see new keys as moderators edit — the cap is load-bearing, not defensive. A tab
|
||||
// normally holds the pre-load (static) filter and the current one.
|
||||
const MAX_CACHED_FILTERS = 2;
|
||||
const filtersByWhitelist = new Map<string, SimpleProfanityFilter>();
|
||||
export function getProfanityFilter(
|
||||
moderatorWhitelist: string[] | null = null
|
||||
): SimpleProfanityFilter {
|
||||
// `null` (no row) and `[]` (emptied by a moderator) mean different things, so they must not
|
||||
// collide on the same cache key.
|
||||
const key = moderatorWhitelist === null ? 'no-moderator-row' : `list:${moderatorWhitelist}`;
|
||||
let filter = filtersByWhitelist.get(key);
|
||||
if (!filter) {
|
||||
filter = createProfanityFilter({ moderatorWhitelist });
|
||||
if (filtersByWhitelist.size >= MAX_CACHED_FILTERS) {
|
||||
const oldest = filtersByWhitelist.keys().next().value;
|
||||
if (oldest !== undefined) filtersByWhitelist.delete(oldest);
|
||||
}
|
||||
filtersByWhitelist.set(key, filter);
|
||||
}
|
||||
return filter;
|
||||
}
|
||||
|
||||
@@ -632,8 +632,15 @@ async function getTagsFromIncomingTags({
|
||||
const imageMeta = image.meta as Prisma.JsonObject | undefined;
|
||||
const prompt = imageMeta?.prompt as string | undefined;
|
||||
if (prompt) {
|
||||
// Detect real person in prompt
|
||||
const realPersonName = includesPoi(prompt);
|
||||
// Detect real person in prompt. Same moderator-managed benign phrases the audit
|
||||
// path strips (`auditImageScanResults`) — without this a whitelisted proper noun
|
||||
// is still written as a confidence-100 tag, which reaches the search index.
|
||||
const realPersonName = includesPoi(
|
||||
// Normalized first, as both audit paths do. Stripping the raw text while the audit that
|
||||
// runs next reads the normalized copy means two different alphabets decide what counts
|
||||
// as whitelisted for the same prompt.
|
||||
await stripBenignPhrases(normalizeText(prompt), BlocklistType.PromptBenignPhrase)
|
||||
);
|
||||
if (realPersonName) {
|
||||
const tagName =
|
||||
typeof realPersonName === 'object' ? realPersonName.matchedText : realPersonName;
|
||||
|
||||
@@ -380,6 +380,10 @@ export enum BlocklistType {
|
||||
// false-flag `needsReview='minor'`. Edited by moderators at /moderator/blocklists.
|
||||
PromptBenignPhrase = 'PromptBenignPhrase',
|
||||
NegativeBenignPhrase = 'NegativeBenignPhrase',
|
||||
// Single words that innocently CONTAIN a profanity token ("spreadsheet" holds
|
||||
// "spread"). Suppresses a substring match from the profanity filter rather than
|
||||
// being blanked from the text — different matcher to the two phrase lists above.
|
||||
ProfanityBenignWord = 'ProfanityBenignWord',
|
||||
}
|
||||
|
||||
export enum ToolSort {
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
import { edgeCacheIt } from '~/server/middleware.trpc';
|
||||
import { CacheTTL } from '~/server/common/constants';
|
||||
import { dbKV } from '~/server/db/db-helpers';
|
||||
import { getClientBenignLists } from '~/server/services/blocklist.service';
|
||||
import { TokenScope } from '~/shared/constants/token-scope.constants';
|
||||
|
||||
/**
|
||||
@@ -35,6 +36,28 @@ export const systemRouter = router({
|
||||
.meta({ requiredScope: TokenScope.Full })
|
||||
.use(edgeCacheIt({ ttl: CacheTTL.hour }))
|
||||
.query(() => getCreationBlockedTags()),
|
||||
// Moderator benign lists, shipped to the browser because the search gates run
|
||||
// client-side against Meili and have no server hop to strip on.
|
||||
getBenignPhrases: publicProcedure
|
||||
.meta({ requiredScope: TokenScope.Full })
|
||||
.use(edgeCacheIt({ ttl: CacheTTL.hour }))
|
||||
.query(async ({ ctx }) => {
|
||||
const lists = await getClientBenignLists();
|
||||
// A fail-open result is a 200 the edge would otherwise hold for an hour, pinning empty
|
||||
// whitelists site-wide off one transient error. `ctx.cache.skip` does NOT work from
|
||||
// here: `edgeCacheIt` reads it to compute the TTL BEFORE it calls this resolver, so it
|
||||
// is only usable by something upstream of the procedure. `canCache` is read after.
|
||||
//
|
||||
// All three, not just `canCache`: with `canCache: false` the middleware skips the block
|
||||
// that assigns the TTLs, so they keep the context defaults — which for an ANONYMOUS
|
||||
// caller are 60, not 0, and `s-maxage=60` would still go out.
|
||||
if (!lists.available && ctx.cache) {
|
||||
ctx.cache.canCache = false;
|
||||
ctx.cache.edgeTTL = 0;
|
||||
ctx.cache.browserTTL = 0;
|
||||
}
|
||||
return lists;
|
||||
}),
|
||||
getDbKV: publicProcedure
|
||||
.meta({ requiredScope: TokenScope.Full })
|
||||
.input(z.object({ key: z.enum(PUBLIC_DB_KV_KEYS) }))
|
||||
|
||||
@@ -3,14 +3,16 @@ import { dbRead, dbWrite } from '~/server/db/client';
|
||||
import { logToAxiom } from '~/server/logging/client';
|
||||
import { REDIS_SYS_KEYS, sysRedis } from '~/server/redis/client';
|
||||
import {
|
||||
addToAllowlistSchema,
|
||||
backfillRestrictionTriggersSchema,
|
||||
debugAuditPromptSchema,
|
||||
getGenerationRestrictionsSchema,
|
||||
resolveRestrictionSchema,
|
||||
saveSuspiciousMatchSchema,
|
||||
submitRestrictionContextSchema,
|
||||
} from '~/server/schema/user-restriction.schema';
|
||||
import type { BlockedPromptEntry } from '~/server/services/orchestrator/promptAuditing';
|
||||
import { debugAuditPrompt, type DebugAuditMatch } from '~/utils/metadata/audit';
|
||||
import { bustPromptAllowlistCache } from '~/server/services/orchestrator/promptAuditing';
|
||||
import { resolveUserRestriction } from '~/server/services/user-restriction-resolve.service';
|
||||
import { moderatorProcedure, protectedProcedure, router } from '~/server/trpc';
|
||||
import { TokenScope } from '~/shared/constants/token-scope.constants';
|
||||
|
||||
@@ -40,39 +42,52 @@ export const userRestrictionRouter = router({
|
||||
// }),
|
||||
// --- Moderator endpoints ---
|
||||
|
||||
/** Moderator adds a trigger to the prompt allowlist (marks as benign). */
|
||||
addToAllowlist: moderatorProcedure
|
||||
.input(addToAllowlistSchema)
|
||||
.mutation(async ({ ctx, input }) => {
|
||||
const { trigger, category, reason, userRestrictionId } = input;
|
||||
const moderatorId = ctx.user.id;
|
||||
/** Paginated list of generation restrictions for moderator review. */
|
||||
getAll: moderatorProcedure.input(getGenerationRestrictionsSchema).query(async ({ input }) => {
|
||||
const { limit, page, status, username, userId } = input;
|
||||
const offset = (page - 1) * limit;
|
||||
|
||||
await dbWrite.promptAllowlist.upsert({
|
||||
where: { trigger_category: { trigger, category } },
|
||||
create: {
|
||||
trigger,
|
||||
category,
|
||||
addedBy: moderatorId,
|
||||
reason,
|
||||
userRestrictionId,
|
||||
const where: NonNullable<Parameters<typeof dbRead.userRestriction.findMany>[0]>['where'] = {
|
||||
type: 'generation',
|
||||
...(status && { status }),
|
||||
...(userId && { userId }),
|
||||
user: {
|
||||
deletedAt: null,
|
||||
...(username && { username: { contains: username, mode: 'insensitive' as const } }),
|
||||
},
|
||||
};
|
||||
|
||||
const [items, totalCount] = await Promise.all([
|
||||
dbRead.userRestriction.findMany({
|
||||
where,
|
||||
orderBy: { createdAt: 'desc' },
|
||||
skip: offset,
|
||||
take: limit,
|
||||
select: {
|
||||
id: true,
|
||||
userId: true,
|
||||
status: true,
|
||||
triggers: true,
|
||||
createdAt: true,
|
||||
resolvedAt: true,
|
||||
resolvedBy: true,
|
||||
resolvedMessage: true,
|
||||
userMessage: true,
|
||||
userMessageAt: true,
|
||||
user: { select: { id: true, username: true, image: true } },
|
||||
},
|
||||
update: {
|
||||
addedBy: moderatorId,
|
||||
reason,
|
||||
},
|
||||
});
|
||||
}),
|
||||
dbRead.userRestriction.count({ where }),
|
||||
]);
|
||||
|
||||
// Bust the cached allowlist so the change takes effect immediately
|
||||
await bustPromptAllowlistCache();
|
||||
return { items, totalCount };
|
||||
}),
|
||||
|
||||
logToAxiom({
|
||||
name: 'prompt-allowlist-entry-added',
|
||||
type: 'info',
|
||||
details: { trigger, category, moderatorId, userRestrictionId },
|
||||
});
|
||||
|
||||
return { success: true };
|
||||
}),
|
||||
/** Moderator resolves a restriction — uphold or overturn. */
|
||||
resolve: moderatorProcedure.input(resolveRestrictionSchema).mutation(async ({ ctx, input }) => {
|
||||
await resolveUserRestriction({ ...input, moderatorId: ctx.user.id });
|
||||
return { success: true };
|
||||
}),
|
||||
|
||||
/** Debug endpoint to test prompt auditing without triggering any actions. */
|
||||
debugAudit: moderatorProcedure.input(debugAuditPromptSchema).mutation(async ({ input }) => {
|
||||
@@ -80,6 +95,101 @@ export const userRestrictionRouter = router({
|
||||
return debugAuditPrompt(prompt, negativePrompt);
|
||||
}),
|
||||
|
||||
/** Get today's prohibited request counts per user from ClickHouse. */
|
||||
getTodaysUserCounts: moderatorProcedure.query(async () => {
|
||||
if (!clickhouse) return { userCounts: [] };
|
||||
|
||||
const queryResult = await clickhouse.query({
|
||||
query: `
|
||||
SELECT userId, count() AS count
|
||||
FROM prohibitedRequests
|
||||
WHERE toDate(createdDate) = today()
|
||||
GROUP BY userId
|
||||
ORDER BY count DESC
|
||||
`,
|
||||
format: 'JSONEachRow',
|
||||
});
|
||||
|
||||
const userCounts = (await queryResult.json()) as Array<{
|
||||
userId: number;
|
||||
count: string;
|
||||
}>;
|
||||
|
||||
return {
|
||||
userCounts: userCounts.map((row) => ({
|
||||
userId: row.userId,
|
||||
count: Number(row.count),
|
||||
})),
|
||||
};
|
||||
}),
|
||||
|
||||
/** Get today's prohibited prompts from ClickHouse and run them through audit. */
|
||||
getTodaysAuditResults: moderatorProcedure.query(async () => {
|
||||
if (!clickhouse) return { results: [] };
|
||||
|
||||
// Fetch today's prohibited requests from ClickHouse
|
||||
const queryResult = await clickhouse.query({
|
||||
query: `
|
||||
SELECT userId, prompt, negativePrompt, source, createdDate
|
||||
FROM prohibitedRequests
|
||||
WHERE toDate(createdDate) = today()
|
||||
ORDER BY createdDate DESC
|
||||
LIMIT 500
|
||||
`,
|
||||
format: 'JSONEachRow',
|
||||
});
|
||||
|
||||
const rows = (await queryResult.json()) as Array<{
|
||||
odometer: number;
|
||||
userId: number;
|
||||
prompt: string;
|
||||
negativePrompt: string;
|
||||
source: string;
|
||||
createdDate: string;
|
||||
}>;
|
||||
|
||||
// Run each prompt through the audit system
|
||||
const results = rows.map((row) => {
|
||||
const auditResult = debugAuditPrompt(row.prompt, row.negativePrompt || undefined);
|
||||
return {
|
||||
userId: row.userId,
|
||||
prompt: row.prompt,
|
||||
negativePrompt: row.negativePrompt,
|
||||
source: row.source,
|
||||
createdDate: row.createdDate,
|
||||
matches: auditResult.matches,
|
||||
wouldBlock: auditResult.wouldBlock,
|
||||
blockReason: auditResult.blockReason,
|
||||
};
|
||||
});
|
||||
|
||||
return { results };
|
||||
}),
|
||||
|
||||
/** Save suspicious audit matches to Redis for later review. */
|
||||
saveSuspiciousMatches: moderatorProcedure
|
||||
.input(saveSuspiciousMatchSchema)
|
||||
.mutation(async ({ ctx, input }) => {
|
||||
const { matches } = input;
|
||||
const moderatorId = ctx.user.id;
|
||||
|
||||
// Add each match to a Redis list with timestamp and moderator info
|
||||
const entries = matches.map((match) => ({
|
||||
...match,
|
||||
flaggedBy: moderatorId,
|
||||
flaggedAt: new Date().toISOString(),
|
||||
}));
|
||||
|
||||
for (const entry of entries) {
|
||||
await sysRedis.lPush(REDIS_SYS_KEYS.SYSTEM.SUSPICIOUS_AUDIT_MATCHES, JSON.stringify(entry));
|
||||
}
|
||||
|
||||
// Keep only the last 1000 entries
|
||||
await sysRedis.lTrim(REDIS_SYS_KEYS.SYSTEM.SUSPICIOUS_AUDIT_MATCHES, 0, 999);
|
||||
|
||||
return { success: true, savedCount: entries.length };
|
||||
}),
|
||||
|
||||
/** Get suspicious audit matches from Redis. */
|
||||
getSuspiciousMatches: moderatorProcedure.query(async () => {
|
||||
const entries = await sysRedis.lRange(REDIS_SYS_KEYS.SYSTEM.SUSPICIOUS_AUDIT_MATCHES, 0, -1);
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import * as z from 'zod';
|
||||
import { paginationSchema } from '~/server/schema/base.schema';
|
||||
import { UserRestrictionStatus } from '~/shared/utils/prisma/enums';
|
||||
|
||||
export const submitRestrictionContextSchema = z.object({
|
||||
userRestrictionId: z.number(),
|
||||
@@ -6,13 +8,25 @@ export const submitRestrictionContextSchema = z.object({
|
||||
});
|
||||
export type SubmitRestrictionContextInput = z.infer<typeof submitRestrictionContextSchema>;
|
||||
|
||||
export const addToAllowlistSchema = z.object({
|
||||
trigger: z.string().min(1),
|
||||
category: z.string().min(1),
|
||||
reason: z.string().max(500).optional(),
|
||||
userRestrictionId: z.number().optional(),
|
||||
export const getGenerationRestrictionsSchema = paginationSchema.extend({
|
||||
status: z
|
||||
.enum([
|
||||
UserRestrictionStatus.Pending,
|
||||
UserRestrictionStatus.Upheld,
|
||||
UserRestrictionStatus.Overturned,
|
||||
])
|
||||
.optional(),
|
||||
username: z.string().optional(),
|
||||
userId: z.number().optional(),
|
||||
});
|
||||
export type AddToAllowlistInput = z.infer<typeof addToAllowlistSchema>;
|
||||
export type GetGenerationRestrictionsInput = z.infer<typeof getGenerationRestrictionsSchema>;
|
||||
|
||||
export const resolveRestrictionSchema = z.object({
|
||||
userRestrictionId: z.number(),
|
||||
status: z.enum([UserRestrictionStatus.Upheld, UserRestrictionStatus.Overturned]),
|
||||
resolvedMessage: z.string().max(1000).optional(),
|
||||
});
|
||||
export type ResolveRestrictionInput = z.infer<typeof resolveRestrictionSchema>;
|
||||
|
||||
export const debugAuditPromptSchema = z.object({
|
||||
prompt: z.string().min(1).max(10000),
|
||||
@@ -20,6 +34,22 @@ export const debugAuditPromptSchema = z.object({
|
||||
});
|
||||
export type DebugAuditPromptInput = z.infer<typeof debugAuditPromptSchema>;
|
||||
|
||||
export const saveSuspiciousMatchSchema = z.object({
|
||||
matches: z.array(
|
||||
z.object({
|
||||
odometer: z.number(),
|
||||
userId: z.number(),
|
||||
prompt: z.string(),
|
||||
negativePrompt: z.string().optional(),
|
||||
check: z.string(),
|
||||
matchedText: z.string(),
|
||||
regex: z.string().optional(),
|
||||
context: z.string().optional(),
|
||||
})
|
||||
),
|
||||
});
|
||||
export type SaveSuspiciousMatchInput = z.infer<typeof saveSuspiciousMatchSchema>;
|
||||
|
||||
export const backfillRestrictionTriggersSchema = z.object({
|
||||
userRestrictionId: z.number().optional(),
|
||||
limit: z.number().min(1).max(100).default(10),
|
||||
|
||||
@@ -6,12 +6,16 @@ import { TRPCError } from '@trpc/server';
|
||||
// return a JSON blocklist is enough to drive throwOnBlockedLinkDomain end-to-end.
|
||||
import {
|
||||
buildBenignPhraseRegex,
|
||||
getBlocklistDTO,
|
||||
getClientBenignLists,
|
||||
stripBenignPhrases,
|
||||
throwOnBlockedLinkDomain,
|
||||
upsertBlocklist,
|
||||
} from '../blocklist.service';
|
||||
import { BlocklistType } from '~/server/common/enums';
|
||||
import { dbMock } from '~/__tests__/mocks/db.mock';
|
||||
import { redisMock } from '~/__tests__/mocks/redis.mock';
|
||||
import { loggingMock } from '~/__tests__/mocks/logging.mock';
|
||||
const redisGet = redisMock.redis.get;
|
||||
|
||||
/** Make getBlocklistData return the given domains (already lower-cased in prod). */
|
||||
@@ -106,3 +110,198 @@ describe('stripBenignPhrases', () => {
|
||||
expect(await stripBenignPhrases(undefined, BlocklistType.NegativeBenignPhrase)).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('a type with more than one Blocklist row', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
// Force the DB read: a cache hit short-circuits before the rows are ever seen.
|
||||
redisGet.mockResolvedValue(null);
|
||||
});
|
||||
|
||||
// The fake HONOURS orderBy. With a fixed array the "lowest id wins" assertion passes
|
||||
// whatever the query asks for, so the fake would be deciding the outcome instead of the
|
||||
// code — and flipping the real orderBy to desc would stay green.
|
||||
// A row of ANOTHER type, with a LOWER id than either EmailDomain row. Without it the fake
|
||||
// returns the same rows whatever `where` says, so dropping the type filter from the query
|
||||
// stays green while production serves the link blocklist to an email-domain read.
|
||||
const rowsInDbOrder = [
|
||||
{ id: 8, type: BlocklistType.EmailDomain, data: ['c.example'] },
|
||||
{ id: 1, type: BlocklistType.EmailDomain, data: ['a.example', 'b.example'] },
|
||||
{ id: 0, type: BlocklistType.LinkDomain, data: ['wrong-list.example'] },
|
||||
];
|
||||
const respectOrderBy = (rows: typeof rowsInDbOrder) =>
|
||||
dbMock.dbWrite.blocklist.findMany.mockImplementation(
|
||||
async (args?: { where?: { type?: string }; orderBy?: { id?: 'asc' | 'desc' } }) => {
|
||||
// The fake honours BOTH `where` and `orderBy`. A fake that ignores an argument is a
|
||||
// fake that decides the outcome the assertion is supposed to be testing.
|
||||
const type = args?.where?.type;
|
||||
const filtered = type ? rows.filter((row) => row.type === type) : [...rows];
|
||||
const direction = args?.orderBy?.id;
|
||||
if (!direction) return filtered;
|
||||
return filtered.sort((a, b) => (direction === 'desc' ? b.id - a.id : a.id - b.id));
|
||||
}
|
||||
);
|
||||
|
||||
const twoRows = () => respectOrderBy(rowsInDbOrder);
|
||||
|
||||
it('CONTROL: a single row is returned as-is', async () => {
|
||||
respectOrderBy([
|
||||
{ id: 1, type: BlocklistType.EmailDomain, data: ['a.example'] },
|
||||
{ id: 0, type: BlocklistType.LinkDomain, data: ['wrong-list.example'] },
|
||||
]);
|
||||
|
||||
const result = await getBlocklistDTO({ type: BlocklistType.EmailDomain });
|
||||
expect(result.data).toEqual(['a.example']);
|
||||
});
|
||||
|
||||
it('picks the lowest id deterministically rather than whichever row the DB returned first', async () => {
|
||||
twoRows();
|
||||
|
||||
const result = await getBlocklistDTO({ type: BlocklistType.EmailDomain });
|
||||
expect(result.id).toBe(1);
|
||||
// NOT a union: merging silently would be a moderation bypass on the benign lists,
|
||||
// where a wider list strips more rather than blocking more.
|
||||
expect(result.data).toEqual(['a.example', 'b.example']);
|
||||
});
|
||||
|
||||
it('asks the DB for an ordered read, so the pick cannot depend on physical row order', async () => {
|
||||
twoRows();
|
||||
await getBlocklistDTO({ type: BlocklistType.EmailDomain });
|
||||
|
||||
expect(dbMock.dbWrite.blocklist.findMany).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
where: { type: BlocklistType.EmailDomain },
|
||||
orderBy: { id: 'asc' },
|
||||
})
|
||||
);
|
||||
});
|
||||
|
||||
it('reports the duplicate rather than picking silently', async () => {
|
||||
twoRows();
|
||||
await getBlocklistDTO({ type: BlocklistType.EmailDomain });
|
||||
|
||||
expect(loggingMock.logToAxiom).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
name: 'blocklist-duplicate-rows',
|
||||
type: 'error',
|
||||
details: expect.objectContaining({ usedId: 1, ignoredIds: [8] }),
|
||||
})
|
||||
);
|
||||
});
|
||||
|
||||
// The negative half matters on its own: this read gates account signup, so a report that
|
||||
// fired on every read would be hot-path log spam rather than a signal.
|
||||
it('CONTROL: reports nothing when the type has exactly one row', async () => {
|
||||
respectOrderBy([
|
||||
{ id: 1, type: BlocklistType.EmailDomain, data: ['a.example'] },
|
||||
{ id: 0, type: BlocklistType.LinkDomain, data: ['wrong-list.example'] },
|
||||
]);
|
||||
|
||||
await getBlocklistDTO({ type: BlocklistType.EmailDomain });
|
||||
|
||||
expect(loggingMock.logToAxiom).not.toHaveBeenCalledWith(
|
||||
expect.objectContaining({ name: 'blocklist-duplicate-rows' })
|
||||
);
|
||||
});
|
||||
|
||||
it('returns an empty list when the type has no rows at all', async () => {
|
||||
respectOrderBy([]);
|
||||
|
||||
const result = await getBlocklistDTO({ type: BlocklistType.EmailDomain });
|
||||
expect(result.data).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('what an edit writes into the cache', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
redisGet.mockResolvedValue(null);
|
||||
});
|
||||
|
||||
// The production incident was not the read: `upsertBlocklist` cached the row it had just
|
||||
// written under a key scoped to the TYPE, so editing the duplicate row promoted it to the
|
||||
// live answer for a month. Reverting to `data: result` leaves every read-side test green,
|
||||
// so this is the only thing standing between that bug and a re-introduction.
|
||||
const twoRows = () => {
|
||||
// Deliberately the SAME order-and-where-honouring fake as the read tests: a fake that
|
||||
// returns a fixed array would make `id: 1` below the fixture's answer rather than the
|
||||
// code's.
|
||||
dbMock.dbWrite.blocklist.findMany.mockImplementation(
|
||||
async (args?: { where?: { type?: string }; orderBy?: { id?: 'asc' | 'desc' } }) => {
|
||||
const rows = [
|
||||
{ id: 8, type: BlocklistType.EmailDomain, data: ['c.example'] },
|
||||
{ id: 1, type: BlocklistType.EmailDomain, data: ['a.example', 'b.example'] },
|
||||
];
|
||||
const type = args?.where?.type;
|
||||
const filtered = type ? rows.filter((row) => row.type === type) : [...rows];
|
||||
const direction = args?.orderBy?.id;
|
||||
if (!direction) return filtered;
|
||||
return filtered.sort((a, b) => (direction === 'desc' ? b.id - a.id : a.id - b.id));
|
||||
}
|
||||
);
|
||||
dbMock.dbWrite.blocklist.findUnique.mockResolvedValue({ data: ['c.example'] });
|
||||
dbMock.dbWrite.blocklist.update.mockResolvedValue({
|
||||
id: 8,
|
||||
type: BlocklistType.EmailDomain,
|
||||
data: ['c.example', 'new.example'],
|
||||
});
|
||||
};
|
||||
|
||||
const cachedPayload = () => {
|
||||
const call = redisMock.redis.set.mock.calls.at(-1);
|
||||
return call ? JSON.parse(call[1] as string) : undefined;
|
||||
};
|
||||
|
||||
it('caches the row that WINS the read, not the row that was just edited', async () => {
|
||||
twoRows();
|
||||
|
||||
await upsertBlocklist({
|
||||
id: 8,
|
||||
type: BlocklistType.EmailDomain,
|
||||
blocklist: ['new.example'],
|
||||
});
|
||||
|
||||
const cached = cachedPayload();
|
||||
expect(cached?.id).toBe(1);
|
||||
expect(cached?.data).toEqual(['a.example', 'b.example']);
|
||||
});
|
||||
|
||||
it('writes under a key scoped to the type', async () => {
|
||||
twoRows();
|
||||
|
||||
await upsertBlocklist({
|
||||
id: 8,
|
||||
type: BlocklistType.EmailDomain,
|
||||
blocklist: ['new.example'],
|
||||
});
|
||||
|
||||
expect(redisMock.redis.set.mock.calls.at(-1)?.[0]).toBe('system:blocklist:EmailDomain');
|
||||
});
|
||||
});
|
||||
|
||||
describe('getClientBenignLists', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
redisGet.mockResolvedValue(null);
|
||||
});
|
||||
|
||||
// Swapping the two fields of the returned object is invisible to a shape assertion, and
|
||||
// in the browser it would feed prompt phrases to the profanity filter and single words to
|
||||
// the phrase stripper — disarming both gates quietly. So the lists must be distinguishable.
|
||||
it('returns each list under the field the client expects', async () => {
|
||||
dbMock.dbWrite.blocklist.findMany.mockImplementation(
|
||||
async (args?: { where?: { type?: string } }) => {
|
||||
if (args?.where?.type === BlocklistType.PromptBenignPhrase)
|
||||
return [{ id: 7, type: BlocklistType.PromptBenignPhrase, data: ['teen titans'] }];
|
||||
if (args?.where?.type === BlocklistType.ProfanityBenignWord)
|
||||
return [{ id: 9, type: BlocklistType.ProfanityBenignWord, data: ['spreadsheet'] }];
|
||||
return [];
|
||||
}
|
||||
);
|
||||
|
||||
const result = await getClientBenignLists();
|
||||
|
||||
expect(result.prompt).toEqual(['teen titans']);
|
||||
expect(result.profanityWords).toEqual(['spreadsheet']);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { readFileSync } from 'fs';
|
||||
import path from 'path';
|
||||
|
||||
/**
|
||||
* Every POI check that WRITES A TAG must read the benign-stripped prompt.
|
||||
*
|
||||
* This exists because the fix for that bug was applied to one of two live copies. The image
|
||||
* scan pipeline has two implementations — `getTagsFromIncomingTags` in the webhook (legacy
|
||||
* `TagSource.WD14`/`Clavata`/`Hive` bodies) and `processTags` in the service (bodies carrying
|
||||
* `workflowId`, i.e. the orchestrator) — and the webhook dispatches between them by body
|
||||
* shape. Patching the one the audit named left the other writing a confidence-100 POI tag
|
||||
* from the raw prompt, which reaches the image search index. Behavioural tests did not catch
|
||||
* it because each drives one branch, and the untested branch was the live one.
|
||||
*
|
||||
* So this is a source-level guard, and it is deliberately about the SHAPE of the call rather
|
||||
* than its behaviour: a behavioural test proves the copy it drives, and the failure mode here
|
||||
* is the copy nobody drove.
|
||||
*
|
||||
* TWO LIMITS, stated because a guard whose edges are undocumented gets read as covering more
|
||||
* than it does:
|
||||
*
|
||||
* 1. The enclosing-scope check walks back to the nearest `function ` keyword, so it does not
|
||||
* understand an arrow function or a class method — one of those would inherit the previous
|
||||
* declaration's strip and pass. Both taggers are `function` declarations today; a new one
|
||||
* written as an arrow needs the strip INLINE at the call to be checked properly.
|
||||
* 2. Scope is the two files that WRITE A TAG from a POI hit, which is this guard's title. Two
|
||||
* other server-side `includesPoi` calls read a raw prompt and are deliberately NOT covered,
|
||||
* because neither writes a tag:
|
||||
* - `services/apps/shared-content-safety.ts` — throws `SharedContentBlockedError('poi')`,
|
||||
* so a whitelisted proper noun hard-refuses the content rather than mislabelling it.
|
||||
* - `services/orchestrator/orchestration-new.service.ts` — a second POI decision on the
|
||||
* generation path, beside the `auditPromptServer` one that does strip.
|
||||
* Both are real gaps in the whitelist's reach, and both are follow-up work rather than
|
||||
* oversights — they are named here so the two-file list does not read as settled.
|
||||
*/
|
||||
|
||||
const REPO_ROOT = path.resolve(__dirname, '../../../..');
|
||||
|
||||
const TAG_WRITING_SCAN_FILES = [
|
||||
'src/pages/api/webhooks/image-scan-result.ts',
|
||||
'src/server/services/image-scan-result.service.ts',
|
||||
];
|
||||
|
||||
const POI_CALL = /includesPoi\(/g;
|
||||
|
||||
/**
|
||||
* The argument list of one `includesPoi(` call — paren-balanced, so it ends at that call's own
|
||||
* closing paren. A fixed-size forward window was tried and is wrong: it reads past the end of
|
||||
* the call, so an unrelated `stripBenignPhrases` on the NEXT statement satisfies it, and both
|
||||
* of these files contain several. Balancing means only this call's own argument counts.
|
||||
*/
|
||||
function callArguments(source: string, start: number) {
|
||||
let depth = 0;
|
||||
for (let i = source.indexOf('(', start); i < source.length; i++) {
|
||||
if (source[i] === '(') depth++;
|
||||
else if (source[i] === ')') {
|
||||
depth--;
|
||||
if (depth === 0) return source.slice(start, i + 1);
|
||||
}
|
||||
}
|
||||
return source.slice(start);
|
||||
}
|
||||
|
||||
function readSource(rel: string) {
|
||||
const source = readFileSync(path.join(REPO_ROOT, rel), 'utf8');
|
||||
// Fail closed: an empty or moved file must not read as "no offending call sites".
|
||||
expect(source.length, `${rel} is empty or unreadable`).toBeGreaterThan(1000);
|
||||
// Comments are stripped before scanning, and the reason is NOT the one this used to give:
|
||||
// the fixed inspection window that comments could push code out of is gone, replaced by
|
||||
// paren-balancing. The reason now is the scope check — a comment MENTIONING
|
||||
// `stripBenignPhrases` anywhere between a function's declaration and the call would
|
||||
// otherwise satisfy `strippedInScope` and pass a call that strips nothing.
|
||||
return source.replace(/\/\*[\s\S]*?\*\//g, ' ').replace(/\/\/[^\r\n]*/g, ' ');
|
||||
}
|
||||
|
||||
describe('POI taggers read the benign-stripped prompt', () => {
|
||||
it('CONTROL: the matcher can actually see includesPoi calls in both files', () => {
|
||||
for (const rel of TAG_WRITING_SCAN_FILES) {
|
||||
const matches = [...readSource(rel).matchAll(POI_CALL)];
|
||||
expect(matches.length, `no includesPoi call found in ${rel}`).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
it.each(TAG_WRITING_SCAN_FILES)(
|
||||
'every includesPoi call in %s passes a stripped or already-audited prompt',
|
||||
(rel) => {
|
||||
const source = readSource(rel);
|
||||
|
||||
for (const match of source.matchAll(POI_CALL)) {
|
||||
const argumentList = callArguments(source, match.index);
|
||||
const call = argumentList.split(/\r?\n/)[0].trim();
|
||||
|
||||
// Two ways to be safe, and the identifier alone can decide neither: the audit
|
||||
// functions shadow `prompt` with the stripped copy, so the safe call and the unsafe
|
||||
// one are spelled identically.
|
||||
// - the strip is inline in the call itself, or
|
||||
// - the ENCLOSING function stripped before reaching it.
|
||||
const inlineStrip = argumentList.includes('stripBenignPhrases');
|
||||
const declaration = source.lastIndexOf('function ', match.index);
|
||||
const strippedInScope = source
|
||||
.slice(declaration, match.index)
|
||||
.includes('stripBenignPhrases');
|
||||
|
||||
expect(
|
||||
inlineStrip || strippedInScope,
|
||||
`${rel}: \`${call}\` reads a prompt nothing stripped. A POI hit here writes a ` +
|
||||
'confidence-100 tag that reaches the search index, so a moderator-whitelisted ' +
|
||||
'phrase would still flag the image.'
|
||||
).toBe(true);
|
||||
}
|
||||
}
|
||||
);
|
||||
});
|
||||
@@ -6,6 +6,8 @@ import { redis, REDIS_KEYS } from '~/server/redis/client';
|
||||
import type { UpsertBlocklistSchema } from '~/server/schema/blocklist.schema';
|
||||
import { throwBadRequestError } from '~/server/utils/errorHandling';
|
||||
import { createLruCache } from '~/server/utils/lru-cache';
|
||||
import { logToAxiom } from '~/server/logging/client';
|
||||
import { buildBenignPhraseRegex, stripBenignPhrasesWith } from '~/shared/utils/benign-phrases';
|
||||
|
||||
export type BlocklistDTO = {
|
||||
id?: number;
|
||||
@@ -44,19 +46,60 @@ export async function upsertBlocklist({ id, type, blocklist }: UpsertBlocklistSc
|
||||
select: { id: true, type: true, data: true },
|
||||
});
|
||||
if (!result) throw new Error('failed to update blocklist');
|
||||
await setCache({ type: result.type, data: result });
|
||||
// Refresh from the deterministic read, not from `result`: caching the row just written
|
||||
// is what let an edit to a duplicate row silently promote it to the live one.
|
||||
await setCache({ type: result.type, data: await readBlocklistRow(result.type as BlocklistType) });
|
||||
}
|
||||
|
||||
/**
|
||||
* Nothing stops a type having more than one row, and `EmailDomain` has two in production
|
||||
* — 8292 entries against 3, with the 3 present in neither the other row nor anything
|
||||
* else. The old read took `findFirst` with no `orderBy`, so which set was live was up to
|
||||
* Postgres; worse, `upsertBlocklist` wrote the row it had just touched into a cache key
|
||||
* scoped to the TYPE, so the last row a moderator edited won for a month regardless. The
|
||||
* loser's entries were simply not enforced, and the winner could change on an unrelated
|
||||
* edit.
|
||||
*
|
||||
* This picks the lowest id, always, and reports the duplicate. It deliberately does NOT
|
||||
* union the rows: for a deny-list a union blocks more, but for the benign lists a union
|
||||
* strips more, which is a moderation bypass — one helper cannot silently pick a safe
|
||||
* direction for both. Nor does it throw: this read gates account signup, so a duplicate
|
||||
* row would become an outage. Deterministic-and-loud is the compromise until the rows are
|
||||
* merged and a unique index on `type` makes it unrepresentable.
|
||||
*/
|
||||
async function readBlocklistRow(type: BlocklistType): Promise<BlocklistDTO> {
|
||||
const rows = await dbWrite.blocklist.findMany({
|
||||
where: { type },
|
||||
select: { id: true, type: true, data: true },
|
||||
orderBy: { id: 'asc' },
|
||||
});
|
||||
|
||||
if (rows.length > 1) {
|
||||
logToAxiom({
|
||||
name: 'blocklist-duplicate-rows',
|
||||
type: 'error',
|
||||
message:
|
||||
'More than one Blocklist row for a type; entries on the ignored rows are not enforced',
|
||||
details: {
|
||||
blocklistType: type,
|
||||
usedId: rows[0].id,
|
||||
ignoredIds: rows.slice(1).map((row) => row.id),
|
||||
ignoredEntryCounts: rows.slice(1).map((row) => row.data.length),
|
||||
},
|
||||
}).catch(() => undefined);
|
||||
}
|
||||
|
||||
// The absent-row fallback deliberately carries NO `id`: `getClientBenignLists` reads that as
|
||||
// "no moderator row yet, use the bundled list". The moderator spoke writes the same shape into
|
||||
// the same Redis key, so this sentinel is a cross-app contract, not a local detail.
|
||||
return rows[0] ?? { type, data: [] };
|
||||
}
|
||||
|
||||
export async function getBlocklistDTO({ type }: { type: BlocklistType }) {
|
||||
const cached = await redis.get(getBlocklistKey(type));
|
||||
if (cached) return JSON.parse(cached) as BlocklistDTO;
|
||||
|
||||
const result = await dbWrite.blocklist
|
||||
.findFirst({
|
||||
where: { type },
|
||||
select: { id: true, type: true, data: true },
|
||||
})
|
||||
.then((result): BlocklistDTO => (result ? result : { type, data: [] }));
|
||||
const result = await readBlocklistRow(type);
|
||||
|
||||
await setCache({ type: result.type, data: result });
|
||||
return result;
|
||||
@@ -99,21 +142,6 @@ export async function throwOnBlockedLinkDomain(value: string) {
|
||||
// #endregion
|
||||
|
||||
// #region [benign phrases]
|
||||
const escapeRegex = (str: string) => str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
|
||||
// Compile a moderator-managed phrase list into a single whole-word, case-insensitive
|
||||
// matcher. Whitespace in a phrase is loosened to `[^a-zA-Z0-9]+` (the same inter-word
|
||||
// separator audit.ts uses) so "teen titans" also matches "teen titans" / "teen-titans"
|
||||
// / "teen.titans". Zero-width alnum boundaries (again matching audit.ts) instead of `\b`
|
||||
// so a phrase whose edge is punctuation still anchors. Returns null for an empty list so
|
||||
// callers can skip the replace.
|
||||
export function buildBenignPhraseRegex(phrases: string[]): RegExp | null {
|
||||
const cleaned = phrases.map((p) => p.trim()).filter((p) => p.length > 0);
|
||||
if (!cleaned.length) return null;
|
||||
const alternation = cleaned.map((p) => escapeRegex(p).replace(/\s+/g, '[^a-zA-Z0-9]+')).join('|');
|
||||
return new RegExp(`(?<![a-zA-Z0-9])(?:${alternation})(?![a-zA-Z0-9])`, 'gi');
|
||||
}
|
||||
|
||||
// In-process TTL cache over the Redis-backed blocklist: the scan path strips benign
|
||||
// phrases on every image, so a short-lived local copy avoids a Redis round-trip per
|
||||
// scan. TTL (not cross-pod invalidation) is the freshness bound — a moderator edit
|
||||
@@ -125,11 +153,56 @@ const benignPhraseRegexCache = createLruCache<BlocklistType, { pattern: RegExp |
|
||||
fetchFn: async (type) => ({ pattern: buildBenignPhraseRegex(await getBlocklistData(type)) }),
|
||||
});
|
||||
|
||||
export { buildBenignPhraseRegex };
|
||||
|
||||
export async function stripBenignPhrases(text = '', type: BlocklistType) {
|
||||
if (!text) return text;
|
||||
const { pattern } = await benignPhraseRegexCache.fetch(type);
|
||||
if (!pattern) return text;
|
||||
return text.replace(pattern, ' ');
|
||||
// Same helper as the client gates on purpose — it carries the refusal that stops a
|
||||
// letter-bearing gap being swallowed, and the two sides must strip identically.
|
||||
return stripBenignPhrasesWith(text, pattern);
|
||||
}
|
||||
/**
|
||||
* The benign lists the BROWSER needs. The search gates (`AutocompleteSearch`,
|
||||
* `SearchLayout`) run their POI / minor / profanity checks client-side against Meili
|
||||
* directly, so there is no server hop to strip on — the lists have to be shipped down.
|
||||
* Public and edge-cached; these are phrases moderators have declared SAFE, so the list
|
||||
* says nothing about what we block.
|
||||
*/
|
||||
export type ClientBenignLists = {
|
||||
prompt: string[];
|
||||
/** `null` means "no moderator row", which is NOT the same as an empty one — see the filter. */
|
||||
profanityWords: string[] | null;
|
||||
/** False when the lists could not be read. The caller MUST NOT let the edge cache this. */
|
||||
available: boolean;
|
||||
};
|
||||
|
||||
export async function getClientBenignLists(): Promise<ClientBenignLists> {
|
||||
try {
|
||||
const [prompt, profanityRow] = await Promise.all([
|
||||
getBlocklistData(BlocklistType.PromptBenignPhrase),
|
||||
getBlocklistDTO({ type: BlocklistType.ProfanityBenignWord }),
|
||||
]);
|
||||
// A row that EXISTS but is empty is a moderator having deleted every entry — the
|
||||
// strongest possible "do not whitelist" intent. Only a MISSING row means "not migrated",
|
||||
// which is the case that falls back to the list shipped in the bundle.
|
||||
const profanityWords = profanityRow.id == null ? null : profanityRow.data;
|
||||
return { prompt, profanityWords, available: true };
|
||||
} catch (error) {
|
||||
// Fails OPEN, and empty is the safe direction: nothing is stripped, so the gates flag
|
||||
// more rather than less. `available: false` exists because the caller must then skip the
|
||||
// edge cache — a 200 carrying empty lists would otherwise be held for an hour, and every
|
||||
// session started in that window pins it for its whole life, reinstating globally the
|
||||
// false positives this work removes. The throw it replaced was never cached.
|
||||
logToAxiom({
|
||||
name: 'benign-lists-unavailable',
|
||||
type: 'warning',
|
||||
message: 'Serving empty benign lists to the client; search gates will not strip',
|
||||
details: { error: error instanceof Error ? error.message : String(error) },
|
||||
}).catch(() => undefined);
|
||||
return { prompt: [], profanityWords: null, available: false };
|
||||
}
|
||||
}
|
||||
// #endregion
|
||||
|
||||
|
||||
@@ -741,8 +741,17 @@ async function processTags({
|
||||
prompt?: string;
|
||||
}): Promise<ProcessedTag[]> {
|
||||
if (prompt) {
|
||||
// Detect real person in prompt
|
||||
const realPersonName = includesPoi(prompt);
|
||||
// Detect real person in prompt. Strips the moderator-managed benign phrases first, as
|
||||
// the audit path does — without it a whitelisted proper noun is still written as a
|
||||
// confidence-100 tag, which reaches the image search index. This is the orchestrator
|
||||
// path; `getTagsFromIncomingTags` in the webhook is the legacy twin of this function
|
||||
// and needs the same treatment.
|
||||
const realPersonName = includesPoi(
|
||||
// Normalized first, as both audit paths do. Stripping the raw text while the audit that
|
||||
// runs next reads the normalized copy means two different alphabets decide what counts
|
||||
// as whitelisted for the same prompt.
|
||||
await stripBenignPhrases(normalizeText(prompt), BlocklistType.PromptBenignPhrase)
|
||||
);
|
||||
if (realPersonName) {
|
||||
const tagName =
|
||||
typeof realPersonName === 'object' ? realPersonName.matchedText : realPersonName;
|
||||
|
||||
@@ -1,13 +1,11 @@
|
||||
import { CacheTTL, constants } from '~/server/common/constants';
|
||||
import { constants } from '~/server/common/constants';
|
||||
import { BlocklistType } from '~/server/common/enums';
|
||||
import { dbRead } from '~/server/db/client';
|
||||
import { extModeration } from '~/server/integrations/moderation';
|
||||
import { logToAxiom } from '~/server/logging/client';
|
||||
import { REDIS_KEYS, REDIS_SYS_KEYS, sysRedis, withSysReadDeadline } from '~/server/redis/client';
|
||||
import { REDIS_SYS_KEYS, sysRedis, withSysReadDeadline } from '~/server/redis/client';
|
||||
import { decodeRedisString } from '~/server/redis/buffer-decode';
|
||||
import { stripBenignPhrases } from '~/server/services/blocklist.service';
|
||||
import { applyPendingReviewMute } from '~/server/services/user-restriction.service';
|
||||
import { fetchThroughCache, bustFetchThroughCache } from '~/server/utils/cache-helpers';
|
||||
import { throwBadRequestError } from '~/server/utils/errorHandling';
|
||||
import {
|
||||
auditPromptEnriched,
|
||||
@@ -192,45 +190,6 @@ async function clearBlockedPromptsAfterMute(userId: number) {
|
||||
await sysRedis.del(key);
|
||||
}
|
||||
|
||||
// --- Prompt Allowlist Cache ---
|
||||
// Caches the set of allowlisted (trigger, category) pairs used to filter out
|
||||
// false positives from prompt auditing before counting toward mute thresholds.
|
||||
type AllowlistEntry = { trigger: string; category: string };
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
async function getCachedPromptAllowlist(): Promise<Set<string>> {
|
||||
const entries = await fetchThroughCache(
|
||||
REDIS_KEYS.SYSTEM.PROMPT_ALLOWLIST,
|
||||
async () => {
|
||||
const rows = await dbRead.promptAllowlist.findMany({
|
||||
select: { trigger: true, category: true },
|
||||
});
|
||||
return rows as AllowlistEntry[];
|
||||
},
|
||||
{ ttl: CacheTTL.day }
|
||||
);
|
||||
// Build a Set of "trigger:category" keys for O(1) lookup
|
||||
return new Set(entries.map((e) => `${e.trigger}:${e.category}`));
|
||||
}
|
||||
|
||||
/** Bust the prompt allowlist cache (call after adding/removing entries). */
|
||||
export async function bustPromptAllowlistCache() {
|
||||
await bustFetchThroughCache(REDIS_KEYS.SYSTEM.PROMPT_ALLOWLIST);
|
||||
}
|
||||
|
||||
/** Filter triggers against the allowlist, returning only non-allowlisted triggers. */
|
||||
// eslint-disable-next-line @typescript-eslint/no-unused-vars
|
||||
function filterAllowlistedTriggers(
|
||||
triggers: PromptTrigger[],
|
||||
allowlist: Set<string>
|
||||
): PromptTrigger[] {
|
||||
if (allowlist.size === 0) return triggers;
|
||||
return triggers.filter((t) => {
|
||||
if (!t.matchedWord) return true; // no specific word to match — keep it
|
||||
return !allowlist.has(`${t.matchedWord}:${t.category}`);
|
||||
});
|
||||
}
|
||||
|
||||
export interface AuditPromptOptions {
|
||||
prompt: string;
|
||||
negativePrompt?: string;
|
||||
@@ -282,12 +241,6 @@ export async function auditPromptServer(options: AuditPromptOptions): Promise<vo
|
||||
// If isGreen is false (civitai.com/red), run standard NSFW blocking
|
||||
const checkProfanity = isGreen;
|
||||
|
||||
// NOTE: Allowlist runtime filtering is disabled for now. The allowlist management
|
||||
// endpoints remain active so moderators can curate entries. To re-enable, uncomment
|
||||
// the allowlist fetch below and use filterAllowlistedTriggers() on each trigger set.
|
||||
// const allowlist = await getCachedPromptAllowlist();
|
||||
const allowlist = new Set<string>();
|
||||
|
||||
// Moderator-managed benign phrases (proper nouns / technical terms that
|
||||
// coincidentally contain a detection token) are blanked before auditing, so the
|
||||
// generation gate and the post-generation scan audit agree on what's benign.
|
||||
@@ -309,19 +262,17 @@ export async function auditPromptServer(options: AuditPromptOptions): Promise<vo
|
||||
null;
|
||||
|
||||
if (!success) {
|
||||
// Filter out allowlisted triggers before counting toward mute
|
||||
const remainingTriggers = filterAllowlistedTriggers(triggers, allowlist);
|
||||
if (remainingTriggers.length > 0) {
|
||||
if (triggers.length > 0) {
|
||||
const regexBlock = {
|
||||
blockedFor: remainingTriggers.map((t) => t.message),
|
||||
triggers: remainingTriggers,
|
||||
blockedFor: triggers.map((t) => t.message),
|
||||
triggers,
|
||||
type: 'regex',
|
||||
};
|
||||
// A hard block short-circuits. A soft one must NOT — throwing here would
|
||||
// skip the external classifier below, so appending any overridable word
|
||||
// ("… pee") to a prompt would buy a click-through past it. Hold the block
|
||||
// and let external moderation run first; it can only escalate.
|
||||
if (!isSoftBlock(remainingTriggers)) throw regexBlock;
|
||||
if (!isSoftBlock(triggers)) throw regexBlock;
|
||||
softRegexBlock = regexBlock;
|
||||
}
|
||||
}
|
||||
@@ -340,12 +291,10 @@ export async function auditPromptServer(options: AuditPromptOptions): Promise<vo
|
||||
message: cat,
|
||||
matchedWord: cat,
|
||||
}));
|
||||
// Filter out allowlisted external triggers
|
||||
const remainingTriggers = filterAllowlistedTriggers(externalTriggers, allowlist);
|
||||
if (remainingTriggers.length > 0) {
|
||||
if (externalTriggers.length > 0) {
|
||||
throw {
|
||||
blockedFor: remainingTriggers.map((t) => t.message),
|
||||
triggers: remainingTriggers,
|
||||
blockedFor: externalTriggers.map((t) => t.message),
|
||||
triggers: externalTriggers,
|
||||
type: 'external',
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { buildBenignPhraseRegex, stripBenignPhrasesWith } from '~/shared/utils/benign-phrases';
|
||||
import { includesPoi } from '~/utils/metadata/audit';
|
||||
import { createProfanityFilter, getProfanityFilter } from '~/libs/profanity-simple';
|
||||
|
||||
describe('benign phrases reach the client-side search gates', () => {
|
||||
const strip = (text: string, phrases: string[]) =>
|
||||
stripBenignPhrasesWith(text, buildBenignPhraseRegex(phrases));
|
||||
|
||||
it('CONTROL: a POI name the moderator has NOT whitelisted still trips the search gate', () => {
|
||||
expect(includesPoi(strip('emma stone portrait', ['teen titans']))).toBe('emma stone');
|
||||
});
|
||||
|
||||
it('a whitelisted phrase is blanked, so the search gate lets the query through', () => {
|
||||
expect(includesPoi(strip('emma stone portrait', ['emma stone']))).toBe(false);
|
||||
});
|
||||
|
||||
it('leaves the rest of the query intact', () => {
|
||||
expect(strip('emma stone portrait', ['emma stone']).trim()).toBe('portrait');
|
||||
});
|
||||
|
||||
// Each case is one the DETECTOR matches on the raw text, so the strip has real work to do.
|
||||
// A case the detector already misses (e.g. one whose punctuation its preprocessor deletes)
|
||||
// would pass here with no matcher at all — that is how the previous version of this test
|
||||
// was vacuous.
|
||||
it.each([
|
||||
'emma stone',
|
||||
'emma. stone',
|
||||
'emma,,,, stone',
|
||||
'emma , stone',
|
||||
'emma____stone',
|
||||
'emma :: | stone',
|
||||
])('agrees with the detector on %j, however many separators sit between the words', (text) => {
|
||||
expect(Boolean(includesPoi(text)), `${text} should trip the detector raw`).toBe(true);
|
||||
expect(includesPoi(strip(text, ['emma stone']))).toBe(false);
|
||||
});
|
||||
|
||||
it('does not blank a longer word that merely contains the phrase', () => {
|
||||
expect(strip('stonemason', ['stone'])).toBe('stonemason');
|
||||
});
|
||||
|
||||
// The gate this protects: the nsfw word list decides whether the POI and minor sub-checks
|
||||
// run at all, so swallowing the only nsfw signal in an input silences them rather than
|
||||
// merely hiding one term. `[^a-zA-Z0-9]` excludes only ASCII alphanumerics, so a non-ASCII
|
||||
// word is a separator to this pattern and a word to the detector.
|
||||
it('refuses to strip when the gap between the words holds a letter', () => {
|
||||
const withLetters = 'emma шок stone';
|
||||
expect(strip(withLetters, ['emma stone'])).toBe(withLetters);
|
||||
});
|
||||
|
||||
// Letters are not the only thing a detection list holds: every entry surviving normalization
|
||||
// in `blocklist.json` is an emoji, carrying no `\p{L}` at all. Those are out of reach today
|
||||
// only because of which list `auditMetaData` selects on `nsfw === false` — a coincidence
|
||||
// between two files that do not reference each other.
|
||||
it.each([
|
||||
['an emoji', 'emma \u{1F600} stone'],
|
||||
['non-ASCII digits', 'emma १२ stone'],
|
||||
])('refuses to strip when the gap holds %s', (_label, text) => {
|
||||
expect(strip(text, ['emma stone'])).toBe(text);
|
||||
});
|
||||
|
||||
// …and the other half of that: non-ASCII PUNCTUATION is not content, so it must still strip.
|
||||
// A "whitespace or ASCII punctuation only" predicate would fail these two.
|
||||
it.each(['emma—stone', 'emma、stone'])(
|
||||
'CONTROL: still strips across the non-ASCII punctuation gap %j',
|
||||
(text) => {
|
||||
expect(strip(text, ['emma stone']).trim()).toBe('');
|
||||
}
|
||||
);
|
||||
|
||||
it('CONTROL: the same shape with a punctuation-only gap still strips', () => {
|
||||
expect(strip('emma ,, stone', ['emma stone']).trim()).toBe('');
|
||||
});
|
||||
|
||||
it('blanks the phrase for the ordinary separators a moderator would expect', () => {
|
||||
for (const text of ['emma stone', 'emma stone', 'emma-stone', 'emma. stone']) {
|
||||
expect(strip(text, ['emma stone']).trim(), text).toBe('');
|
||||
}
|
||||
});
|
||||
|
||||
it('returns null for an empty list so callers can skip the replace', () => {
|
||||
expect(buildBenignPhraseRegex([])).toBeNull();
|
||||
expect(buildBenignPhraseRegex([' '])).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('moderator benign words reach the profanity filter', () => {
|
||||
it('CONTROL: with no moderator list, the static whitelist is in effect', () => {
|
||||
const filter = createProfanityFilter();
|
||||
// `spreadsheet` is not in the static list, so it is flagged for containing `spread`.
|
||||
expect(filter.analyze('spreadsheet').isProfane).toBe(true);
|
||||
// `cockpit` IS in the static list and IS rescued by it (verified: 49 of the 424 static
|
||||
// words are flagged without it, and this is one of them — most of the rest are already
|
||||
// covered by obscenity's own recommended whitelist, so picking an arbitrary static word
|
||||
// would have made this assertion vacuous).
|
||||
expect(filter.analyze('cockpit').isProfane).toBe(false);
|
||||
});
|
||||
|
||||
it('a moderator-whitelisted word is no longer flagged', () => {
|
||||
const filter = createProfanityFilter({ moderatorWhitelist: ['spreadsheet'] });
|
||||
expect(filter.analyze('spreadsheet').isProfane).toBe(false);
|
||||
});
|
||||
|
||||
// The point of REPLACING rather than unioning: a moderator can take a word OUT. Under a
|
||||
// union every static word stays whitelisted forever and the UI's Remove control is a
|
||||
// silent no-op, since the seed migration copies the static list verbatim.
|
||||
it('the moderator list REPLACES the static one, so removing an entry takes effect', () => {
|
||||
const filter = createProfanityFilter({ moderatorWhitelist: ['spreadsheet'] });
|
||||
expect(filter.analyze('cockpit').isProfane).toBe(true);
|
||||
});
|
||||
|
||||
it('falls back to the static list when there is NO moderator row', () => {
|
||||
const filter = createProfanityFilter({ moderatorWhitelist: null });
|
||||
expect(filter.analyze('cockpit').isProfane).toBe(false);
|
||||
});
|
||||
|
||||
// `[]` and `null` are different states and must not collapse: an empty row is a moderator
|
||||
// having deleted every entry, which is the strongest possible "do not whitelist" intent.
|
||||
// Restoring the ~450 shipped words over the top of that would be the opposite of the ask.
|
||||
it('honours an EMPTY moderator row instead of restoring the static list', () => {
|
||||
const filter = createProfanityFilter({ moderatorWhitelist: [] });
|
||||
expect(filter.analyze('cockpit').isProfane).toBe(true);
|
||||
});
|
||||
|
||||
it('whitelisting one word does not disarm the filter for the token itself', () => {
|
||||
const filter = createProfanityFilter({ moderatorWhitelist: ['spreadsheet'] });
|
||||
expect(filter.analyze('spread').isProfane).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// Every production caller goes through `getProfanityFilter`, not `createProfanityFilter`, and
|
||||
// the null/empty distinction the tests above establish is decided by ITS cache key. Asserting
|
||||
// the semantics one layer below where they are chosen leaves the key free to collapse them.
|
||||
describe('getProfanityFilter keeps "no row" and "emptied" apart', () => {
|
||||
it('CONTROL: the same input returns the same shared instance', () => {
|
||||
expect(getProfanityFilter(['spreadsheet'])).toBe(getProfanityFilter(['spreadsheet']));
|
||||
});
|
||||
|
||||
it('does not serve the no-row filter to a caller passing an empty list', () => {
|
||||
const noRow = getProfanityFilter(null);
|
||||
const emptied = getProfanityFilter([]);
|
||||
|
||||
expect(emptied).not.toBe(noRow);
|
||||
// And they behave differently, which is the point of keeping them apart.
|
||||
expect(noRow.analyze('cockpit').isProfane).toBe(false);
|
||||
expect(emptied.analyze('cockpit').isProfane).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,97 @@
|
||||
// Client-safe half of the moderator benign-phrase machinery. The server copy lives in
|
||||
// `blocklist.service`, which pulls in Prisma and Redis and therefore cannot be imported
|
||||
// from a component — the search gates run in the browser, so the matcher has to live here.
|
||||
|
||||
const escapeRegex = (str: string) => str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
|
||||
// Compile a moderator-managed phrase list into a single whole-word, case-insensitive
|
||||
// matcher. Whitespace in a phrase is loosened to `[^a-zA-Z0-9]+` so "teen titans" also matches
|
||||
// "teen titans" / "teen-titans" / "teen.titans". Zero-width alnum boundaries (matching
|
||||
// audit.ts) instead of a word boundary, so a phrase whose edge is punctuation still anchors.
|
||||
// Returns null for an empty list so callers can skip the replace.
|
||||
//
|
||||
// 🔴 The separator MUST stay `+`, identical to `prepareWordRegex` in audit-base.ts. This
|
||||
// matcher exists to shadow that one, and any divergence breaks the whitelist in whichever
|
||||
// direction the detector is more permissive. A `{1,3}` bound was tried here and reverted: the
|
||||
// detector matched "emma,,,, stone" while the strip did not, so a whitelisted phrase went
|
||||
// back to tripping the gate on ordinary prompt punctuation — the exact bug this PR fixes.
|
||||
//
|
||||
// The bound was added on the reasoning that unbounded strips more. That instinct is the right
|
||||
// one to carry on a whitelist, but it has to yield to shadowing — and note what the class is
|
||||
// NOT: `[^a-zA-Z0-9]` excludes only ASCII alphanumerics, so it happily consumes non-ASCII
|
||||
// LETTERS. This separator can therefore swallow text that reads as a word.
|
||||
//
|
||||
// 🔴 What keeps that NARROW — not closed — is that callers strip the NORMALIZED copy:
|
||||
// `normalizeText` folds accented Latin to ASCII, and those letters then break the separator
|
||||
// run themselves. Load-bearing, not tidiness. Move a strip back ahead of `normalizeText` and
|
||||
// every accented spelling of every detection term becomes swallowable, which is a far larger
|
||||
// surface than what remains. What remains is scripts NFD cannot fold (CJK, Cyrillic, Greek,
|
||||
// …), reproduced against the real lists — private note has the input and the counts.
|
||||
//
|
||||
// If a bound is ever genuinely wanted, derive it from audit-base.ts rather than restating a
|
||||
// number, and pin the two matchers agreeing at the boundary.
|
||||
export function buildBenignPhraseRegex(phrases: string[]): RegExp | null {
|
||||
const cleaned = phrases.map((p) => p.trim()).filter((p) => p.length > 0);
|
||||
if (!cleaned.length) return null;
|
||||
// The separator is CAPTURED so the strip can inspect what it matched — see
|
||||
// `stripBenignPhrasesWith`. Capturing changes nothing about what the pattern matches, which
|
||||
// is what keeps it identical to `prepareWordRegex`.
|
||||
const alternation = cleaned
|
||||
.map((p) => escapeRegex(p).replace(/\s+/g, '([^a-zA-Z0-9]+)'))
|
||||
.join('|');
|
||||
return new RegExp(`(?<![a-zA-Z0-9])(?:${alternation})(?![a-zA-Z0-9])`, 'gi');
|
||||
}
|
||||
|
||||
/**
|
||||
* Anything a detection list could hold, not letters specifically. `\p{L}` alone was tried and
|
||||
* misses 9 of the 54 entries that survive normalization across the lists — all emoji, which
|
||||
* carry no letter. Those are unreachable today only because `auditMetaData` picks that list on
|
||||
* `nsfw === false` while every caller passing a stripped prompt sits inside `if (nsfw && …)` —
|
||||
* a coincidence between two files that do not reference each other, which either side can
|
||||
* close without noticing. `\p{N}` and pictographics cost nothing and make the predicate about
|
||||
* CONTENT rather than about letters, which is the property that survives someone adding a new
|
||||
* kind of entry to a list.
|
||||
*
|
||||
* Measured: covers all 54 survivors, and refuses none of the ordinary gaps — including em-dash
|
||||
* and ideographic comma, which a "whitespace or ASCII punctuation only" formulation would have
|
||||
* wrongly refused, handing back a false positive on `emma—stone`.
|
||||
*/
|
||||
const GAP_CONTENT = /[\p{L}\p{N}\p{Extended_Pictographic}]/u;
|
||||
|
||||
/**
|
||||
* Blank whitelisted phrases, REFUSING any occurrence whose inter-word gap holds content.
|
||||
*
|
||||
* 🔴 This is the guard that stops a swallow flipping a moderation gate. `[^a-zA-Z0-9]` excludes
|
||||
* only ASCII alphanumerics, so anything non-ASCII a detection list holds — letters, digits,
|
||||
* emoji — is content to the detector and a separator to this pattern. That matters more than
|
||||
* "one term hidden": the nsfw word list GATES the POI and minor sub-checks (`audit.ts` —
|
||||
* `if (!nsfw && !includesNsfw(...)) return false`, and every search caller passes no `nsfw`
|
||||
* argument), so swallowing the only nsfw signal in an input stops those sub-checks running.
|
||||
*
|
||||
* Judged PER OCCURRENCE, chosen rather than defaulted into: refusing the whole text when any
|
||||
* one occurrence is dirty would suppress whitelisting for the entire prompt, which is the false
|
||||
* positive this exists to remove; stripping the whole text when any one occurrence is clean
|
||||
* would license the swallow elsewhere in it. Per-occurrence is also the only rule that stays
|
||||
* correct as the text grows, since nothing bounds how often an entry appears in user input.
|
||||
*
|
||||
* Consequence to know: a refused occurrence leaves the phrase in place, so the detector fires on
|
||||
* THE WHITELISTED PHRASE ITSELF, and the flag names the moderator's own entry rather than what
|
||||
* caused the refusal. Correct, but it reads as the whitelist being broken — which is why
|
||||
* `/moderator/blocklists` says so in the phrase-list descriptions.
|
||||
*
|
||||
* Refusing after the match rather than bounding the separator is deliberate: a bound made this
|
||||
* matcher disagree with `prepareWordRegex` and handed back the false positives on ordinary
|
||||
* punctuation (`emma,,,, stone`). This leaves the pattern identical and only declines an
|
||||
* occurrence whose gap holds content — a gap of punctuation or whitespace, which is every
|
||||
* ordinary case, still strips.
|
||||
*/
|
||||
export function stripBenignPhrasesWith(text: string | undefined, pattern: RegExp | null) {
|
||||
if (!text || !pattern) return text ?? '';
|
||||
return text.replace(pattern, (...args: unknown[]) => {
|
||||
const match = args[0] as string;
|
||||
// Drop the trailing `offset` and `whole string` arguments; the rest are the captured gaps.
|
||||
const gaps = args.slice(1, -2) as (string | undefined)[];
|
||||
const gapHoldsContent = gaps.some((gap) => typeof gap === 'string' && GAP_CONTENT.test(gap));
|
||||
return gapHoldsContent ? match : ' ';
|
||||
});
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: Terms of Service
|
||||
description: The Terms of Service for the model sharing platform Civitai
|
||||
lastmod: 2026-08-13
|
||||
lastmod: 2026-08-19
|
||||
---
|
||||
|
||||
Welcome, and thank you for your interest in Civit AI, Inc. (“Civitai,” “we,” or “us”) and our website at [civitai.com](https://civitai.com), along with our related websites, hosted applications, mobile or other downloadable applications, and other services provided by us (collectively, the “Service”). These Terms of Service are a legally binding contract between you and Civitai regarding your use of the Service.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: Terms of Service
|
||||
description: The Terms of Service for the model sharing platform Civitai
|
||||
lastmod: 2026-08-13
|
||||
lastmod: 2026-08-19
|
||||
---
|
||||
|
||||
Welcome, and thank you for your interest in Civit AI, Inc. (“Civitai,” “we,” or “us”) and our website at [civitai.red](https://civitai.red), along with our related websites, hosted applications, mobile or other downloadable applications, and other services provided by us (collectively, the “Service”). These Terms of Service are a legally binding contract between you and Civitai regarding your use of the Service.
|
||||
|
||||
@@ -850,6 +850,10 @@ export function includesInappropriate(
|
||||
const harmfulCombo = includesHarmfulCombinations(input.prompt);
|
||||
if (harmfulCombo) return harmfulCombo;
|
||||
|
||||
// 🔴 This makes the nsfw word list load-bearing for more than nsfw: the POI and minor checks
|
||||
// below run only if it matches (callers that pass no `nsfw` argument — every search gate —
|
||||
// depend entirely on it). So anything that can suppress a term on that list turns off two
|
||||
// other checks for that input, not just its own.
|
||||
if (!nsfw && !includesNsfw(input.prompt)) return false;
|
||||
|
||||
// Check for harmful combinations first
|
||||
@@ -876,6 +880,8 @@ function includesInappropriateEnriched(
|
||||
const harmfulCombo = includesHarmfulCombinationsEnriched(input.prompt);
|
||||
if (harmfulCombo) return { type: harmfulCombo.type, matchedWord: harmfulCombo.matchedText };
|
||||
|
||||
// Same gate as in `includesInappropriate` above, and the same consequence: the nsfw word
|
||||
// list decides whether the POI and minor checks below run at all.
|
||||
if (!nsfw && !includesNsfw(input.prompt)) return false;
|
||||
|
||||
// Negative prompt harmful combinations
|
||||
|
||||
@@ -206,6 +206,7 @@ export const CACHEABLE_PROCEDURES: ReadonlySet<string> = new Set([
|
||||
'nowPayments.getBuzzConversionRate',
|
||||
'nowPayments.getMinAmount',
|
||||
'nowPayments.getSupportedCurrencies',
|
||||
'system.getBenignPhrases',
|
||||
'system.getCreationBlockedTags',
|
||||
'system.getDbKV',
|
||||
'system.getLiveNow',
|
||||
|
||||
Reference in New Issue
Block a user