Commit Graph

222 Commits

Author SHA1 Message Date
Briant Diehl 779b52702e updated scanner-review styles and policies 2026-05-18 15:21:54 -06:00
Briant Diehl f49c412ef7 fix orphans/misses in wildcard auditing pipeline 2026-05-18 14:44:04 -06:00
Briant Diehl 9a8df1b49f include wildcard snippets in prompt enhancement flow.
add wildcard snippets preview modal with seed value
2026-05-18 13:25:04 -06:00
Briant Diehl 183ee411bb Merge pull request #2277 from civitai/feature/prompt-wildcards
Feature/prompt wildcards
2026-05-15 17:37:09 -06:00
Briant Diehl 11f9d811e1 commit docs related to scanner policy changes 2026-05-15 17:31:35 -06:00
Briant Diehl 07725d8d78 updated xguard labels 2026-05-15 17:27:37 -06:00
Briant Diehl d92a78b3a1 remove snippets if they don't return from fetch 2026-05-15 12:40:32 -06:00
Briant Diehl 9c8e35f199 Merge remote-tracking branch 'origin/main' into feature/prompt-wildcards 2026-05-15 11:52:49 -06:00
Briant Diehl 7f693392c4 data-graph typescript optimizations 2026-05-15 11:52:07 -06:00
Briant Diehl 7d45b356af wildcards: tighten audit gate + audit-on-mutation + retry-job integration
- Audit relaxation flipped from `auditStatus != 'Dirty'` back to
  `auditStatus = 'Clean'` at all three call sites (getWildcardSets,
  getMyUserWildcardSet, expandSnippetsToTargets). Audit fires on every
  User-kind value mutation (saveUserSnippet, updateUserSnippet,
  removeUserSnippet), so Pending categories transition to Clean / Dirty
  quickly enough that the strict gate doesn't starve the picker.

- Paired `nsfwLevel === 0` fallbacks removed from the resolver and
  getResourceData's Wildcards visibility check. Clean categories always
  carry a non-zero `nsfwLevel` (PG default when no `nsfw` label triggered)
  so the bitmask check stands on its own.

- submitWildcardCategoryAudit migrated to the shared
  createXGuardModerationRequest helper (matches the Article path). Passes
  entityType: 'WildcardSetCategory' + entityId: categoryId; keeps the
  ?type=wildcardCategoryValue callback URL override so the webhook
  dispatches to the wildcard branch.

- EntityModeration row created on every wildcard audit submission, mirrored
  on success/failure callbacks. The retry-failed-text-moderation job now
  resubmits failed wildcard categories alongside Article rows -- a
  WildcardSetCategory content resolver was added to bulkContentResolvers,
  and the retry loop dispatches via a per-entityType `resubmitters`
  registry (Article -> submitTextModeration, WildcardSetCategory ->
  submitWildcardCategoryAudit).

- v1.md updated: the "re-tighten audit filters" open question is closed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 15:41:54 -06:00
Manuel Emilio Urena 814c613c0e feat(challenge): add Civitai LLM client + per-function model defaults (#2274)
* feat(challenge): add Civitai LLM client + per-function model defaults

Add an OpenAI-compatible chat completions client targeting the Civitai
Orchestrator so Civitai-hosted models (urn:air:*) can be reached with the
same call shape as OpenRouter. A small dispatcher in generative-content.ts
routes URN-prefixed models to the new client and leaves all other models on
OpenRouter, so freshdesk-agent and other consumers are unaffected.

Per-function defaults in the daily-challenge flow:
- Content generation (collection, article, theme, winners) -> gpt-4o-mini
- Image review -> gpt-5-nano (runs stricter in practice)

Playground UX:
- ModelSelector lists every wired model (mod-only); dropped the freeform
  "Other..." input and dead customModelId store field.
- Store persist version bumped to v3 with migration rewriting stale
  defaults (grok / Qwen URN / gpt-5-nano) to gpt-4o-mini.

Client defenses (documented in docs/features/civitai-llm-client.md):
- Flatten text-only content arrays (Orchestrator validator rejects arrays).
- Append "JSON only" instruction to suppress Qwen3 thinking-mode preamble.
- JSON extraction fallbacks: raw, fenced, first-{ to last-} slice.
- Trailing-slash normalization on ORCHESTRATOR_ENDPOINT.
- Opt-in per-call debug logging.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(challenge): address Copilot review feedback

- docs: clarify pickClient is file-local in generative-content.ts, not an
  importable helper; note the promotion path if a second consumer arrives.
- store: guard migrate against undefined version with (version ?? 0).
- civitai-llm: warn at boot when ORCHESTRATOR_ENDPOINT/TOKEN are missing
  so operators learn about the misconfiguration before a request fails.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(challenge): cache civitaiLLM on globalThis in dev to match openrouter pattern

Survives Next.js HMR module reloads so dev sessions don't churn the client
on every save. Matches the sibling pattern in openrouter.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(civitai-llm): make thinking suppression opt-in per call

The client is meant to be a thin OpenAI-compatible wrapper, not a
Qwen-specific shim. Move the JSON-only preamble injection behind a
`suppressThinking?: boolean` input flag (default false) so callers
decide per-request when it's needed.

Also clarify in generative-content.ts and the feature doc that the
daily-challenge defaults run on OpenRouter (gpt-4o-mini / gpt-5-nano),
and that the civitai-llm dispatcher is available for Playground
experimentation with urn:air:* models — not a pending swap target.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Apply prettier formatting

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 16:54:24 -04:00
Briant Diehl 231e579fa5 Merge remote-tracking branch 'origin/main' into feature/prompt-wildcards 2026-05-14 13:26:20 -06:00
Briant Diehl da1f0d4b23 wildcards: set-level nsfwLevel rollup + audit-driven classification
- WildcardSet.nsfwLevel column (folded into the original add_wildcard_sets
  migration). Bitwise OR of every non-Dirty category's nsfwLevel; maintained
  by recomputeWildcardSetAuditStatus so visibility checks ("does this set
  have content fitting .com SFW vs .red NSFW?") are a single-table bitmask
  test, no category sub-query.

- Audit pipeline classifies severity via XGuard. Two label sets per workflow:
  hard-fail labels (csam, urine, diaper, scat, menstruation, bestiality) ->
  Dirty; level label (nsfw for v1) -> NsfwLevel.R, else NsfwLevel.PG. v1
  restricts to the binary nsfw evaluator because pg/pg13/r/x/xxx aren't
  well-tuned for text yet -- the schema stays bitwise so re-introducing
  finer levels later is code-only.

- Callback recomputes Dirty from per-label triggered flags rather than
  trusting output.blocked, which would otherwise flip on triggered level
  labels (ordinary NSFW content would be marked Dirty).

- getResourceData stamps wildcardSetId + overrides canGenerate for
  Wildcards-type ModelVersions, using the set-level rollup for visibility.
  Single-table query, no JOIN.

- GenerationResource.wildcardSetId optional field -- downstream callers
  (form hydration, model detail page "Generate" handoff) read it to route
  the id into snippets.wildcardSetIds rather than appending to resources[].

- Docs (prompt-snippets-v1.md, prompt-snippets-schema.md) updated to
  describe the rollup, audit label sets, and "wildcards aren't generation
  resources" routing model.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 13:14:28 -06:00
Briant Diehl 51d29d8e8c check in 2026-05-14 11:16:22 -06:00
Briant Diehl 94b7fd31a1 Merge pull request #2265 from civitai/feature/moderation-output
Feature/moderation output
2026-05-13 15:24:29 -06:00
Briant Diehl 44275e48d1 Switches the ClickHouse audit log to AggregatingMergeTree so duplicate
scans collapse into one decision-grain row with occurrences summed.
Mod review tables collapse to a single `ScannerLabelReview` keyed by
the same three columns — verdict once, covers all future identical
scans.

- policyHash -> version end-to-end (column + workflow metadata)
- writer computes contentHash from scan input; standalone
  recordXGuardScan dropped in favor of the workflow helper
- queue + detail queries use GROUP BY with explicit aggregates in
  HAVING/ORDER BY + SETTINGS prefer_column_name_to_alias = 1 for
  partition-pruning WHERE on lastSeenAt
- /moderator/scanner-audit rebuilt around the new row unit with
  occurrence count + per-label verdict buttons
- new docs/features/scanner-pending-migrations.md lists the
  Postgres migration, ClickHouse SQL, and Prisma regen step

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 14:44:08 -06:00
Briant Diehl 69fa598974 refactor(xguard): drop redis-backed policy overrides; use orchestrator policyHash
The original reason for the civitai-side policy override system was that
the orchestrator wasn't surfacing a policy hash, so we couldn't record a
meaningful policyVersion in the audit log. The latest @civitai/client now
includes results[number].policyHash, so we can read it directly and let
the orchestrator stay the source of truth for policy text + thresholds.

- Delete /moderator/xguard-policies page + tRPC router + service + schema
- Remove XGUARD.POLICIES_TEXT / POLICIES_PROMPT Redis key constants
- createXGuardModerationRequest no longer fetches policies from Redis or
  builds labelOverrides; just passes the caller's labels filter through
- recordXGuardScan reads policyHash from each result instead of from
  a metadata-stamped policyVersions map
- @civitai/client bump (typing brings the new field along)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 11:55:39 -06:00
Briant Diehl df30cb9d3e docs(scanner-prompt-tuning): refresh scanner audit design doc
Updates the prompt-tuning design doc to reflect what's actually been
built: the single-table ClickHouse audit log with workflow timing,
mediaRating step output expansion, and the Postgres review tables.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 11:42:15 -06:00
Justin Maier 7ec50531fc refactor(retool-api): gate privileged actions via granted permissions
Drops the bespoke SUPER_ADMIN_USER_IDS env allowlist in favor of the
existing `granted`-availability feature-flag system that already backs
permission-bearing actions like paddleAdjustments, announcements, etc.

- `RetoolAction.privileged` changes from `boolean` to `string` (the
  permission key).
- Wrapper now checks `user.permissions?.includes(action.privileged)`
  instead of the env allowlist.
- New flags `retoolUpdateIdentity` and `retoolToggleModerator` in
  feature-flags.service.ts (both `['granted']`).
- User endpoint actions now name their permission keys.
- SUPER_ADMIN_USER_IDS removed from server-schema.ts.
- Audit row's `privileged` column derives from `Boolean(privileged)` so
  the truthy column behavior is unchanged.
- Test suite expanded to 11 cases (covers no-permissions-array and
  has-other-permission rejection paths plus the happy path).

Grants flow through the standard permission-grant pipeline — no
new env wiring needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 11:17:14 -06:00
Justin Maier 4e01aa5a68 feat(mod): retool-callable API endpoints replace direct DB writes
Adds the Retool API migration foundation and ships endpoints for all 14
moderator workflows that previously ran raw SQL from Retool. Every action
goes through a typed handler, applies a per-action rate limit, and emits
a ClickHouse audit row before responding.

Foundation
- defineRetoolEndpoint + retoolAction registry helper (auth + rate
  limit + dispatch + audit) at src/server/utils/retool-endpoint.ts
- Tracker.retoolAudit() emits to default.retoolAuditLog in ClickHouse
- SUPER_ADMIN_USER_IDS env allowlist gates privileged actions
- RETOOL_ENDPOINT.RATE_LIMIT redis namespace

Endpoints (under src/pages/api/mod/retool/)
- model.ts        bump (Group 13)
- review.ts       setExclude, delete (Group 5)
- comment.ts      bulkDelete, removeAsTos (Groups 3, 4)
- user.ts         clearProfile, mute, unmute, updateIdentity*,
                  toggleModerator* (Groups 1, 6, 7, 8)
- image.ts        tagVote, setNsfwLevel (Groups 11, 12)
- cosmetic.ts     assignByTarget, unassign + Cosmetic CRUD
                  (Groups 9, 10)
- homeblock.ts    create, update, delete, reorder (Group 14)
  * privileged — requires SUPER_ADMIN_USER_IDS allowlist

Group 2 (UserLink cleanup) folded directly into the ban pipeline in
user.service.toggleBan rather than exposed as an endpoint.

New service functions back every action; no endpoint touches dbWrite
directly. Highlights:
- bumpModel with the 3-call cache fan-out
- bulkSetCommentTosViolation + new bulkSetCommentV2TosViolation
  mirroring the legacy setTosViolationHandler flow
- forceUpdateUserIdentity bypasses the email-overwrite guard
- assignCosmeticByTarget resolves collection / userIds targets and
  supports dryRun previews

Tests: 10 vitest cases on the registry helper cover auth, role gate,
privileged allowlist, rate limit, schema validation, and audit
emission (success + error paths).

Docs: docs/features/retool-api.md (auth, request shape, examples,
audit schema, how-to-add-a-new-action). Plan doc with full design
discussion in docs/plans/retool-api-migration.md.

Closes the parent migration ticket
https://app.clickup.com/t/868jk3qh8

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 20:04:25 -06:00
Briant Diehl 3eda440637 use custom xguard labels when submitting xguard requests 2026-05-12 17:22:18 -06:00
Briant Diehl f4f8eab8eb initial updates to include additional mediaRating flags 2026-05-12 17:07:09 -06:00
Briant Diehl c471d6015f added hidream-o1 ecosystem and generation support 2026-05-12 12:34:32 -06:00
Briant Diehl cd29f1eaaa typed feature flags 2026-05-12 12:03:45 -06:00
Briant Diehl e1cf521399 only return feature flag keys if value is true 2026-05-12 11:38:22 -06:00
Briant Diehl 9f42d7d5cb normalize feature flag calls 2026-05-12 10:47:21 -06:00
Briant Diehl 92333d539b fixes to align with original docs/design 2026-05-11 16:32:51 -06:00
Manuel Emilio Urena 178bb19cc4 chore(model-file-scan): remove legacy HTTP scanner path (Phase 3) (#2253)
* chore(model-file-scan): remove legacy HTTP scanner path (Phase 3)

The orchestrator scan workflow has been at 100% rollout — drop the
legacy `scanFilesJob` / `requestScannerTasks` / `scan-result` webhook
and the `MODEL_FILE_SCAN_ORCHESTRATOR` Flipt gate so the codebase
commits fully to the new path.

- Delete `src/pages/api/webhooks/scan-result.ts` and its test
- Gut `scan-files.ts` down to `scanFilesFallbackJob` (cron key kept as
  `scan-files-fallback` for operational continuity)
- Drop `SCANNING_ENDPOINT` / `SCANNING_TOKEN` from env schema
- Remove `MODEL_FILE_SCAN_ORCHESTRATOR` flag + all gates in
  `createFileHandler`, `rescanModel`, `clean-up.ts`, and the scan job
- Update affected tests; doc Phase 3 checklist marked complete

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(model-file-scan): rewrite doc for post-Phase-3 steady state

Address PR review feedback:
- Drop "Rollout Strategy", "TODO Tracker", and "Webhook Comparison"
  sections — they described the now-removed Flipt gate and side-by-side
  legacy/orchestrator coexistence
- Consolidate the two architecture diagrams into a single steady-state
  diagram
- Refresh "Key Implementation Files" so each entry describes the file's
  current responsibility (no more "MODIFIED" / "NEW" rollout markers)
- Add an "Admin Clean-Up Behavior" callout: clean-up.ts now submits the
  full orchestrator workflow per file (not the legacy Hash + ParseMetadata
  subset). This consolidation was vetted during the Phase 2 canary; doc
  it explicitly per Copilot review feedback.
- Move migration narrative + decisions + known-gap notes into a single
  "Migration History" appendix at the end

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 14:20:08 -04:00
Briant Diehl 2f42cc972a Merge remote-tracking branch 'origin/main' into feature/prompt-wildcards 2026-05-08 15:52:12 -06:00
Briant Diehl ebdadfd6cb snippets in generation form 2026-05-08 15:22:32 -06:00
Justin Maier 055e54f6e6 feat(oauth): scoped tokens, OAuth 2.0 server, per-subject buzz limits
Squashed merge of feature/scoped-tokens onto latest main.

OAuth 2.0 server (authorization code + PKCE, refresh, revoke, device
flow, OIDC discovery), bitwise TokenScope enum (25 flags) with a
fail-safe enforceTokenScope middleware (un-annotated procedures
default to requiring Full), 83 routers annotated, 15 buzz-spending
procedures gated with blockApiKeys: true.

Per-subject buzz limits: opaque (type, id) subject pair,
BuzzBudget[] shape supporting absolute/sliding/rollover variants
with optional currency filters, stored on ApiKey.buzzLimit
(User-type keys) or OauthConsent.buzzLimit (OAuth grants — stable
across access-token rotations). Civitai stores limits + busts
cache + cleans up subjects via /v1/manager/users/:userId/{auth,
limits/auth}/:type/:id; orchestrator owns enforcement and
rolling-window math.

Account UI: card-based ApiKeys, OAuthApps, ConnectedApps surfaces
with inline spend bars + a shared EditBuzzLimitModal. OAuth consent
screen collects an optional buzz limit when AIServicesWrite is
requested. OAuth Apps + Connected Apps gated behind the
`oauth-apps` Flipt flag (mod-only). Audit via the existing
ClickHouse `actions` table (BuzzLimit_Set ActionType).

DB: one new migration 20260507165710_add_buzz_limit_to_oauth_consent
adds OauthConsent.buzzLimit JSONB. Legacy KeyScope[] column drop is
deferred to a follow-up PR after this is stable in prod.

Demo client: civitai/civitai-oauth-demo (separate repo).

Conflict resolution during the rebase onto main: kept main's newer
multi-image candidate handling in comics.router.ts (it was a
substantive content divergence, not a metadata conflict; the
blockApiKeys: true annotation on purchaseChapterAccess was already
preserved through the non-conflicting merge regions). Prisma types
regenerated post-merge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 11:07:41 -06:00
Briant Diehl 7bc60e7621 model file scanning workflow (#2175)
* model file scanning workflow

* phase 1: flag-gated orchestrator scan path with legacy parity

Migrates model-file scanning from the legacy HTTP scanner to orchestrator
workflows behind the MODEL_FILE_SCAN_ORCHESTRATOR Flipt flag. Ships with
flag OFF; canary rollout is Phase 2.

- New shared applyScanOutcome() service consumed by both webhook adapters
  guarantees zero behavioral drift between paths during rollout
- Legacy /api/webhooks/scan-result rewritten as a thin adapter
- Inline submission in createFileHandler when flag is ON; fallback cron
  scanFilesFallbackJob handles stalled/missed scans
- All call sites (createFileHandler, both jobs, rescanModel, clean-up.ts)
  gated on the flag with explicit OFF/ON behavior — no double-submit risk
- Decisions D1-D5 baked in (see docs/features/model-file-scanning.md):
  rawScanResult normalized envelope, hash-blocking kept commented for
  parity, model-hash-fix synthesized from AutoV2 diff, failed-scan
  natural 24h backoff, dataForModelsCache.refresh() proactive re-warm
- Production-safe dev-skip in createModelFileScanRequest (only fires in
  non-prod with missing token; never silently fake-succeeds in prod)
- Full Axiom telemetry across all paths for canary skew comparison
- moved unpublishBlockedModel from the legacy webhook to model.service.ts
  so retroactive-hash-blocking has a stable import after Phase 3 deletion

Phase 3 (legacy webhook + scanFilesJob + SCANNING_ENDPOINT env var
deletion) ships in a separate PR after the flag has been at 100% for
>=1 week.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(model-scan): align doc with shipped code; resolve cosmetic 🟢 items

Doc audit found three places where the doc disagreed with the code, plus
a few items that could be marked resolved now:

- D4 description corrected: scanRequestedAt is bumped to now() (24h
  backoff via the stale-cutoff path), not reset to null. The code review
  fix changed this to prevent tight retry loops on permanently-broken
  AIRs; the doc still described the older behavior.
- Key Implementation Files table: added the 4 missing entries
  (model.service.ts, clean-up.ts, retroactive-hash-blocking.ts,
  flipt/client.ts) and updated descriptions to reflect what shipped.
- Migration from Legacy System: past-tense the work that's already done
  and clarified that scanFilesFallbackJob is the orchestrator-side
  counterpart of scanFilesJob.
- Existing Result Processing to Preserve: /⚠️/🟢 status added per item;
  fixed the cache-method reference (legacy used refresh, not bust).
- admin/test.ts checkbox: reflects reality (kept as scan-tester for
  manual canary verification, not reverted).
- Resolved 3 🟢 items as decisions:
  - Convert/Import task drops — already done, no longer "worth
    considering"
  - 404 vs 200 on missing file — keep-as-is decision; 200 prevents
    upstream retry storms on legitimately-deleted files
- Removed stale Phase-1-blocker warning about double-submit (gate is in
  place).

No code changes — doc-only follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* address copilot review: fix double-submit, submission-failure, scanned semantics

Six issues from the Copilot PR review, all valid:

1. Dev-skip guard was OR-instead-of-AND: in prod with missing token we
   would silently fake-success virus scans, and in dev with a working
   token we would never run real scans. Restored to AND so prod always
   surfaces real submitWorkflow errors and dev can test the new flow
   when configured.

2. createModelFileScanRequest now throws on submission failure instead
   of silently returning undefined. Several callers had try/catch +
   .catch() that never fired because the await resolved successfully
   even when submission failed, leading to broken `failed` counters and
   missing Axiom error logs in createFileHandler.

3. createFileHandler inline path was leaving scanRequestedAt=null,
   which caused scanFilesFallbackJob's next 5-min tick to re-submit
   workflows for newly uploaded files before the webhook callback
   arrived. createModelFileScanRequest now sets scanRequestedAt=now()
   immediately after a successful submitWorkflow so concurrent paths
   don't duplicate work.

4. scanFilesFallbackJob marked the entire batch scanRequestedAt=now()
   upfront. If submission then failed (transient orchestrator outage),
   files would wait for the 24h stale-cutoff before retrying — far too
   long for the common case. Catch handler now resets scanRequestedAt
   to null on submission failure so the next tick retries. The
   workflow-level failure path (D4) keeps its 24h backoff for
   permanently-broken AIRs.

5. scanFilesFallbackJob was missing the null-guard for soft-deleted
   modelVersion that rescanModel and clean-up.ts already had.

6. applyScanOutcome unconditionally set scannedAt=new Date(). Legacy
   callers that pass only [Hash, ParseMetadata] (e.g. clean-up.ts)
   would mark files as "scanned" without any virus check having run.
   scannedAt now only advances when virusScan or pickleScan is in the
   outcome — matches legacy semantics where scannedAt only set when the
   Scan task ran.

Doc updated to reflect new behavior in the Operational Hygiene
checklist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(model-scan): consolidate scan service, extract rescanModel and unpublishBlockedModel

Rename model-file-scan-result.service.ts -> model-file-scan.service.ts so the
file owns the full scan lifecycle (dispatch + result), not just result handling.
Move rescanModel and unpublishBlockedModel out of the now-overgrown model.service.ts
and into the renamed file. Update importers directly (no re-export shim) so the
former bidirectional model.service.ts dependency cleanly becomes one-way.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(model-scan): add unit tests covering the orchestrator scan pipeline

Adds 94 tests across four files locking in behavior of the model-file scan
flow during the legacy/orchestrator rollout window:

- model-file-scan.service.test.ts (54): examinePickleImports, applyScanOutcome,
  processModelFileScanResult, rescanModel (orchestrator + legacy branches),
  unpublishBlockedModel
- webhooks/scan-result.test.ts (18): legacy ScanResult -> ScanOutcome translation
  parity (Scan/Hash/ParseMetadata branches, exit-code mapping, hasDanger override,
  case-insensitive hash keys, ss_tag_frequency quirk, error path)
- jobs/scan-files.test.ts (10): scanFilesJob and scanFilesFallbackJob flag-branch
  gating, batch-upfront marking, soft-deleted modelVersion handling, submission-
  failure scanRequestedAt reset
- orchestrator/createModelFileScanRequest.test.ts (12): dev-skip gate, workflow
  shape, scanRequestedAt-on-success, throw-on-submission-failure

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(model-scan): make admin scan-tester accept modelVersionId via query

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(model-scan): thread fileId through scan AIR + mini endpoint

Bump @civitai/client to 0.2.0-beta.56 for Air.stringify's modelFileId
field. stringifyAIR now accepts an optional fileId, emitted as
`+<fileId>` so the orchestrator can disambiguate among multiple files
attached to the same modelVersion — addresses the duplicate-hash issue
where two files under one version received identical hashes because
the URN only encoded modelId@versionId.

Mini endpoint accepts ?modelFileId=N to return that exact file's
download url, hashes, size, and an AIR ending in `+<fileId>`. Falls
back to getPrimaryFile when omitted, preserving legacy behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Cleanup

* fix(model-scan): handle orchestrator status enum + fix hash field casing

Orchestrator now reports clamScan/pickleScan results via a `status` enum
and explicit boolean flags instead of POSIX exit codes; derive scan
results from those, with exitCode as a legacy fallback. Also corrects
hash output field names (sha256/crc32) to match orchestrator output.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(model-scan): tombstone files via pre-flight URL resolution

Restore the legacy `requestScannerTasks` parity gap: when a file can't be
fetched, mark `ModelFile.exists=false` so the scan jobs stop polling it
forever. submitWorkflow only enqueues, so we mirror legacy and run a
pre-flight `resolveDownloadUrl(fileId, url)` ourselves with a 60s
sync-lag retry. Failure throws a typed `ModelFileScanSubmissionError`
with `code: 'not-found'`; submission failures are `'transient'`.

Caller policy is asymmetric on purpose:
  - scanFilesFallbackJob, rescanModel, clean-up.ts → run pre-flight; on
    'not-found' tombstone exists=false, on 'transient' reset
    scanRequestedAt=null for the next 5-min retry.
  - createFileHandler → preflight=false to avoid blocking the upload
    response on the 60s retry; the fallback job catches genuine misses
    five minutes later.
  - api/testing/model-file-scan.ts → catch + report, no DB write.

Also moves the scan tester from /api/admin/test.ts to
/api/testing/model-file-scan.ts per project convention, with a
doc-block header listing params and behavior.

Auto-unpublish-on-missing-file is consciously not ported. Legacy
unpublish() lived in scan-result.ts but only fired when
tasks.includes('Import'), and Import was confirmed dead in production
(no caller ever passed it). Tombstoning is the load-bearing piece;
unpublishing creator content on a single resolution failure is a
separate product decision, not parity.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(model-scan): wire ModelFileScanSubmissionError into rescanModel test

The rescanModel orchestrator-path test was importing only
createModelFileScanRequest from the mocked orchestrator module, but
rescanModel now also references ModelFileScanSubmissionError for the
not-found tombstone branch. Add a test-local class through vi.hoisted
so the `instanceof` check resolves, and update the call-shape
assertion to include the new `url` arg.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(model-scan): note orchestrator AIR-resolution limitation

Orchestrator team confirmed they cannot add an AIR-not-found signal at
submit time. Pre-flight resolveDownloadUrl is therefore the only
not-found detection — same coverage legacy had. Document the residual
gap (file present in storage but unfetchable by orchestrator) and
follow-up options if it surfaces in production.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Manuel Emilio Urena <manuel.ureh@hotmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:47:32 -04:00
Briant Diehl e4f8d8e700 add wildcard resolver service 2026-05-07 17:09:15 -06:00
Briant Diehl 95d84c2526 WildcardSet services 2026-05-07 15:22:17 -06:00
Briant Diehl 8f5d58d139 job to scan wildcard models to create new wildcard sets 2026-05-07 14:14:30 -06:00
Briant Diehl 9abd492aa3 update prompt snippets docs with v1 plan 2026-05-05 15:06:59 -06:00
Briant Diehl 7c21953c38 update prompt snippet docs 2026-05-05 11:01:48 -06:00
Briant Diehl 86f1d61db9 updated prompt-snippets docs 2026-05-05 10:17:40 -06:00
Briant Diehl d12736580e update prompt-snippets docs 2026-05-04 14:50:12 -06:00
Briant Diehl 6f3a7d03f3 update Gated content logic and sitemaps 2026-05-04 12:57:37 -06:00
Briant Diehl f040ed6b7f remove unused middleware 2026-05-01 11:56:21 -06:00
Briant Diehl 0caf12220c create bot detection middleware 2026-04-30 16:51:06 -06:00
Briant Diehl b92effcb30 move registered topics to zustand store 2026-04-30 15:42:07 -06:00
Briant Diehl 241ca6ff2c update doc 2026-04-29 16:14:39 -06:00
Briant Diehl 35e87a09a8 Merge branch 'main' of https://github.com/civitai/civitai 2026-04-29 16:01:01 -06:00
Briant Diehl 7f80816adf server logging 2026-04-29 16:00:49 -06:00
Justin Maier 26099bea73 Merge pull request #2221 from civitai/worktree-multi-host-domain-aliases
feat(domain): multi-host aliases per ColorDomain
2026-04-29 15:24:57 -06:00
Justin Maier 6d5f24d019 docs(domain): sync scoping doc with login flow change
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 14:25:22 -06:00
Briant Diehl a956582c80 turn ace audio back on 2026-04-29 10:28:10 -06:00
Briant Diehl c199edba20 Merge branch 'main' of https://github.com/civitai/civitai 2026-04-28 17:00:33 -06:00