mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
Justin Maier
c930757e2c
* feat(cosmetic-phash): widen the lane to 256 bits, and make the badge tunable At 64 bits the near-match panel could not separate a copy from a coincidence. Measured over the 1,719 hashable cosmetics, the two badges reported as imitations of official artwork ranked 7th and 116th of their corpus against a 1st percentile of 18 — inside the noise. The orchestrator now offers `perceptualDct256`, and at 256 bits the same two rank 1st and 5th. Both were confirmed by eye, along with three pixel-identical re-uploads of official badges that the submission-time sha256 cannot see, because official cosmetics carry no `imageHash`. The sweep drains the corpus on `pHashVersion`, so bumping the lane is the backfill. `COSMETIC_SIMILARITY_CLOSE_RATIO` is unchanged at 0.125, and that is a finding rather than an omission: it selects the same five cross-creator pairs anywhere between 8 and 40 of 256, so the value sits on a plateau rather than an edge. Being a fraction of the width is what carried it across the upgrade. Three things that were silent before: - A lane bump that moved only two of the three fields left the third disagreeing, and nothing failed. Two of those three mutations passed the suite. `COSMETIC_PHASH_LANE` is now asserted to spell one lane in all three fields, which kills both. - The near-identical decision moved out of the component. It is now decided server-side and carried on the match, because the threshold is operator- tunable at runtime and a client recomputing it from a bundled constant disagrees with the server until every tab reloads. - The threshold reads from a `KeyValue` row so it can be moved without a deploy. Read per call rather than memoised — a TTL would delay the change by the length of the TTL, which is the whole point of the row. It degrades to the built-in default on a malformed or out-of-range value rather than throwing, because this gates a badge and a throw would take out the ranking underneath it. Out-of-range is rejected, not clamped: clamping 1.5 to 1 would badge every match as near-identical and read as a working threshold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(cosmetic-phash): close the gaps the five review lanes found The knob shipped wired to nothing, and the tests said otherwise. Two mutations survived the previous commit's suite: replacing the new `close` expression with a constant, and dropping the KeyValue read in favour of the built-in default. Both passed 36/36. `close` had no assertion at either layer, and every `getSimilarCosmetics` test reset the KeyValue mock to undefined, so the whole suite only ever exercised the fallback path. One test now drives the ranking at the default ratio and again at an operator-set one, and both mutations die. `getCosmeticSimilarityCloseRatio` guarded the value it read but not the read. An unguarded rejection escapes `getSimilarCosmetics` AFTER the ranking is complete, and the card renders that as "this artwork was not compared against anything" — false, and the exact confusion the card exists to remove. The read is wrapped; the docblock now describes what the code does. Three of the five malformed-value cases were passing on residue. `loggingMock.logToAxiom` is reset once per FILE, so a later case was satisfied by the first case's warning. With a per-test clear and `toHaveBeenCalledTimes`, a mutant that returns the default silently kills five tests instead of two. Dropped a test that asserted a ratio it claimed was "what the UI thresholds on". It wasn't: both imitation pairs sit above the shipped 0.125, so it checked nothing the neighbouring test didn't already pin exactly, and it hardcoded /256 so the next lane bump would have divided by the wrong denominator in silence. Boundary corrected. `0` is now accepted — "only an exact match is near-identical" is a coherent request, and rejecting it fell back to the LOOSER built-in, i.e. more red badges than the operator asked for. `1` is rejected instead, being the same degenerate outcome `1.5` was already refused for. `normalizeCosmeticHashHex` now refuses a hash wider than the lane. `padStart` returns an over-wide hash unchanged, so it would be stored at full width under the current version — accepted as a comparison target, then excluded from every candidate set by the length filter, reporting "nothing was close" forever on a row that looks correctly hashed. Adds `scripts/oneoffs/drain-cosmetic-phash-lane.ts`. The mitigation the PR depends on — crossing the two-lane window in minutes rather than the sweep's ~2h15m — existed only as a scratch file. It loops the tested sweep rather than issuing its own UPDATE, so the rules about what a correct row looks like stay in one place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(cosmetic-phash): make a refused hash say so, and stop the drain script hiding a timeout The previous commit added a silent failure while removing silent failures. `normalizeCosmeticHashHex` now throws on an over-wide hash, and the sweep's catch was a bare `} catch {` with no log — so a lane whose `hexLength` was left behind would hash every row (billed), throw at store, count them all as `failed`, stamp them, and retry once a day forever, with nothing anywhere distinguishing that from dead artwork. The catch now names the row and the reason. It logs fire-and-forget with its own `.catch`, because a logger that throws inside a catch converts a diagnosable failure into a lost one. The drain script asserted a cause it cannot know. `failed` counts rows the sweep could not store this run, and three different things land in it identically: dead CDN artwork, an orchestrator still working when the 30s wait elapsed, and a hash the store refused. The old closing line called all of them "permanently unhashable", which is a guess printed in the voice of a measurement — and the operator's next move differs for each. It now says what was counted and points at the per-row log. Dropped the drain script to the cron's 200/5 from 500/10. `getPerceptualHash` returns `undefined` for both a real failure and a workflow still running, and the sweep stamps either for 24h. More concurrency means more timeouts, so the script written to shorten the window a row spends outside the lane could have EXTENDED it to a day for a slice of the corpus — then printed "done", because a stamped row drops out of the predicate and the next batch comes back empty. The speed was always in removing the 15-minute gap, not in working a tick harder. `MAX_BATCHES` 40 -> 20 with the smaller batch, ~2x the corpus rather than 11x. Corrected the lane docblock, which still described the pre-guard symptom: a stale `hexLength` no longer reaches a comparison-time throw, it fails at the store and writes nothing. The new log path is asserted rather than assumed — deleting it turns the sweep suite red, with the unmutated control green either side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(cosmetic-phash): say what the drain script's failure count can and cannot tell you Comment and console output only; no logic. The closing line has now asserted a cause twice and been wrong both times, in opposite directions. First it called every failure "permanently unhashable". Then, correcting that, it claimed the cause "is not recorded per row" and sent the operator away — but two of the three paths to an undefined hash DO log, and so does the store-refused throw. That version would have had an operator ignore records that exist, including the one class whose correct action is neither "nothing" nor "re-run" but "fix the lane". The coverage is now spelt out rather than summarised, because summarising it is what went wrong twice: a relative media url and any network error or abort log as `perceptual-hash`; a refused width logs as `cosmetic-phash-sweep`; a workflow that simply did not succeed logs nothing, and that silent case is both the 30s timeout and dead artwork — the two most likely reasons a row is in the count. Also drops "re-run tomorrow" for "re-run once 24h have elapsed". The stamp is a 24-hour window, not a calendar day, so a next-morning re-run of an overnight drain finds every row still suppressed, breaks immediately on an empty batch and prints a clean zero — indistinguishable from a completed drain, which is the silent-failure shape this script keeps trying not to reproduce. And it no longer calls a repeat failure dead artwork. A width that disagrees with the lane fails identically on every run forever, an unset NEXT_PUBLIC_IMAGE_LOCATION fails for every row, and timeouts are correlated rather than independent draws — so "failed twice" does not imply "dead". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>