mirror of
https://github.com/conorbronsdon/avoid-ai-writing.git
synced 2026-09-19 01:32:11 +08:00
proof: the false-positive rate exists; say so and say why it isn't a claim (#78)
PROOF.md said 'there is no false-positive rate on this page, because we have not measured one' while corpus/README.md, same version and same date, published the full FPR/TPR table with confidence intervals. Both cannot be true, and the page asserting the absence is the one people read first. The honest version is narrower than either: the rate is measured, and it is withheld as a headline because it fails this repo's own publication gate on the register test. That is a stronger statement than 'we have not measured one' -- it names the number, names the gate, and names which clause the number fails. npm run self-scan:check: PASS (PROOF.md 13 exempt against a budget of 20). Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -107,10 +107,24 @@ slightly changes document-level metrics that are computed over the whole text
|
||||
(em-dash rate, type-token ratio, bold-phrase count), because the denominator
|
||||
shrinks. Vocabulary and phrase categories are unaffected.
|
||||
|
||||
There is no false-positive rate on this page, because we have not measured one.
|
||||
Until that exists, no accuracy claim on this repo should be read as measured.
|
||||
A false-positive rate has been measured, and it is not quoted here as a claim.
|
||||
[`corpus/README.md`](corpus/README.md) publishes the full table against 875
|
||||
human and 779 machine paragraphs: at `score >= 5`, 4.2% FPR (95% CI 3.1–5.8)
|
||||
against 7.2% TPR. Paragraph-level ROC-AUC is 0.501 pooled — a coin flip — and
|
||||
0.623 at document level. Read plainly, the composite score cannot reliably
|
||||
separate machine text from human text, and no threshold on it buys a useful
|
||||
true-positive rate at a tolerable false-positive cost.
|
||||
|
||||
The intake for it is open: if a rule fires on writing a person wrote, the
|
||||
Those numbers stay in the corpus write-up rather than becoming a headline
|
||||
because they do not clear this repo's own publication gate. The gate requires
|
||||
each claim cell to carry n >= 100, a confidence interval, and more than one
|
||||
register people actually write in today. The current run satisfies the first
|
||||
two and **fails the register test outright**: HC3 is ChatGPT only, RAID is a
|
||||
different task shape, and nothing in either is a modern instruction-tuned model
|
||||
writing a LinkedIn post. A rate measured on 2022–2024 assistant prose is an
|
||||
upper bound on performance against current models, not an estimate of it.
|
||||
|
||||
The intake to fix that is open: if a rule fires on writing a person wrote, the
|
||||
[false-positive report](https://github.com/conorbronsdon/avoid-ai-writing/issues/new?template=false_positive.yml)
|
||||
form collects the shortest text that fires, the register, how the text was
|
||||
actually written, and whether it can become a public fixture. Register is the
|
||||
|
||||
Reference in New Issue
Block a user