fix: trace corpus attachment changes and preserve ordinal prose

Pair attached headings with unique exact legacy body spans while preserving source identities and recording absorbed skipped headings. Keep mid-paragraph ordinals in prose, including indented-wrap and real code-boundary distinctions. Add continuation provenance, separate preprocessor/harness fingerprints, legacy detector-delta regressions, and historical measurement labels.

All 18 suites pass. Independent Sol review and verified 38-source comparisons preserve headline scoring metrics and expose the corrected diagnostic change buckets. Follows Claude's final #290 findings.

Fixes #293.

Co-authored-by: Codex <codex@openai.com>

CHANGELOG.md

@@ -17,6 +17,7 @@ All notable changes to this project are documented here.
 
 ### Fixed
 
+- Keep mid-paragraph years and other ordered markers above one in prose during false-positive measurement; expose blank-separated continuation merges and distinct measurement/preprocessor fingerprints; and pair attached headings with their unique legacy body span in comparison output without changing source spans or unit IDs (#293).
 - Preserve Markdown structure and document content during corpus measurement. Separate structural cleanup from paragraph selection, retain eligible 400-word bodies after headings, and account for oversized units without silently deleting text. Keep tab-indented fences atomic, reject source hash mismatches, and construct scored rows from the verified source snapshot (#178, #179, #180, #288, #289).
 - Report top detection categories for documents the self-scan scores in chunks. `scoreLongText()` now counts issue types across every accepted chunk, so `scanFile()` returns a populated `topTypes` for a chunked file and the `--check` over-budget diagnostic names categories instead of printing `none` (#264).
 - Validate CLI `--unit` argument in `scripts/fp-measure.js` before starting measurement, exiting with code 2 on missing, unrecognized, or repeated values, and on `--unit=VALUE` syntax (`paragraph` and `document` accepted).

PROOF.md

@@ -109,12 +109,15 @@ slightly changes document-level metrics that are computed over the whole text
 shrinks. Vocabulary and phrase categories are unaffected.
 
 A false-positive rate has been measured, and it is not quoted here as a claim.
-[`corpus/README.md`](corpus/README.md) publishes the full table against 875
-human and 779 machine paragraphs: at `score >= 5`, 4.2% FPR (95% CI 3.1–5.8)
-against 7.2% TPR. Paragraph-level ROC-AUC is 0.501 pooled — a coin flip — and
-0.623 at document level. Read plainly, the composite score cannot reliably
-separate machine text from human text, and no threshold on it buys a useful
-true-positive rate at a tolerable false-positive cost.
+[`corpus/README.md`](corpus/README.md) publishes the legacy-preparation table
+against 875 human and 779 machine paragraphs: at `score >= 5`, 4.2% FPR (95% CI
+3.1–5.8) against 7.2% TPR. Paragraph-level ROC-AUC is 0.501 pooled — a coin flip
+— and 0.623 at document level. The legacy and repaired paths were compared
+with a pinned detector in the immutable [#290 comparison
+evidence](https://github.com/conorbronsdon/avoid-ai-writing/tree/39accce14131723c796ee640d88c3fe1b0223815/corpus/reports/fp-preprocessing-86ef5ab3).
+Read plainly, the composite score cannot reliably separate machine text from
+human text, and no threshold on it buys a useful true-positive rate at a
+tolerable false-positive cost.
 
 Those numbers stay in the corpus write-up rather than becoming a headline
 because they do not clear this repo's own publication gate. The gate requires

corpus/README.md

@@ -43,14 +43,20 @@ Preparation has three steps: classify source lines, join ordinary prose wraps,
 then select units. Explicit Markdown headings, blockquotes, lists, indented
 code, and complete fenced regions retain their line structure. CRLF and lone
 CR become LF. Fences stay intact across blank lines, including an unclosed
-fence that runs to the end of the document.
+fence that runs to the end of the document. A bullet or an ordered marker whose
+start number is 1 can interrupt prose; another ordered marker starts a list at a
+block boundary, following the relevant [CommonMark list-item
+rule](https://spec.commonmark.org/0.31.2/#list-items). Thus a hard-wrapped line
+that starts `1859.` remains prose.
 
 Document mode preserves the ordered words and markers in the input; it applies
 no paragraph word limit. Paragraph mode retains bodies of 50–400 whitespace
 tokens. The nearest preceding heading attaches when the combined unit fits;
 otherwise an eligible body is scored alone. Oversized bodies are skipped with
 a reason, without being split. Multiple headings cannot consume each other
-and discard the body.
+and discard the body. Selection counts literal source tokens, including ATX
+markers such as `##` and setext underlines such as `=====`. It does not strip
+heading syntax at the 50- and 400-token boundaries.
 
 A short initial line ending in a colon can still be inferred as a heading.
 An immediate lowercase prose continuation prevents that inference; explicit
@@ -79,11 +85,14 @@ The destination must be a new file. The first JSONL record contains schema
 version, Git revision, manifest and code fingerprints, verified source hashes,
 detector options, and totals. Subsequent records include selected and skipped
 units: original source spans, row identity, class, register, model, structural
-kinds, heading attachment, input and detector word counts, score, categories,
-and skip reason. Spans use JavaScript UTF-16 offsets into the original row,
-with an inclusive start and exclusive end. Unit IDs use source identity and
-spans; indexes are only local ordering hints. Normalized text has its own hash.
-Unavailable sources have separate records and do not count as skipped units.
+kinds, heading attachment, optional blank-separated continuation-merge
+provenance, input and detector word counts, score, categories, and skip reason.
+Spans use JavaScript UTF-16 offsets into the original row, with an inclusive
+start and exclusive end. Unit IDs use source identity and spans; indexes are
+only local ordering hints. Normalized text has its own hash. The shared
+measurement-harness fingerprint is separate from the branch-specific legacy
+and current preprocessor fingerprints. Unavailable sources have separate
+records and do not count as skipped units.
 
 The comparison command runs both preprocessing paths with the same detector
 and available corpus. Its legacy path reproduces main at `fabd62d9`:
@@ -201,6 +210,10 @@ current models, not an estimate of it. Nothing here has seen a 2026 model.
 
 ## Results (v3.22.0, 2026-07-31)
 
+These results used the historical whitespace-flattening preparation path. The
+legacy and repaired paths were compared with a pinned detector in the immutable
+[#290 comparison evidence](https://github.com/conorbronsdon/avoid-ai-writing/tree/39accce14131723c796ee640d88c3fe1b0223815/corpus/reports/fp-preprocessing-86ef5ab3).
+
 875 human paragraphs, 779 machine paragraphs.
 
 | Threshold | FPR (95% CI) | TPR (95% CI) |

scripts/fp-compare.js

@@ -144,6 +144,15 @@ function identity(record) {
   ]);
 }
 
+function rowIdentity(record) {
+  return JSON.stringify([
+    record.doc,
+    record.rowId,
+    record.rowIndex,
+    record.rowSourceHash,
+  ]);
+}
+
 function diagnostic(record) {
   if (!record) return null;
   return {
@@ -162,6 +171,7 @@ function diagnostic(record) {
     reason: record.reason,
     headingAttached: record.headingAttached,
     headingKind: record.headingKind,
+    mergedContinuation: record.mergedContinuation,
     kinds: record.kinds,
     inputWords: record.inputWords,
     detectorWords: record.detectorWords,
@@ -173,14 +183,15 @@ function diagnostic(record) {
 
 function changedFields(a, b) {
   const fields = [
-    'normalizedHash', 'selectionStatus', 'status', 'reason', 'headingAttached',
-    'headingKind', 'kinds', 'inputWords', 'detectorWords', 'detectorStatus',
+    'spans', 'normalizedHash', 'selectionStatus', 'status', 'reason', 'headingAttached',
+    'headingKind', 'mergedContinuation', 'kinds', 'inputWords', 'detectorWords', 'detectorStatus',
     'score', 'types',
   ];
   return fields.filter((field) => !sameJson(a[field], b[field]));
 }
 
 function changeImpact(change) {
+  if (change.absorbedInto) return 'segmentation';
   if (change.kind !== 'modified') return 'population';
   const detectorFields = new Set([
     'selectionStatus', 'status', 'reason', 'detectorWords', 'detectorStatus',
@@ -202,24 +213,87 @@ function changePriority(change) {
 
 function changeSet(legacyRecords, currentRecords, exampleLimit = 40) {
   const units = (records) => records.filter((record) => record.recordKind === 'unit');
-  const before = new Map(units(legacyRecords).map((record) => [identity(record), record]));
-  const after = new Map(units(currentRecords).map((record) => [identity(record), record]));
-  const keys = [...new Set([...before.keys(), ...after.keys()])].sort();
+  const beforeRecords = units(legacyRecords);
+  const afterRecords = units(currentRecords);
+  const before = new Map(beforeRecords.map((record) => [identity(record), record]));
+  const after = new Map(afterRecords.map((record) => [identity(record), record]));
+  const pairedBefore = new Set();
+  const pairedAfter = new Set();
+  const attachmentPairs = [];
   const changes = [];
 
-  for (const key of keys) {
-    const a = before.get(key);
+  // Preserve exact-span matching as the first and strongest identity rule.
+  for (const [key, a] of before) {
     const b = after.get(key);
-    if (!a) changes.push({ kind: 'added', key: JSON.parse(key), legacy: null, current: diagnostic(b) });
-    else if (!b) changes.push({ kind: 'removed', key: JSON.parse(key), legacy: diagnostic(a), current: null });
-    else {
-      const fields = changedFields(a, b);
-      if (fields.length) changes.push({ kind: 'modified', key: JSON.parse(key), fields, legacy: diagnostic(a), current: diagnostic(b) });
+    if (!b) continue;
+    pairedBefore.add(a);
+    pairedAfter.add(b);
+    const fields = changedFields(a, b);
+    if (fields.length) changes.push({ kind: 'modified', key: JSON.parse(key), fields, legacy: diagnostic(a), current: diagnostic(b) });
+  }
+
+  // Heading attachment adds a heading span to an otherwise stable body. Pair
+  // only exact body spans, within one source row, and only when the candidate
+  // is unique in both directions. Ambiguous merges and splits stay added and
+  // removed rather than being assigned greedily.
+  const unmatchedBefore = beforeRecords.filter((record) => !pairedBefore.has(record));
+  const unmatchedAfter = afterRecords.filter((record) => !pairedAfter.has(record));
+  const candidatesByCurrent = new Map();
+  const candidatesByLegacy = new Map();
+  for (const current of unmatchedAfter) {
+    if (!current.headingAttached || current.spans.length < 2) continue;
+    const bodySpan = current.spans[current.spans.length - 1];
+    const candidates = unmatchedBefore.filter((legacy) =>
+      rowIdentity(legacy) === rowIdentity(current)
+      && legacy.spans.length === 1
+      && sameJson(legacy.spans[0], bodySpan));
+    candidatesByCurrent.set(current, candidates);
+    for (const legacy of candidates) {
+      if (!candidatesByLegacy.has(legacy)) candidatesByLegacy.set(legacy, []);
+      candidatesByLegacy.get(legacy).push(current);
     }
   }
+  for (const [current, candidates] of candidatesByCurrent) {
+    if (candidates.length !== 1) continue;
+    const legacy = candidates[0];
+    if (candidatesByLegacy.get(legacy)?.length !== 1) continue;
+    pairedBefore.add(legacy);
+    pairedAfter.add(current);
+    attachmentPairs.push([legacy, current]);
+    const fields = changedFields(legacy, current);
+    changes.push({
+      kind: 'modified',
+      key: JSON.parse(identity(legacy)),
+      currentKey: JSON.parse(identity(current)),
+      fields,
+      legacy: diagnostic(legacy),
+      current: diagnostic(current),
+    });
+  }
+
+  for (const current of afterRecords) {
+    if (!pairedAfter.has(current)) {
+      changes.push({ kind: 'added', key: JSON.parse(identity(current)), legacy: null, current: diagnostic(current) });
+    }
+  }
+  for (const legacy of beforeRecords) {
+    if (pairedBefore.has(legacy)) continue;
+    const absorbedBy = legacy.selectionStatus === 'skipped' && legacy.spans.length === 1
+      ? attachmentPairs.map(([, current]) => current).filter((current) =>
+        rowIdentity(legacy) === rowIdentity(current)
+        && current.spans.slice(0, -1).some((span) => sameJson(span, legacy.spans[0])))
+      : [];
+    changes.push({
+      kind: 'removed',
+      key: JSON.parse(identity(legacy)),
+      legacy: diagnostic(legacy),
+      current: null,
+      ...(absorbedBy.length === 1 ? { absorbedInto: JSON.parse(identity(absorbedBy[0])) } : {}),
+    });
+  }
 
   const counts = { added: 0, removed: 0, modified: 0 };
-  const byImpact = { population: 0, detector: 0, normalization: 0, 'metadata-only': 0 };
+  const byImpact = { population: 0, detector: 0, normalization: 0, segmentation: 0, 'metadata-only': 0 };
   for (const change of changes) {
     change.impact = changeImpact(change);
     counts[change.kind]++;
@@ -230,7 +304,7 @@ function changeSet(legacyRecords, currentRecords, exampleLimit = 40) {
 }
 
 function assertCommonInputs(legacy, current) {
-  const fields = ['manifestHash', 'detectorHash', 'detectorOptions', 'sources'];
+  const fields = ['manifestHash', 'detectorHash', 'measurementHarnessHash', 'detectorOptions', 'sources'];
   for (const field of fields) {
     if (!sameJson(legacy.metadata[field], current.metadata[field])) {
       throw new Error(`Comparison inputs differ between legacy and current runs: ${field}`);
@@ -247,6 +321,7 @@ function comparisonMetadata(legacy, current) {
   return {
     manifestHash: current.metadata.manifestHash,
     detectorHash: current.metadata.detectorHash,
+    measurementHarnessHash: current.metadata.measurementHarnessHash,
     detectorOptions: current.metadata.detectorOptions,
     revision: current.metadata.revision,
     spanEncoding: current.metadata.spanEncoding,
@@ -260,6 +335,10 @@ function comparisonMetadata(legacy, current) {
       legacy: legacy.metadata.preprocessorHash,
       current: current.metadata.preprocessorHash,
     },
+    preprocessorImplementations: {
+      legacy: legacy.metadata.preprocessorImplementation,
+      current: current.metadata.preprocessorImplementation,
+    },
     legacyReference: legacy.metadata.legacyReference,
   };
 }

scripts/fp-compare.test.js

@@ -6,7 +6,7 @@ const fs = require('node:fs');
 const os = require('node:os');
 const path = require('node:path');
 const { sha256 } = require('./corpus.js');
-const { compare, parseArgs, publicSummary, writeArtifacts } = require('./fp-compare.js');
+const { changeSet, compare, parseArgs, publicSummary, writeArtifacts } = require('./fp-compare.js');
 
 let passed = 0;
 function test(name, fn) {
@@ -72,14 +72,19 @@ test('runs the same verified source through both preprocessors in both unit mode
     assert.deepStrictEqual(metadata.sourceCounts, { verified: 1, unavailable: 0, injected: 0 });
     assert.strictEqual(metadata.sourceVerification[0].sha256, manifest.documents[0].sha256);
     assert.strictEqual(metadata.sourceVerification[0].status, 'verified');
+    assert.notStrictEqual(metadata.preprocessorHashes.legacy, metadata.preprocessorHashes.current);
+    assert.deepStrictEqual(metadata.preprocessorImplementations, {
+      legacy: 'legacy-inline', current: 'structural-module',
+    });
+    assert.ok(metadata.measurementHarnessHash);
   }
 });
 
-test('reports added, removed, and modified decisions by original spans', () => {
+test('reports decision changes by original spans', () => {
   const changes = result.modes.paragraph.comparison.changes;
-  assert.ok(changes.counts.added > 0, JSON.stringify(changes.counts));
   assert.ok(changes.counts.removed > 0, JSON.stringify(changes.counts));
   assert.ok(changes.counts.modified > 0, JSON.stringify(changes.counts));
+  assert.strictEqual(changes.total, changes.counts.added + changes.counts.removed + changes.counts.modified);
   for (const example of changes.examples) {
     assert.ok(Array.isArray(example.key[4]), 'identity key ends with original spans');
     assert.strictEqual('text' in (example.legacy || {}), false);
@@ -87,6 +92,71 @@ test('reports added, removed, and modified decisions by original spans', () => {
   }
 });
 
+test('pairs an attached heading with its unique legacy body without changing either identity', () => {
+  const base = {
+    recordKind: 'unit', doc: 'fixture', rowId: 'row', rowIndex: 0,
+    rowSourceHash: 'source', cls: 'human', register: 'docs', model: null,
+    selectionStatus: 'selected', status: 'selected', reason: null,
+    headingAttached: false, headingKind: null, kinds: ['legacy-flat'],
+    inputWords: 55, detectorWords: 55, detectorStatus: 'Clean', score: 0, types: [],
+  };
+  const legacyHeading = {
+    ...base, spans: [{ start: 0, end: 10 }], unitId: 'legacy-heading', normalizedHash: 'heading',
+    selectionStatus: 'skipped', status: 'skipped', reason: 'below-min', inputWords: 2,
+    detectorWords: null, detectorStatus: null, score: null,
+  };
+  const legacyBody = {
+    ...base, spans: [{ start: 12, end: 100 }], unitId: 'legacy-body', normalizedHash: 'body',
+  };
+  const current = {
+    ...base, spans: [{ start: 0, end: 10 }, { start: 12, end: 100 }],
+    unitId: 'current-attached', normalizedHash: 'heading-body', headingAttached: true,
+    headingKind: 'atx', kinds: ['atx-heading', 'prose'], inputWords: 57,
+    detectorWords: 57, detectorStatus: 'Minimal AI signals', score: 4,
+    types: ['title-case-header'],
+  };
+
+  const changes = changeSet([legacyHeading, legacyBody], [current], 10);
+  assert.deepStrictEqual(changes.counts, { added: 0, removed: 1, modified: 1 });
+  const paired = changes.examples.find((change) => change.kind === 'modified');
+  assert.deepStrictEqual(paired.key[4], legacyBody.spans);
+  assert.deepStrictEqual(paired.currentKey[4], current.spans);
+  assert.strictEqual(paired.legacy.unitId, 'legacy-body');
+  assert.strictEqual(paired.current.unitId, 'current-attached');
+  assert.ok(paired.fields.includes('spans'));
+  assert.strictEqual(paired.impact, 'detector');
+
+  const absorbed = changes.examples.find((change) => change.kind === 'removed');
+  assert.deepStrictEqual(absorbed.absorbedInto[4], current.spans);
+  assert.strictEqual(absorbed.impact, 'segmentation');
+  assert.strictEqual(changes.byImpact.population, 0);
+  assert.strictEqual(changes.byImpact.segmentation, 1);
+});
+
+test('leaves ambiguous attachment candidates unpaired and counts each record once', () => {
+  const base = {
+    recordKind: 'unit', doc: 'fixture', rowId: 'row', rowIndex: 0,
+    rowSourceHash: 'source', cls: 'human', register: 'docs', model: null,
+    selectionStatus: 'selected', status: 'selected', reason: null,
+    headingAttached: false, headingKind: null, kinds: ['legacy-flat'],
+    inputWords: 55, detectorWords: 55, detectorStatus: 'Clean', score: 0, types: [],
+  };
+  const legacy = { ...base, spans: [{ start: 20, end: 100 }], unitId: 'legacy', normalizedHash: 'body' };
+  const first = {
+    ...base, spans: [{ start: 0, end: 10 }, { start: 20, end: 100 }],
+    unitId: 'first', normalizedHash: 'first', headingAttached: true, headingKind: 'atx',
+  };
+  const second = {
+    ...base, spans: [{ start: 11, end: 18 }, { start: 20, end: 100 }],
+    unitId: 'second', normalizedHash: 'second', headingAttached: true, headingKind: 'atx',
+  };
+
+  const changes = changeSet([legacy], [first, second], 10);
+  assert.deepStrictEqual(changes.counts, { added: 2, removed: 1, modified: 0 });
+  assert.strictEqual(changes.total, 3);
+  assert.strictEqual(changes.examples.some((change) => change.currentKey), false);
+});
+
 test('compares an empty row that produces no current decisions', () => {
   const changes = result.modes.paragraph.comparison.changes;
   const empty = changes.examples.find((change) => change.legacy?.rowId === 'empty-human');

scripts/fp-measure.js

@@ -38,6 +38,7 @@ const { prepareUnits, normalizeUnit, splitUnits, unitsForText } = require('./fp-
 // near 10; a table starting at 25 reports 0.0% everywhere and hides that fact
 // instead of showing it.
 const THRESHOLDS = [3, 5, 10, 15, 25, 50];
+const LEGACY_REFERENCE = 'fabd62d9c8785dd0edda35201359bcc635b7d3de';
 
 /** Wilson interval. These rates run near the boundaries, where the normal
  *  approximation produces impossible bounds. */
@@ -105,6 +106,27 @@ function legacyPrepareUnits(text, mode = 'paragraph') {
   };
 }
 
+function preprocessorFingerprint(preprocess) {
+  if (preprocess === 'legacy') {
+    return sha256(JSON.stringify({
+      implementation: 'legacy-inline',
+      reference: LEGACY_REFERENCE,
+      dependencies: [wordCount.toString(), legacyPrepareUnits.toString()],
+    }));
+  }
+  return sha256(JSON.stringify({
+    implementation: 'structural-module',
+    source: fs.readFileSync(path.join(__dirname, 'fp-preprocess.js'), 'utf8'),
+  }));
+}
+
+function measurementHarnessFingerprint() {
+  return sha256(
+    fs.readFileSync(__filename, 'utf8')
+    + fs.readFileSync(path.join(__dirname, 'fp-preprocess.js'), 'utf8'),
+  );
+}
+
 function accountingFor(records) {
   const totals = { selected: 0, skipped: 0, unavailableSources: 0, reasons: {} };
   const bySource = {};
@@ -204,10 +226,12 @@ function measure(opts = {}) {
   }
   const metadata = {
     schemaVersion: 1, unit, preprocess, revision: revision(),
-    legacyReference: 'fabd62d9c8785dd0edda35201359bcc635b7d3de',
+    legacyReference: LEGACY_REFERENCE,
     manifestHash: sha256(JSON.stringify(manifest)), sources, rows: rowIdentities,
     detectorHash: detector === AIDetector ? sha256(fs.readFileSync(path.join(__dirname, '../detector/patterns.js'), 'utf8')) : null,
-    preprocessorHash: sha256(fs.readFileSync(__filename, 'utf8') + fs.readFileSync(path.join(__dirname, 'fp-preprocess.js'), 'utf8')),
+    measurementHarnessHash: measurementHarnessFingerprint(),
+    preprocessorImplementation: preprocess === 'legacy' ? 'legacy-inline' : 'structural-module',
+    preprocessorHash: preprocessorFingerprint(preprocess),
     detectorOptions: { contextMode: 'general', sourceMode: 'plain' },
     spanEncoding: 'original UTF-16 code units; start inclusive, end exclusive',
   };

scripts/fp-measure.test.js

@@ -100,6 +100,10 @@ t('stable identities use original spans and retain class metadata', () => {
   const rows = [{ id: 'paired-answer', class: 'machine', register: 'academic', model: 'fixture-model', text }];
   const current = fixture(rows);
   const legacy = fixture(rows, { preprocess: 'legacy' });
+  assert.notStrictEqual(current.metadata.preprocessorHash, legacy.metadata.preprocessorHash);
+  assert.strictEqual(current.metadata.preprocessorImplementation, 'structural-module');
+  assert.strictEqual(legacy.metadata.preprocessorImplementation, 'legacy-inline');
+  assert.strictEqual(current.metadata.measurementHarnessHash, legacy.metadata.measurementHarnessHash);
   assert.deepStrictEqual(current.records.map((r) => r.unitId), legacy.records.map((r) => r.unitId));
   for (const record of current.records) {
     assert.strictEqual(record.cls, 'machine');

scripts/fp-preprocess.js

@@ -13,6 +13,8 @@ const MAX_WORDS = 400;
 
 const ATX = /^ {0,3}#{1,6}(?:[ \t]+|$)/;
 const FENCE = /^[ \t]{0,3}(`{3,}|~{3,})(.*)$/;
+const BULLET_LIST = /^[ \t]*[-+*•][ \t]+/;
+const ORDERED_LIST = /^[ \t]*(\d+)[.)][ \t]+/;
 const LIST = /^[ \t]*(?:[-+*•]|\d+[.)])[ \t]+/;
 const LIST_CONTINUATION = /^(?: {2,}|\t)\S/;
 const QUOTE = /^[ \t]*>/;
@@ -67,6 +69,27 @@ function addKind(kinds, kind) {
   if (!kinds.includes(kind)) kinds.push(kind);
 }
 
+function isIndentedCodeLine(lines, index) {
+  if (index < 0 || !INDENTED.test(lines[index].text)) return false;
+  let start = index;
+  while (start > 0 && INDENTED.test(lines[start - 1].text) && !isBlank(lines[start - 1])) start -= 1;
+  return start === 0 || isBlank(lines[start - 1]);
+}
+
+// An ordered marker other than numeric 1 cannot interrupt an open paragraph.
+// This narrow distinction follows https://spec.commonmark.org/0.31.2/#list-items
+// and keeps a hard-wrapped year such as `1859.` in prose.
+function startsListRun(lines, kinds, index) {
+  if (BULLET_LIST.test(lines[index].text)) return true;
+  const ordered = lines[index].text.match(ORDERED_LIST);
+  if (!ordered) return false;
+  if (Number(ordered[1]) === 1 || index === 0 || isBlank(lines[index - 1])) return true;
+  const previousKinds = kinds[index - 1];
+  return previousKinds.length > 0
+    || QUOTE.test(lines[index - 1].text)
+    || isIndentedCodeLine(lines, index - 1);
+}
+
 /** Classify lines without changing their source offsets. */
 function classify(lines) {
   const kinds = lines.map(() => []);
@@ -118,7 +141,7 @@ function classify(lines) {
   // subsequent items or indented continuation. This keeps line structure and
   // prevents continuation text from being folded as an unrelated paragraph.
   for (let i = 0; i < lines.length; i++) {
-    if (kinds[i].length || !LIST.test(lines[i].text)) continue;
+    if (kinds[i].length || !startsListRun(lines, kinds, i)) continue;
     let j = i;
     while (j < lines.length) {
       if (kinds[j].some((kind) => kind.endsWith('heading') || kind === 'fenced-code')) break;
@@ -274,6 +297,7 @@ function buildAtoms(lines, classified) {
     }
 
     const start = i;
+    let mergedContinuation = false;
     let sawList = false;
     let sawQuote = false;
     let sawIndented = false;
@@ -302,6 +326,7 @@ function buildAtoms(lines, classified) {
       const mergeFits = structuralContinuation
         && wordCount(normalizedRange(lines, classified, start, prospectiveEnd)) <= MAX_WORDS;
       if (mergeFits) {
+        mergedContinuation = true;
         i = next;
         continue;
       }
@@ -314,12 +339,13 @@ function buildAtoms(lines, classified) {
       kinds: collectKinds(classified, start, i),
       isHeading: false,
       headingKind: null,
+      mergedContinuation,
     });
   }
   return atoms;
 }
 
-function makeDecision(text, spans, kinds, attached, attachedKind, reasonOverride) {
+function makeDecision(text, spans, kinds, attached, attachedKind, reasonOverride, mergedContinuation = false) {
   const inputWords = wordCount(text);
   let status = 'selected';
   let reason = null;
@@ -341,6 +367,7 @@ function makeDecision(text, spans, kinds, attached, attachedKind, reasonOverride
     kinds,
     headingAttached: attached,
     headingKind: attachedKind,
+    ...(mergedContinuation ? { mergedContinuation: true } : {}),
     inputWords,
     status,
     reason,
@@ -377,6 +404,8 @@ function decisionsForParagraphs(input, lines, classified) {
             kinds,
             true,
             atom.headingKind,
+            null,
+            next.mergedContinuation,
           ));
           i += 1;
           continue;
@@ -411,6 +440,8 @@ function decisionsForParagraphs(input, lines, classified) {
       atom.kinds,
       false,
       null,
+      null,
+      atom.mergedContinuation,
     ));
   }
 

scripts/fp-preprocess.test.js

@@ -3,6 +3,7 @@
 
 const assert = require('node:assert/strict');
 const AIDetector = require('../detector/patterns.js');
+const { legacyPrepareUnits } = require('./fp-measure.js');
 const {
   MIN_WORDS,
   MAX_WORDS,
@@ -228,6 +229,41 @@ test('only ordinary hard-wrapped prose is joined', () => {
   assert.equal(output.includes('\r'), false);
 });
 
+test('mid-paragraph ordinals do not start a list run', () => {
+  const historical = [
+    'The decisive year was',
+    '1859. The publication of that work changed the temper of the debate,',
+    `and no one who has read it can afford to pass over the evidence. ${words(35, 'tail')}`,
+  ].join('\n');
+  const decision = prepareUnits(historical).decisions[0];
+  assert.deepEqual(decision.kinds, ['prose']);
+  assert.equal(decision.text.includes('\n'), false);
+  assert.match(decision.text, /year was 1859\. The publication/);
+
+  const indentedContinuation = [
+    'The decisive year was',
+    '    discussed by the committee in',
+    `1859. The publication changed the debate. ${words(40, 'tail')}`,
+  ].join('\n');
+  const indented = prepareUnits(indentedContinuation).decisions[0];
+  assert.deepEqual(indented.kinds, ['prose']);
+  assert.equal(indented.text.includes('\n'), false);
+
+  for (const input of [
+    `Prose opens here\n1. First item ${words(48)}`,
+    `Prose opens here\n01. First item ${words(48)}`,
+    `Prose opens here\n- Bullet item ${words(48)}`,
+    `Prose opens here\n\n1859. Ordered item ${words(48)}`,
+    `## Context\n2. Ordered item ${words(48)}`,
+  ]) {
+    assert.ok(prepareUnits(input).decisions.some((item) => item.kinds.includes('list')), input);
+  }
+
+  const afterCode = prepareUnits(`    code\n2. Second item ${words(50)}`).decisions;
+  assert.ok(afterCode.some((item) => item.kinds.includes('indented-code')));
+  assert.ok(afterCode.some((item) => item.kinds.includes('list')));
+});
+
 test('mixed prose and five short bullets remain one eligible source body', () => {
   const intro = 'The report contains a list of the available components for this release. Each component has a corresponding entry in the inventory and a named owner on the review team. The owner checks the entry against the published release manifest before every scheduled deployment review.';
   const bullets = ['Cloud platform', 'API gateway', 'Data pipeline', 'Event stream', 'Message queue'];
@@ -252,17 +288,19 @@ test('mixed prose and five short bullets remain one eligible source body', () =>
   const twoWordBullets = ['Cloud platform', 'API gateway', 'Data pipeline', 'Event stream', 'Message queue']
     .map((item) => `- ${item}`)
     .join('\n');
-  for (const proseWords of [390, 399]) {
+  for (const proseWords of [386, 390, 399]) {
     const over = prepareUnits(`${words(proseWords)}\n\n${twoWordBullets}`).decisions;
     assert.equal(over[0].status, 'selected');
     assert.equal(over[0].inputWords, proseWords);
     assert.equal(over[0].text, words(proseWords));
     assert.equal(over[1].reason, 'below-min');
+    assert.equal(Object.hasOwn(over[0], 'mergedContinuation'), false);
   }
   const exact = prepareUnits(`${words(385)}\n\n${twoWordBullets}`).decisions;
   assert.equal(exact.length, 1);
   assert.equal(exact[0].status, 'selected');
   assert.equal(exact[0].inputWords, 400);
+  assert.equal(exact[0].mergedContinuation, true);
 
   const unicodeBullets = ['Cloud platform', 'API gateway', 'Data pipeline', 'Event stream', 'Message queue']
     .map((item) => `• ${item}`)
@@ -296,6 +334,54 @@ test('mixed prose and five short bullets remain one eligible source body', () =>
   assert.ok(preparedIndentedTypes.includes('bullet-np-list'));
 });
 
+test('detector deltas are pinned against the frozen legacy preparation', () => {
+  const detect = (prepare, text) => prepare(text).decisions
+    .filter((decision) => decision.status === 'selected')
+    .map((decision) => AIDetector.analyzeText(decision.text, { contextMode: 'general', sourceMode: 'plain' }));
+  const types = (results) => results.flatMap((result) => result.issues.map((issue) => issue.type));
+
+  const quote = [
+    '> The record was genuinely useful.',
+    '> At dawn, the second group carefully recorded every ordinary observation from the northern room before returning home.',
+    '> The smaller result was truly useful.',
+    '> Nothing changed during this trial, although several reviewers stayed late to compare individual pages against older copies in storage and record their doubts.',
+    '> Everyone left.',
+  ].join('\n');
+  const legacyQuote = detect(legacyPrepareUnits, quote);
+  const currentQuote = detect(prepareUnits, quote);
+  assert.deepEqual(legacyQuote.map((result) => result.score), [4]);
+  assert.equal(types(legacyQuote).length, 2);
+  assert.deepEqual(currentQuote.map((result) => result.score), [0]);
+  assert.deepEqual(types(currentQuote), []);
+
+  const bullets = `${words(45, 'intro')}\n${['Cloud platform', 'API gateway', 'Data pipeline', 'Event stream', 'Message queue'].map((item) => `- ${item}`).join('\n')}`;
+  assert.deepEqual(types(detect(legacyPrepareUnits, bullets)), []);
+  assert.deepEqual(types(detect(prepareUnits, bullets)), ['bullet-np-list']);
+
+  const setext = `Benefits And Strategic Considerations\n=====\n${words(50, 'body')}`;
+  assert.deepEqual(types(detect(legacyPrepareUnits, setext)), []);
+  assert.deepEqual(types(detect(prepareUnits, setext)), ['title-case-header']);
+
+  const fence = `\`\`\`\n- Cloud platform\n- API gateway\n- Data pipeline\n- Event stream\n- Message queue\n# Benefits And Strategic Considerations\n\`\`\`\n${words(50, 'body')}`;
+  assert.deepEqual(types(detect(legacyPrepareUnits, fence)), []);
+  assert.deepEqual(types(detect(prepareUnits, fence)), []);
+});
+
+test('heading markers remain source tokens at selection boundaries', () => {
+  const atFloor = prepareUnits(`## ${words(49, 'heading')}`).decisions[0];
+  assert.equal(atFloor.inputWords, 50);
+  assert.equal(atFloor.status, 'selected');
+
+  const over = prepareUnits(`#### A B\n${words(398, 'body')}`).decisions;
+  assert.equal(over[0].headingAttached, false);
+  assert.equal(over[0].reason, 'heading-would-exceed-maximum');
+  assert.equal(over[1].inputWords, 398);
+
+  const atCeiling = prepareUnits(`## Context\n${words(398, 'body')}`).decisions[0];
+  assert.equal(atCeiling.inputWords, 400);
+  assert.equal(atCeiling.headingAttached, true);
+});
+
 test('blank-separated list and quote continuations retain their layout', () => {
   const list = `- First item has context\n\n  continued after a blank\n\n- Second item has context\n${words(45)}`;
   const listDecision = prepareUnits(list).decisions[0];