mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-05 15:20:30 +08:00
9b05e5c67e
## Summary Fixes a regression introduced by #17203 (strict-cap atom-split) and a secondary delimiter-handling bug from #17723. **Root cause:** - #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which split oversize units into ≤ token_size pieces. This collapsed `token_size=1` into 1-token chunks and set the cap at 512, mismatching the model-layer truncation boundary (embedding ~8191 / rerank 500/4096/8192/2048). Atom-split is unnecessary: oversize units stay whole and the model layer truncates. - #17723's delimiter handling dropped consecutive delimiters (`A####B` -> `A##B`), glued JSON items with `"".join`, ignored `children_delimiters`, and stripped whitespace delimiters. ## Changes - New pure helper `merge_paragraphs(paragraphs, token_size, strategy)` with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default `OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows `token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while the projected total stays within `token_size`, merging one boundary-overflow paragraph before closing. Oversize paragraphs stand alone. - `naive_merge` / `naive_merge_with_images` / `RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split removed. `naive_merge` / `naive_merge_with_images` always split a section on the delimiter whenever one is present (even when the section already fits `token_size`), so delimiter text never leaks into a chunk. Only the empty-delimiter (size-only) mode skips splitting. - `token_chunker`: delimiter text is dropped (not stripped); JSON flush joins buffered items with `"\n"`; `children_delimiters` and `PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions are now attributed **per segment** — each split chunk carries only the positions of the item(s) that contributed to it — fixing a leak where page-N coordinates were attached to page-M chunks and all segments shared one preview image. - `test_txt_parser.py` rewritten to assert the new contract (not the old strict cap); `naive_merge` and delimiter-case-sensitive matrices updated. ## Contract (refs #17799) - user specified delimiter = chunk boundary; user specified delimiter text never enters a chunk. - `token_size` = soft target + merge strategy; no atom-split. - Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP` (strict cap). - `OVER_CAP` has no hard cap; the model layer truncates oversize units. `UNDER_CAP` enforces a strict cap. ## Notes - Closes the wrong-object revert in #17774 (revert #17723 would re-introduce delimiter-in-chunk and the strict cap). - Go-side alignment (`internal/ingestion/component/chunker/token.go`) is a follow-up PR. --------- Co-authored-by: CodeBuddy <noreply@tencent.com>
64 lines
2.5 KiB
Python
64 lines
2.5 KiB
Python
#
|
|
# Copyright 2025 The InfiniFlow Authors. All Rights Reserved.
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
import logging
|
|
import re
|
|
|
|
from deepdoc.parser.utils import get_text
|
|
from rag.nlp import MergeStrategy, merge_paragraphs
|
|
from rag.nlp.delim import (
|
|
compile_delimiter_pattern,
|
|
normalize_text_newlines,
|
|
parse_delimiter_field,
|
|
)
|
|
|
|
|
|
class RAGFlowTxtParser:
|
|
def __call__(self, fnm, binary=None, chunk_token_num=128, delimiter="\n!?;。；！？", keep_delimiters=False):
|
|
txt = get_text(fnm, binary)
|
|
return self.parser_txt(txt, chunk_token_num, delimiter, keep_delimiters)
|
|
|
|
@classmethod
|
|
def parser_txt(cls, txt, chunk_token_num=128, delimiter="\n!?;。；！？", keep_delimiters=False):
|
|
if not isinstance(txt, str):
|
|
raise TypeError("txt type should be str!")
|
|
txt = normalize_text_newlines(txt)
|
|
parsed_dels = parse_delimiter_field(delimiter)
|
|
dels = compile_delimiter_pattern(parsed_dels)
|
|
logging.debug(
|
|
"RAGFlowTxtParser.parser_txt: delimiter_count=%d, splitting=%s",
|
|
len(parsed_dels),
|
|
bool(dels),
|
|
)
|
|
secs = re.split(r"(%s)" % dels, txt) if dels else [txt]
|
|
paragraphs = []
|
|
for index, sec in enumerate(secs):
|
|
if dels and re.match(f"^{dels}$", sec):
|
|
continue
|
|
if not sec:
|
|
continue
|
|
if keep_delimiters and index + 1 < len(secs) and re.match(f"^{dels}$", secs[index + 1]):
|
|
sec += secs[index + 1]
|
|
paragraphs.append(sec)
|
|
|
|
# Group delimiter-split paragraphs with the OVER_CAP merge strategy: no
|
|
# atom-split, delimiter text never enters a chunk. A paragraph larger
|
|
# than chunk_token_num stands alone; the model layer truncates it.
|
|
groups = merge_paragraphs(paragraphs, chunk_token_num, MergeStrategy.OVER_CAP)
|
|
cks = ["\n".join(g) for g in groups]
|
|
logging.debug("parser_txt: %d sections -> %d chunks (chunk_token_num=%d)", len(secs), len(cks), chunk_token_num)
|
|
return [[c, ""] for c in cks]
|