mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-06 15:44:09 +08:00
9b05e5c67e
## Summary Fixes a regression introduced by #17203 (strict-cap atom-split) and a secondary delimiter-handling bug from #17723. **Root cause:** - #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which split oversize units into ≤ token_size pieces. This collapsed `token_size=1` into 1-token chunks and set the cap at 512, mismatching the model-layer truncation boundary (embedding ~8191 / rerank 500/4096/8192/2048). Atom-split is unnecessary: oversize units stay whole and the model layer truncates. - #17723's delimiter handling dropped consecutive delimiters (`A####B` -> `A##B`), glued JSON items with `"".join`, ignored `children_delimiters`, and stripped whitespace delimiters. ## Changes - New pure helper `merge_paragraphs(paragraphs, token_size, strategy)` with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default `OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows `token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while the projected total stays within `token_size`, merging one boundary-overflow paragraph before closing. Oversize paragraphs stand alone. - `naive_merge` / `naive_merge_with_images` / `RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split removed. `naive_merge` / `naive_merge_with_images` always split a section on the delimiter whenever one is present (even when the section already fits `token_size`), so delimiter text never leaks into a chunk. Only the empty-delimiter (size-only) mode skips splitting. - `token_chunker`: delimiter text is dropped (not stripped); JSON flush joins buffered items with `"\n"`; `children_delimiters` and `PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions are now attributed **per segment** — each split chunk carries only the positions of the item(s) that contributed to it — fixing a leak where page-N coordinates were attached to page-M chunks and all segments shared one preview image. - `test_txt_parser.py` rewritten to assert the new contract (not the old strict cap); `naive_merge` and delimiter-case-sensitive matrices updated. ## Contract (refs #17799) - user specified delimiter = chunk boundary; user specified delimiter text never enters a chunk. - `token_size` = soft target + merge strategy; no atom-split. - Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP` (strict cap). - `OVER_CAP` has no hard cap; the model layer truncates oversize units. `UNDER_CAP` enforces a strict cap. ## Notes - Closes the wrong-object revert in #17774 (revert #17723 would re-introduce delimiter-in-chunk and the strict cap). - Go-side alignment (`internal/ingestion/component/chunker/token.go`) is a follow-up PR. --------- Co-authored-by: CodeBuddy <noreply@tencent.com>
109 lines
4.5 KiB
Python
109 lines
4.5 KiB
Python
#
|
|
# Copyright 2025 The InfiniFlow Authors. All Rights Reserved.
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
#
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
"""Regression tests for ``RAGFlowTxtParser.parser_txt`` under the chunking contract.
|
|
|
|
The contract (see ``rag.nlp.merge_paragraphs``, refs #17799):
|
|
|
|
* delimiter = chunk boundary: delimiter text never enters a chunk;
|
|
* ``token_size`` = soft target + merge strategy (``OVER_CAP`` default); no
|
|
atom-split — a paragraph larger than ``chunk_token_num`` stands alone and the
|
|
model layer truncates it;
|
|
* ``UNDER_CAP`` is available as an explicit alternative strategy (never overflows
|
|
``chunk_token_num``; ``OVER_CAP`` allows one boundary overflow).
|
|
"""
|
|
|
|
from deepdoc.parser.txt_parser import RAGFlowTxtParser
|
|
import rag.nlp as nlp_mod
|
|
|
|
|
|
def _fake_word_tokens(s):
|
|
return len(s.split())
|
|
|
|
|
|
def _nonempty(chunks):
|
|
return [c for c, _ in chunks if c.strip()]
|
|
|
|
|
|
def test_over_cap_accumulates_adjacent_paragraphs(monkeypatch):
|
|
monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
|
|
text = "\n".join(["alpha beta gamma delta" for _ in range(8)]) # 4 tokens each
|
|
chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=50, delimiter="\n"))
|
|
# OVER_CAP greedily accumulates adjacent paragraphs while under cap, instead
|
|
# of capping at fixed pairs: 8 * 4 = 32 tokens all fit under 50 -> 1 chunk.
|
|
assert len(chunks) == 1
|
|
assert len(chunks[0].split()) == 32
|
|
# Content is preserved (32 tokens total).
|
|
assert sum(len(c.split()) for c in chunks) == 32
|
|
|
|
|
|
def test_oversize_unit_not_atom_split(monkeypatch):
|
|
monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
|
|
text = "word " * 200 # ~200 tokens, no delimiter -> one paragraph
|
|
chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=30, delimiter="\n!?;。；！？"))
|
|
# No atom-split: the whole unit is a single chunk.
|
|
assert len(chunks) == 1
|
|
assert "".join(chunks).count("word") == 200
|
|
|
|
|
|
def test_delimiter_text_not_in_chunk(monkeypatch):
|
|
monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
|
|
text = "first##second##third"
|
|
chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=1000, delimiter="##"))
|
|
assert all("##" not in c for c in chunks)
|
|
joined = "\n".join(chunks)
|
|
assert "first" in joined and "second" in joined and "third" in joined
|
|
|
|
|
|
def test_consecutive_delimiters_do_not_leak_delimiter_text(monkeypatch):
|
|
monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
|
|
# pattern "##": consecutive delimiters must not glue the sides with "##".
|
|
text = "A####B"
|
|
chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=1000, delimiter="##"))
|
|
joined = "\n".join(chunks)
|
|
assert "##" not in joined
|
|
assert "A" in joined and "B" in joined
|
|
|
|
|
|
def test_token_size_zero_keeps_each_paragraph_alone(monkeypatch):
|
|
monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
|
|
text = "first second third"
|
|
chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=0, delimiter=" "))
|
|
assert chunks == ["first", "second", "third"]
|
|
|
|
|
|
def test_delimiter_boundary_when_segment_exceeds_cap(monkeypatch):
|
|
monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
|
|
# Each paragraph is 2 tokens (> cap=1) -> its own chunk.
|
|
text = "aa aa\nbb bb\ncc cc"
|
|
chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=1, delimiter="\n"))
|
|
assert chunks == ["aa aa", "bb bb", "cc cc"]
|
|
|
|
|
|
def test_keep_delimiters_preserves_delimiter(monkeypatch):
|
|
monkeypatch.setattr(nlp_mod, "num_tokens_from_string", _fake_word_tokens)
|
|
text = "first|second"
|
|
chunks = _nonempty(RAGFlowTxtParser.parser_txt(text, chunk_token_num=1000, delimiter="|", keep_delimiters=True))
|
|
# When keep_delimiters=True the delimiter is retained in the chunk.
|
|
assert any("|" in c for c in chunks)
|
|
joined = "\n".join(chunks)
|
|
assert "first" in joined and "second" in joined
|
|
|
|
|
|
def test_empty_text_returns_empty():
|
|
assert RAGFlowTxtParser.parser_txt("", chunk_token_num=128) == []
|