mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-05 23:24:05 +08:00
9b05e5c67e
## Summary Fixes a regression introduced by #17203 (strict-cap atom-split) and a secondary delimiter-handling bug from #17723. **Root cause:** - #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which split oversize units into ≤ token_size pieces. This collapsed `token_size=1` into 1-token chunks and set the cap at 512, mismatching the model-layer truncation boundary (embedding ~8191 / rerank 500/4096/8192/2048). Atom-split is unnecessary: oversize units stay whole and the model layer truncates. - #17723's delimiter handling dropped consecutive delimiters (`A####B` -> `A##B`), glued JSON items with `"".join`, ignored `children_delimiters`, and stripped whitespace delimiters. ## Changes - New pure helper `merge_paragraphs(paragraphs, token_size, strategy)` with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default `OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows `token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while the projected total stays within `token_size`, merging one boundary-overflow paragraph before closing. Oversize paragraphs stand alone. - `naive_merge` / `naive_merge_with_images` / `RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split removed. `naive_merge` / `naive_merge_with_images` always split a section on the delimiter whenever one is present (even when the section already fits `token_size`), so delimiter text never leaks into a chunk. Only the empty-delimiter (size-only) mode skips splitting. - `token_chunker`: delimiter text is dropped (not stripped); JSON flush joins buffered items with `"\n"`; `children_delimiters` and `PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions are now attributed **per segment** — each split chunk carries only the positions of the item(s) that contributed to it — fixing a leak where page-N coordinates were attached to page-M chunks and all segments shared one preview image. - `test_txt_parser.py` rewritten to assert the new contract (not the old strict cap); `naive_merge` and delimiter-case-sensitive matrices updated. ## Contract (refs #17799) - user specified delimiter = chunk boundary; user specified delimiter text never enters a chunk. - `token_size` = soft target + merge strategy; no atom-split. - Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP` (strict cap). - `OVER_CAP` has no hard cap; the model layer truncates oversize units. `UNDER_CAP` enforces a strict cap. ## Notes - Closes the wrong-object revert in #17774 (revert #17723 would re-introduce delimiter-in-chunk and the strict cap). - Go-side alignment (`internal/ingestion/component/chunker/token.go`) is a follow-up PR. --------- Co-authored-by: CodeBuddy <noreply@tencent.com>
(1). Deploy RAGFlow services and images
https://ragflow.io/docs/build_docker_image
(2). Configure the required environment for testing
Install Python dependencies (including test dependencies):
uv sync --python 3.13 --only-group test --no-default-groups --frozen
Activate the environment:
source .venv/bin/activate
Install SDK:
uv pip install sdk/python
Modify the .env file: Add the following code:
COMPOSE_PROFILES=${COMPOSE_PROFILES},tei-cpu
TEI_MODEL=BAAI/bge-small-en-v1.5
RAGFLOW_IMAGE=infiniflow/ragflow:v0.26.4 #Replace with the image you are using
Start the container（wait two minutes）:
docker compose -f docker/docker-compose.yml up -d
(3). Test Elasticsearch
a) Run sdk tests against Elasticsearch:
export HTTP_API_TEST_LEVEL=p2
export HOST_ADDRESS=http://127.0.0.1:9380 # Ensure that this port is the API port mapped to your localhost
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Elasticsearch:
pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api
(4). Test Infinity
Modify the .env file:
DOC_ENGINE=${DOC_ENGINE:-infinity}
Start the container:
docker compose -f docker/docker-compose.yml down -v
docker compose -f docker/docker-compose.yml up -d
a) Run sdk tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_sdk_api
b) Run http api tests against Infinity:
DOC_ENGINE=infinity pytest -s --tb=short --level=${HTTP_API_TEST_LEVEL} test/testcases/test_http_api