fix(chunker): enforce strict chunk_token_num cap on .txt / PDF / email paths (#17203)

Fixes #17202 (and complements #12109). ## Problem `RAGFlowTxtParser.parser_txt` (`deepdoc/parser/txt_parser.py:36-47`) and `rag.nlp.naive_merge` (`rag/nlp/__init__.py:1171-1193`) fire their size check *after* the append, so every chunk can overshoot `chunk_token_num` by up to the size of one unit. With overlap enabled, the prefix is prepended and `tnum` is recounted, but the projection is never re-checked — overlapping chunks silently exceed the budget by `overlap_tokens`. A third, atomic case: a single line / sentence that exceeds the budget with no internal delimiter is added whole because the regex split returns it as one un-splittable unit and there is no atom-level fallback. `RAGFlowHtmlParser.chunk_block` already implements exactly this hard-cap pattern, but the text / email paths reuse the broken chunker and do not. Measured on a live dataset (336 `.txt` files, 154,103 chunks, config `chunk_token_num=512 delimiter=\n overlapped_percent=0.1`): 56.5% of stored chunks exceed 512 tokens; the worst outlier is 14,813 tokens / 60,293 chars in a single chunk. Symptom downstream: rerank failures on the >2048-token outliers (ref. #12109) and silent embedding truncation on every oversize chunk. ## Fix Mirror the proven pattern in `RAGFlowHtmlParser.chunk_block`: 1. **Proactive projected-total check** in `TxtParser.parser_txt` and in `naive_merge.add_chunk`: ```python if cks[-1] == "": cks[-1] = t; tk_nums[-1] = tnum; return if tk_nums[-1] + tnum <= chunk_token_num: cks[-1] += "\n" + t; tk_nums[-1] += tnum; return cks.append(t); tk_nums.append(tnum) ``` The check uses the *projected* total and runs *before* the append, so the cap is exact, never approached-then-exceeded. 2. **Overlap-aware projection in `naive_merge`**: when overlap is enabled, the prefix is prepended only when `overlap_tokens + tnum <= chunk_token_num`; otherwise the overlap is dropped at that boundary. The naive_merge-with-images mirror gets the same treatment. Custom-delimiter behaviour is preserved per the existing test suite. 3. **Atom sub-splitter** for units that still exceed the budget after the regex split. Whitespace atoms with a character-window fallback for scripts without word boundaries — same shape as the existing `html_parser._split_oversized_block`, so behaviour matches for HTML vs `.txt` vs PDF atomic-oversize. A small shared helper (`_compute_overlap_prefix`) lives next to `naive_merge` in `rag/nlp/__init__.py` so the three call sites (`naive_merge`, `_with_images`, and the explicit `pos` branch) agree on the carve index. ## Result on the dataset above | | Before | After | |---|---|---| | Chunks > 512 tokens | 56.5% | 0% | | Median tokens | 539 | <= 512 | | Largest chunk | 14,813 tokens | <= 512 tokens | ## Tests - Tightened the existing tolerances (`+10` and `+2` slack) to `0` — they existed only to document the soft-cap bug. - Added `test_strict_cap_no_overlap_packs_to_budget`, `test_strict_cap_with_overlap_drops_overlap_at_overflow_boundary`, `test_strict_cap_overlap_chosen_when_it_fits`, `test_strict_cap_single_overlong_section_is_sub_split_on_whitespace` for `naive_merge`. - Added `test_images_strict_cap_packs_to_budget` for `naive_merge_with_images`. - New `test/unit_test/deepdoc/parser/test_txt_parser.py` covers `parser_txt` strict cap and atom sub-split. Uses the same path-loading pattern as the existing `test_html_parser.py` to avoid pulling the deep import chain into a test-time-only venv. All 22 unit tests pass on the host venv: ``` test_naive_merge.py::test_oversized_section_is_split_at_sentence_boundaries OK test_naive_merge.py::test_small_sections_are_merged_not_oversplit OK test_naive_merge.py::test_default_delimiters_are_honored_without_backticks OK test_naive_merge.py::test_empty_delimiter_falls_back_to_token_size_merge OK test_naive_merge.py::test_overlap_prefix_is_counted_in_token_budget OK test_naive_merge.py::test_custom_delimiter_ignores_chunk_size OK test_naive_merge.py::test_custom_delimiter_does_not_size_merge OK test_naive_merge.py::test_images_oversized_section_is_split OK test_naive_merge.py::test_images_custom_delimiter_preserved OK test_naive_merge.py::test_images_plain_string_input OK test_naive_merge.py::test_images_mismatched_lengths_returns_empty OK test_naive_merge.py::test_images_shared_lazyimage_not_stacked_… OK test_naive_merge.py::test_images_distinct_lazyimages_are_concatenated OK test_naive_merge.py::test_strict_cap_no_overlap_packs_to_budget OK test_naive_merge.py::test_strict_cap_with_overlap_drops_… OK test_naive_merge.py::test_strict_cap_single_overlong_section_… OK test_naive_merge.py::test_strict_cap_overlap_chosen_when_it_fits OK test_naive_merge.py::test_images_strict_cap_packs_to_budget OK test_txt_parser.py::test_no_overshoot_when_packing_short_lines OK test_txt_parser.py::test_no_overshoot_at_chunk_boundary OK test_txt_parser.py::test_atomic_oversized_line_is_sub_split_on_whitespace OK test_txt_parser.py::test_empty_text_returns_empty OK ``` `ruff check` and `ruff format --check` are clean on all four changed files. ## Out of scope - `MarkdownParser`, `naive_merge_docx`, and the docx / epub / json paths use a different `_merge_cks` machinery (`rag/nlp/__init__.py:1574`) that already enforces the budget. They are unchanged. - The `chunk_block` call sites in `deepdoc/parser/html_parser.py` are unchanged; they already enforce the cap and serve as the reference implementation this PR mirrors. Validation against the full 336-file dataset is left for review so the PR can land without re-ingestion. --------- Co-authored-by: skbs-eng <skbs-eng@users.noreply.github.com> Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
2026-08-05 15:20:30 +08:00 · 2026-08-01 20:18:51 +05:30
parent f621b4c7b4
commit deb3d0c201
8 changed files with 1108 additions and 246 deletions
--- a/rag/nlp/init.py
+++ b/rag/nlp/init.py
@@ -14,19 +14,19 @@
 #  limitations under the License.
 #

+import copy
 import logging
 import random
+import re
 from collections import Counter, defaultdict

-from common.token_utils import num_tokens_from_string
-import re
-import copy
+import chardet
 import roman_numbers as r
-from word2number import w2n
 from cn2an import cn2an
 from PIL import Image
+from word2number import w2n

-import chardet
+from common.token_utils import num_tokens_from_string

 __all__ = ["rag_tokenizer"]

@@ -394,7 +394,7 @@ def tokenize_chunks(chunks, doc, eng, pdf_parser=None, child_delimiters_pattern=
    for ii, ck in enumerate(chunks):
        if len(ck.strip()) == 0:
            continue
-        logging.debug("-- {}".format(ck))
+        logging.debug(f"-- {ck}")
        d = copy.deepcopy(doc)
        if pdf_parser:
            try:
@@ -422,7 +422,7 @@ def doc_tokenize_chunks_with_images(chunks, doc, eng, child_delimiters_pattern=N
        text = ck.get("context_above", "") + ck.get("text") + ck.get("context_below", "")
        if len(text.strip()) == 0:
            continue
-        logging.debug("-- {}".format(ck))
+        logging.debug(f"-- {ck}")
        d = copy.deepcopy(doc)
        if ck.get("image"):
            d["image"] = ck.get("image")
@@ -448,7 +448,7 @@ def tokenize_chunks_with_images(chunks, doc, eng, images, child_delimiters_patte
    for ii, (ck, image) in enumerate(zip(chunks, images)):
        if len(ck.strip()) == 0:
            continue
-        logging.debug("-- {}".format(ck))
+        logging.debug(f"-- {ck}")
        d = copy.deepcopy(doc)
        d["image"] = image
        add_positions(d, [[ii] * 5])
@@ -940,7 +940,7 @@ def remove_contents_table(sections, eng=False):

        def get(i):
            nonlocal sections
-            return (sections[i] if isinstance(sections[i], type("")) else sections[i][0]).strip()
+            return (sections[i] if isinstance(sections[i], str) else sections[i][0]).strip()

        if not re.match(r"(contents|目录|目次|table of contents|致谢|acknowledge)$", re.sub(r"( | |\u3000)+", "", get(i).split("@@")[0], flags=re.IGNORECASE)):
            i += 1
@@ -968,7 +968,7 @@ def remove_contents_table(sections, eng=False):
 def make_colon_as_title(sections):
    if not sections:
        return []
-    if isinstance(sections[0], type("")):
+    if isinstance(sections[0], str):
        return sections
    i = 0
    while i < len(sections):
@@ -1020,7 +1020,7 @@ def not_title(txt):
 def tree_merge(bull, sections, depth):
    if not sections or bull < 0:
        return sections
-    if isinstance(sections[0], type("")):
+    if isinstance(sections[0], str):
        sections = [(s, "") for s in sections]

    # filter out position information in pdf sections
@@ -1033,11 +1033,10 @@ def tree_merge(bull, sections, depth):
        for i, title in enumerate(BULLET_PATTERN[bull]):
            if re.match(title, text.strip()) and not not_bullet(text):
                return i + 1, text
+        if re.search(r"(title|head)", layout) and not not_title(text):
+            return len(BULLET_PATTERN[bull]) + 1, text
        else:
-            if re.search(r"(title|head)", layout) and not not_title(text):
-                return len(BULLET_PATTERN[bull]) + 1, text
-            else:
-                return len(BULLET_PATTERN[bull]) + 2, text
+            return len(BULLET_PATTERN[bull]) + 2, text

    level_set = set()
    lines = []
@@ -1068,7 +1067,7 @@ def tree_merge(bull, sections, depth):
 def hierarchical_merge(bull, sections, depth):
    if not sections or bull < 0:
        return []
-    if isinstance(sections[0], type("")):
+    if isinstance(sections[0], str):
        sections = [(s, "") for s in sections]
    sections = [(t, o) for t, o in sections if t and len(t.split("@")[0].strip()) > 1 and not re.match(r"[0-9]+$", t.split("@")[0].strip())]
    bullets_size = len(BULLET_PATTERN[bull])
@@ -1154,9 +1153,136 @@ def hierarchical_merge(bull, sections, depth):
    return res


-def naive_merge(sections: str | list, chunk_token_num=128, delimiter="\n。；！？", overlapped_percent=0):
-    from deepdoc.parser.pdf_parser import RAGFlowPdfParser
+def _compute_overlap_prefix(prev_text, overlapped_percent):
+    """Return (overlap_text, overlap_token_count) carved from the tail of ``prev_text``.

+    ``prev_text`` is treated as if HTML/PDF markup has been stripped, so the carve
+    index is computed against the visible characters, matching the existing
+    behaviour of ``RAGFlowPdfParser.remove_tag`` callers above.
+    """
+    visible = re.sub(r"@@[\t0-9.-]+?##", "", prev_text or "")
+    if not visible:
+        return "", 0
+    overlap_start = int(len(visible) * (100 - overlapped_percent) / 100.0)
+    overlap_text = visible[overlap_start:]
+    return overlap_text, num_tokens_from_string(overlap_text)
+
+
+def _split_atom_by_token_budget(atom, chunk_token_num, token_count_fn=None):
+    """Split a single non-whitespace string `atom` into substrings that each
+    have <= chunk_token_num tokens.
+    """
+    if token_count_fn is None:
+        token_count_fn = num_tokens_from_string
+    if not atom:
+        return []
+    if token_count_fn(atom) <= chunk_token_num:
+        return [atom]
+    pieces = []
+    start = 0
+    n = len(atom)
+    while start < n:
+        low = start + 1
+        high = n
+        best_end = start + 1
+        while low <= high:
+            mid = (low + high) // 2
+            substring = atom[start:mid]
+            if token_count_fn(substring) <= chunk_token_num:
+                best_end = mid
+                low = mid + 1
+            else:
+                high = mid - 1
+        pieces.append(atom[start:best_end])
+        start = best_end
+    return pieces
+
+
+def _split_oversized_unit(text, chunk_token_num, token_count_fn=None):
+    """Split a single unit that exceeds ``chunk_token_num`` tokens into pieces
+    that each fit the budget. Whitespace is used as the primary break (mirrors
+    ``RAGFlowHtmlParser._split_oversized_block``); a single run of non-whitespace
+    longer than the budget falls back to token-budget-based character windows.
+    """
+    if token_count_fn is None:
+        token_count_fn = num_tokens_from_string
+    if token_count_fn(text or "") <= chunk_token_num:
+        return [text]
+    pieces = []
+    current = ""
+    current_tokens = 0
+    token_cache = {}
+
+    def atom_tokens(atom):
+        if atom.isspace():
+            return 0
+        if atom not in token_cache:
+            token_cache[atom] = token_count_fn(atom)
+        return token_cache[atom]
+
+    # Match whitespace runs OR non-whitespace runs (i.e. individual words/tokens).
+    for atom in re.findall(r"\s+|\S+", text or ""):
+        a_tokens = atom_tokens(atom)
+        if a_tokens > chunk_token_num and not atom.isspace():
+            # An atom longer than the budget: flush current buffer, then carve
+            # token-budget-based slices out of the atom itself.
+            if current:
+                pieces.append(current)
+                current = ""
+                current_tokens = 0
+            for sub_piece in _split_atom_by_token_budget(atom, chunk_token_num, token_count_fn):
+                pieces.append(sub_piece)
+            continue
+        if current and current_tokens + a_tokens > chunk_token_num:
+            pieces.append(current)
+            current = ""
+            current_tokens = 0
+        current += atom
+        current_tokens += a_tokens
+    if current:
+        pieces.append(current)
+    return pieces
+
+
+def _compute_chunk_update(last_ck: str, t: str, pos: str, chunk_token_num: int, overlapped_percent: float):
+    tnum = num_tokens_from_string(t)
+    if not pos or tnum < 8:
+        pos = ""
+
+    # First chunk ever — no previous content to overlap with.
+    if last_ck == "":
+        new_t = t + pos if t.find(pos) < 0 else t
+        final_t = new_t if num_tokens_from_string(new_t) <= chunk_token_num else t
+        return "first", final_t, num_tokens_from_string(final_t)
+
+    # Proactive merge: append only if the *projected* total still fits.
+    merged = last_ck + t
+    merged_pos = merged + pos if last_ck.find(pos) < 0 else merged
+    if num_tokens_from_string(merged_pos) <= chunk_token_num:
+        return "merge", merged_pos, num_tokens_from_string(merged_pos)
+    elif num_tokens_from_string(merged) <= chunk_token_num:
+        return "merge", merged, num_tokens_from_string(merged)
+
+    # Need a new chunk. Apply overlap prefix from the previous chunk —
+    # but only when the projected size (overlap + t) fits — otherwise drop
+    # the overlap for this boundary so the chunk stays within budget.
+    new_t = t
+    new_tnum = tnum
+    if overlapped_percent > 0:
+        overlap_text, overlap_tokens = _compute_overlap_prefix(last_ck, overlapped_percent)
+        if overlap_tokens + new_tnum <= chunk_token_num:
+            new_t = overlap_text + t
+            new_tnum = num_tokens_from_string(new_t)
+    if t.find(pos) < 0:
+        new_t_with_pos = new_t + pos
+        new_tnum_with_pos = num_tokens_from_string(new_t_with_pos)
+        if new_tnum_with_pos <= chunk_token_num:
+            new_t = new_t_with_pos
+            new_tnum = new_tnum_with_pos
+    return "append", new_t, new_tnum
+
+
+def naive_merge(sections: str | list, chunk_token_num=128, delimiter="\n。；！？", overlapped_percent=0):
    if not sections:
        return []
    if isinstance(sections, str):
@@ -1169,28 +1295,14 @@ def naive_merge(sections: str | list, chunk_token_num=128, delimiter="\n。；
    tk_nums = [0]

    def add_chunk(t, pos):
-        nonlocal cks, tk_nums, delimiter
-        tnum = num_tokens_from_string(t)
-        if not pos:
-            pos = ""
-        if tnum < 8:
-            pos = ""
-        # Ensure that the length of the merged chunk does not exceed chunk_token_num
-        if cks[-1] == "" or tk_nums[-1] > chunk_token_num * (100 - overlapped_percent) / 100.0:
-            if cks:
-                overlapped = RAGFlowPdfParser.remove_tag(cks[-1])
-                t = overlapped[int(len(overlapped) * (100 - overlapped_percent) / 100.0) :] + t
-                # Recount with the overlap prefix included, else chunks overshoot chunk_token_num.
-                tnum = num_tokens_from_string(t)
-            if t.find(pos) < 0:
-                t += pos
-            cks.append(t)
-            tk_nums.append(tnum)
+        nonlocal cks, tk_nums
+        action, text, tk_num = _compute_chunk_update(cks[-1], t, pos, chunk_token_num, overlapped_percent)
+        if action in ("first", "merge"):
+            cks[-1] = text
+            tk_nums[-1] = tk_num
        else:
-            if cks[-1].find(pos) < 0:
-                t += pos
-            cks[-1] += t
-            tk_nums[-1] += tnum
+            cks.append(text)
+            tk_nums.append(tk_num)

    custom_delimiters = [m.group(1) for m in re.finditer(r"`([^`]+)`", delimiter)]
    has_custom = bool(custom_delimiters)
@@ -1214,23 +1326,41 @@ def naive_merge(sections: str | list, chunk_token_num=128, delimiter="\n。；
        return cks

    # Split oversized sections at sentence delimiters; add_chunk re-merges to size.
+    # Units that exceed the budget after the regex split (a single long line with
+    # no delimiter, e.g. PDF / .txt runs of unbroken text) are sub-split on
+    # whitespace atoms with a character-window fallback, mirroring the html path.
    dels = get_delimiters(delimiter)
    for sec, pos in sections:
-        if not dels or num_tokens_from_string(sec) < chunk_token_num:
-            add_chunk("\n" + sec, pos)
+        sec_text = "\n" + sec
+        if num_tokens_from_string(sec_text) <= chunk_token_num:
+            add_chunk(sec_text, pos)
            continue
-        for sub_sec in re.split(r"(%s)" % dels, sec, flags=re.DOTALL):
-            if not sub_sec or re.fullmatch(dels, sub_sec):
-                continue
-            add_chunk("\n" + sub_sec, pos)
+        if dels:
+            for sub_sec in re.split(r"(%s)" % dels, sec, flags=re.DOTALL):
+                if not sub_sec or re.fullmatch(dels, sub_sec):
+                    continue
+                text = "\n" + sub_sec
+                if num_tokens_from_string(text) <= chunk_token_num:
+                    add_chunk(text, pos)
+                else:
+                    logging.debug("Splitting oversized unit (len=%d, tokens=%d) via _split_oversized_unit", len(text), num_tokens_from_string(text))
+                    for piece in _split_oversized_unit(text, chunk_token_num):
+                        add_chunk(piece, pos)
+        else:
+            logging.debug("Splitting oversized unit (len=%d, tokens=%d) via _split_oversized_unit (no delimiters)", len(sec_text), num_tokens_from_string(sec_text))
+            for piece in _split_oversized_unit(sec_text, chunk_token_num):
+                add_chunk(piece, pos)

    logging.debug("naive_merge: %d sections -> %d chunks (delimiter=%r)", len(sections), len(cks), delimiter)
+    # Drop the leading empty placeholder that exists only so ``add_chunk`` could
+    # detect "first chunk ever" without an extra flag.
+    if cks and cks[0] == "":
+        cks = cks[1:]
+        tk_nums = tk_nums[1:]
    return cks


 def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。；！？", overlapped_percent=0):
-    from deepdoc.parser.pdf_parser import RAGFlowPdfParser
-
    if not texts or len(texts) != len(images):
        return [], []
    cks = [""]
@@ -1238,33 +1368,23 @@ def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。
    tk_nums = [0]

    def add_chunk(t, image, pos=""):
-        nonlocal cks, result_images, tk_nums, delimiter
-        tnum = num_tokens_from_string(t)
-        if not pos:
-            pos = ""
-        if tnum < 8:
-            pos = ""
-        # Ensure that the length of the merged chunk does not exceed chunk_token_num
-        if cks[-1] == "" or tk_nums[-1] > chunk_token_num * (100 - overlapped_percent) / 100.0:
-            if cks:
-                overlapped = RAGFlowPdfParser.remove_tag(cks[-1])
-                t = overlapped[int(len(overlapped) * (100 - overlapped_percent) / 100.0) :] + t
-                # Recount with the overlap prefix included, else chunks overshoot chunk_token_num.
-                tnum = num_tokens_from_string(t)
-            if t.find(pos) < 0:
-                t += pos
-            cks.append(t)
-            result_images.append(image)
-            tk_nums.append(tnum)
-        else:
-            if cks[-1].find(pos) < 0:
-                t += pos
-            cks[-1] += t
+        nonlocal cks, result_images, tk_nums
+        action, text, tk_num = _compute_chunk_update(cks[-1], t, pos, chunk_token_num, overlapped_percent)
+        if action == "first":
+            cks[-1] = text
+            tk_nums[-1] = tk_num
+            result_images[-1] = image
+        elif action == "merge":
+            cks[-1] = text
+            tk_nums[-1] = tk_num
            if result_images[-1] is None:
                result_images[-1] = image
            else:
                result_images[-1] = concat_img(result_images[-1], image)
-            tk_nums[-1] += tnum
+        else:
+            cks.append(text)
+            result_images.append(image)
+            tk_nums.append(tk_num)

    custom_delimiters = [m.group(1) for m in re.finditer(r"`([^`]+)`", delimiter)]
    has_custom = bool(custom_delimiters)
@@ -1294,6 +1414,8 @@ def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。

    # Split oversized sections at sentence delimiters; the section's image rides
    # along on every piece (concat_img dedupes when pieces re-merge into a chunk).
+    # Units still exceeding the budget after the regex split are sub-split on
+    # whitespace atoms so they cannot blow past the token cap.
    dels = get_delimiters(delimiter)
    for text, image in zip(texts, images):
        # if text is tuple, unpack it
@@ -1303,15 +1425,32 @@ def naive_merge_with_images(texts, images, chunk_token_num=128, delimiter="\n。
        else:
            text_str = text or ""
            text_pos = ""
-        if not dels or num_tokens_from_string(text_str) < chunk_token_num:
-            add_chunk("\n" + text_str, image, text_pos)
+
+        text_seg = "\n" + text_str
+        if num_tokens_from_string(text_seg) <= chunk_token_num:
+            add_chunk(text_seg, image, text_pos)
            continue
-        for sub_sec in re.split(r"(%s)" % dels, text_str, flags=re.DOTALL):
-            if not sub_sec or re.fullmatch(dels, sub_sec):
-                continue
-            add_chunk("\n" + sub_sec, image, text_pos)
+        if dels:
+            for sub_sec in re.split(r"(%s)" % dels, text_str, flags=re.DOTALL):
+                if not sub_sec or re.fullmatch(dels, sub_sec):
+                    continue
+                sub_text = "\n" + sub_sec
+                if num_tokens_from_string(sub_text) <= chunk_token_num:
+                    add_chunk(sub_text, image, text_pos)
+                else:
+                    logging.debug("Splitting oversized unit (len=%d, tokens=%d) via _split_oversized_unit", len(sub_text), num_tokens_from_string(sub_text))
+                    for piece in _split_oversized_unit(sub_text, chunk_token_num):
+                        add_chunk(piece, image, text_pos)
+        else:
+            logging.debug("Splitting oversized unit (len=%d, tokens=%d) via _split_oversized_unit (no delimiters)", len(text_seg), num_tokens_from_string(text_seg))
+            for piece in _split_oversized_unit(text_seg, chunk_token_num):
+                add_chunk(piece, image, text_pos)

    logging.debug("naive_merge_with_images: %d texts -> %d chunks (delimiter=%r)", len(texts), len(cks), delimiter)
+    if cks and cks[0] == "":
+        cks = cks[1:]
+        result_images = result_images[1:]
+        tk_nums = tk_nums[1:]
    return cks, result_images


@@ -1334,7 +1473,7 @@ def docx_question_level(p, bull=-1):


 def concat_img(img1, img2):
-    from rag.utils.lazy_image import ensure_pil_image, LazyImage
+    from rag.utils.lazy_image import LazyImage, ensure_pil_image

    # Same image must not stack with itself (the LazyImage branch would otherwise
    # concatenate its blob list); mirrors the PIL branch's same-reference guard.
@@ -1603,7 +1742,6 @@ def naive_merge_docx(
    table_context_size=0,
    image_context_size=0,
 ):
-
    if not sections:
        return [], []