## What problem does this PR solve?
`TenantLLMService.model_instance` constructs vision providers with
`lang` as the third positional argument and `base_url` as a keyword
argument.
`LocalAICV` declared `base_url` as its third parameter, causing:
```text
TypeError: LocalAICV.__init__() got multiple values for argument 'base_url'
```
This prevents LocalAI vision models from being used during document
parsing.
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
chore(rag/app): remove stray debug print() calls
Two hot-path debug print() calls were leaking content/error text to
stdout in production code paths.
* rag/app/naive.py: TxtParser branch in chunk() was printing the entire
parsed sections list (formatted via repr()) wrapped in 150-char banner
lines. For large text documents (e.g. a 1000+-page book ingest) this
dumped tens of thousands of lines per ingest into the docker logs.
Replaced with a structured
`logging.info("TxtParser produced %d sections for %s", len(sections),
filename)` so the parse count is still observable without the content
leak.
* rag/app/presentation.py: Pdf.position parsing had a debug
`print(f"Error parsing position: {e}")` inside an except clause in the
ingest hot path. Replaced with
`logging.warning(f"Error parsing position in {filename}: {e}")` to
match the file's existing logging pattern and add filename context.
Both call sites already had logging imported; no new imports added.
logging was used throughout the surrounding code in the same
logging.{info,warning,error}(...) style.
Unifies the Go TokenChunker merge path on a single `mergeUnits` core and
fixes coordinate-tag drift in the Python JSON merge at `overlap > 0`.
Rebased on top of #17979 (delimiter_mode convergence).
Re-materialize wiki page graph from merged wiki_page rows after each
batch merge. Adds ProjectWikiGraph/DropWikiGraph, full page_type/slug
identity, delete-then-insert, tests.
### What problem does this PR solve?
`NvidiaRerank.__init__` only assigned `self.base_url` inside two
model-specific
`if` branches:
```python
if self.model_name == "nvidia/nv-rerankqa-mistral-4b-v3":
self.base_url = urljoin(base_url, "nv-rerankqa-mistral-4b-v3/reranking")
if self.model_name == "nvidia/rerank-qa-mistral-4b":
self.base_url = urljoin(base_url, "reranking")
```
Any other NVIDIA rerank model therefore left the attribute unset, and
the first
`_compute_rank()` call died with `AttributeError: 'NvidiaRerank' object
has no
attribute 'base_url'`.
This is reachable in normal use: `conf/llm_factories.json` ships no
NVIDIA
rerank entries at all, so every NVIDIA rerank model has to be added by
hand,
and any name other than those two hardcoded strings crashes.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Co-authored-by: Alex Ma <alex_makang@hotmail.com>
Converge `TokenChunker.delimiter_mode` from three values (`token_size`,
`delimiter`, `one`) to two (`delimiter`, `one`). The unified `delimiter`
mode now carries the old `token_size` semantics: when no active
(backtick) delimiter is present, text/JSON chunks are merged up to
`chunk_token_size`; when a backtick delimiter is present, the text is
split by it and not merged. `one` continues to be handled by the
separate `OneChunker`.
Ports dataset knowledge compilation (wiki/graph/tree/mindmap) to the Go
scheduler with a status contract, aligns wiki storage/retrieval with
Python, sizes prompts by content_length, and resolves embedding batch
size from provider capability.
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
## Summary
Fixes a regression introduced by #17203 (strict-cap atom-split) and a
secondary delimiter-handling bug from #17723.
**Root cause:**
- #17203 added `_split_oversized_unit` / `_compute_chunk_update`, which
split oversize units into ≤ token_size pieces. This collapsed
`token_size=1` into 1-token chunks and set the cap at 512, mismatching
the model-layer truncation boundary (embedding ~8191 / rerank
500/4096/8192/2048). Atom-split is unnecessary: oversize units stay
whole and the model layer truncates.
- #17723's delimiter handling dropped consecutive delimiters (`A####B`
-> `A##B`), glued JSON items with `"".join`, ignored
`children_delimiters`, and stripped whitespace delimiters.
## Changes
- New pure helper `merge_paragraphs(paragraphs, token_size, strategy)`
with a `MergeStrategy` enum (`UNDER_CAP` / `OVER_CAP`); **default
`OVER_CAP`**. `UNDER_CAP` is a strict cap (never overflows
`token_size`); `OVER_CAP` greedily accumulates adjacent paragraphs while
the projected total stays within `token_size`, merging one
boundary-overflow paragraph before closing. Oversize paragraphs stand
alone.
- `naive_merge` / `naive_merge_with_images` /
`RAGFlowTxtParser.parser_txt` now use `merge_paragraphs`; atom-split
removed. `naive_merge` / `naive_merge_with_images` always split a
section on the delimiter whenever one is present (even when the section
already fits `token_size`), so delimiter text never leaks into a chunk.
Only the empty-delimiter (size-only) mode skips splitting.
- `token_chunker`: delimiter text is dropped (not stripped); JSON flush
joins buffered items with `"\n"`; `children_delimiters` and
`PDF_POSITIONS_KEY` are preserved on the delimiter path. PDF positions
are now attributed **per segment** — each split chunk carries only the
positions of the item(s) that contributed to it — fixing a leak where
page-N coordinates were attached to page-M chunks and all segments
shared one preview image.
- `test_txt_parser.py` rewritten to assert the new contract (not the old
strict cap); `naive_merge` and delimiter-case-sensitive matrices
updated.
## Contract (refs #17799)
- user specified delimiter = chunk boundary; user specified delimiter
text never enters a chunk.
- `token_size` = soft target + merge strategy; no atom-split.
- Default strategy = `OVER_CAP`; migration can switch to `UNDER_CAP`
(strict cap).
- `OVER_CAP` has no hard cap; the model layer truncates oversize units.
`UNDER_CAP` enforces a strict cap.
## Notes
- Closes the wrong-object revert in #17774 (revert #17723 would
re-introduce delimiter-in-chunk and the strict cap).
- Go-side alignment (`internal/ingestion/component/chunker/token.go`) is
a follow-up PR.
---------
Co-authored-by: CodeBuddy <noreply@tencent.com>
## Summary
Migrates `mistralai` from `==0.4.2` to `>=2.7.2,<3.0.0` to unblock the
orjson CVE fix. The old SDK pinned `orjson>=3.9.10,<3.11`, preventing
upgrade to the patched version.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2025-67221 | HIGH | orjson | 3.10.18 | 3.11.6 |
`mistralai` 2.x (the current maintained version) drops the orjson
dependency entirely. Added `orjson>=3.11.6` to `constraint-dependencies`
to pin the floor for remaining parent packages (`langgraph-sdk`,
`langsmith`, `ranx`).
## What
Adds [**SereneDB**](https://serenedb.com) as a selectable doc-store
engine on **both** RAGFlow paths:
- the **Go** `DocEngine` (`internal/engine/serenedb`), alongside
Elasticsearch and Infinity;
- the **Python** `DocStoreConnection` (`rag/utils/serenedb_conn.py`) +
`DOC_ENGINE=serenedb` registration.
SereneDB is a PostgreSQL-wire engine (DuckDB execution) whose single
inverted index carries **both** a scored text column (`@@`, BM25) and an
IVF vector column (`<#>`, inner product), so hybrid search is one SQL
statement. The Go engine connects with `database/sql` + `lib/pq`
(already a dependency, no new module); the Python connector uses
psycopg2 (already a dependency).
## Storage model
One table per tenant with `kb_id` as a filter column - the
**Elasticsearch / OceanBase** model, not Infinity's per-dataset tables.
This keeps BM25 statistics (IDF, avgdl) computed over the whole tenant
corpus (global IDF). Both connectors use this identical layout, so they
are storage- and retrieval-compatible: `hybrid` proxy routing and
Python↔Go switching are safe. On the Python side the connector is wired
as OceanBase's plain-SQL sibling (chunk_data JSON metadata, inline chunk
vectors, verbatim ES field names); the ES tokenizer path is unchanged.
Metadata stays one table per tenant (`ragflow_doc_meta_<tenant>`).
The query shapes mirror the Python connector, including the five
empirically-found landmines: the scored dictionary needs `frequency +
norm` (else `BM25()` silently returns 0.0), the `@@` query is the
tokenized query, the scored lexical branch matches one column, vectors
use an L2-normalized shadow column with `ip`/`sq8`, and the similarity
threshold goes directly in the ANN scan's `WHERE`. **Minimum engine
version: SereneDB 26.07.4.**
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
`TokenChunker._invoke` discarded the user's configured `delimiters`
whenever no backtick-wrapped delimiter was present: it passed a
hardcoded `""` to `naive_merge`, so the configured delimiters (including
the default `["\n"]`) were never forwarded. `naive_merge` then ignored
the newline sentence boundary and cut chunks **mid-sentence** once the
token budget was exceeded.
## Root cause
`token_chunker.py:326` called `naive_merge(payload, chunk_token_size,
"", overlapped_percent)`. `_compile_delimiter_pattern` intentionally
returns `""` for bare (non-backtick) delimiters — that return value is a
*path selector* (empty → token-budget merge; non-empty → hard
`_split_text_by_pattern` split). The bug was not in that selector but in
the `else` branch, which threw away `self._param.delimiters` instead of
forwarding it.
## Fix
Forward the configured delimiters as a soft boundary:
```python
else naive_merge(
payload,
self._param.chunk_token_size,
"".join(self._param.delimiters),
overlapped_percent,
)
```
`naive_merge` already parses the string via the canonical
`parse_delimiter_field`, so bare and backtick-wrapped delimiters are
honored as soft boundaries while the token budget is still respected.
The path-selection role of `_compile_delimiter_pattern` is untouched:
backtick-wrapped delimiters still select the hard
`_split_text_by_pattern` path; bare delimiters still take the
token-budget merge path. No regression for any previously-working
(wrapped-delimiter) configuration.
## Test
Adds `rag/flow/tests/test_token_chunker_delimiter.py`:
- `test_token_chunker_token_size_mode_does_not_split_sentences` — fails
on the old code (3 sentences cut mid-stream), passes after the fix.
- `test_naive_merge_empty_delimiter_ignores_newline_break` — root-cause
companion asserting `naive_merge("")` cuts while `naive_merge("\n")`
preserves boundaries.
Both tests skip when the tokenizer is unavailable (dead-tokenizer
guard).
## Note
This branch contains only this one-line fix on top of `main`; it is
intentionally independent of the unrelated tokenizer/offline-BPE work.
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
