### Summary
Refs #17885.
Mistral figure enrichment now receives the dataset language through the
production parsing path. `by_mistral_ocr` forwards `lang` to
`MistralParser.parse_pdf`; the parser stores the normalized language and
passes it to the figure-description prompt. Empty or missing values
still fall back to English.
### Summary
Brings both halves of the Tenki sandbox provider onto current SDKs and
removes `project_id`, which Tenki deleted from its API.
**Go:** `github.com/LuxorLabs/tenki-sdk-go/sandbox` `v0.5.2` → `v0.7.0`
(current latest).
**Python:** the provider's SDK was renamed on PyPI — `tenki-sandbox` is
frozen at 0.4.0 and everything from 0.5 ships as
[`tenki`](https://pypi.org/project/tenki/). The docs told operators to
`pip install tenki-sandbox`, which installs a stale SDK that no longer
matches this provider's expectations.
**`project_id` is gone.** Tenki removed project scoping from the sandbox
API in 0.5.x: `Client.create()` no longer accepts `project_id`, so the
current code path would raise `TypeError` against a current SDK. It was
also marked `required: True` in the config schema, so the Admin >
Sandbox Settings form asked for a value that no longer exists.
### What problem does this PR solve?
Incremental Wiki compilation could lose provenance for claim-light
entities, produce unstable page groups across embedding models, route
entities to unrelated pages, and assign topics without sufficient
page-level context. Document removals and page membership changes could
also leave stale Wiki state.
This PR:
- preserves source document and chunk provenance throughout entity
matching, reduction, page generation, and deletion;
- uses embeddings to retrieve candidates and the LLM to make final page
grouping and incremental routing decisions;
- batches embedding and LLM operations with bounded concurrency and
deterministic fallbacks;
- selects source-scoped topic candidates with embeddings before the page
LLM chooses the final topic;
- rebuilds Wiki state when the compilation mode or embedding model
changes;
- normalizes Wiki array fields returned by the API and retains entities
without relations in graph responses.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
## What
This pull request adds **MWS GPT Model Hub** as a built-in model
provider in RAGFlow.
The integration allows users to configure an MWS project endpoint and
token, discover the models available to that project, and use supported
MWS models for chat completion, embeddings, and reranking.
Co-authored-by: ilarionov_n <ilarionov_n@promis.ru>
Centralize the shared golden-doc + alignment helpers in `align_test.go` so the format-specific PRs (text&code, markdown golden, HTML) reuse one implementation instead of each carrying their own copy of the scaffolding.
Golden-parity test infrastructure for the **Go `TokenChunker` ↔ Python alignment**.
It runs the Go chunker over a committed case set (`testdata/parity/cases/`) and diffs each output against a captured Python golden (`testdata/parity/golden/`), honoring a `known_diffs.json` ratchet (`extra_fields` / `chunk_count` / `chunk_text`) so accepted divergences are tracked rather than silently widening.
Port dataset-level wiki incremental compile and refactor splitByTokens
token budgeting. Includes replace-only wiki merge, KNN dedup routing,
and template/config wiring.
## Summary
This PR improves the RAGFlow agentic-search path in three areas: it
stops the outer agent from re-looping over the same rag call, lets the
medium thinking mode discover and follow new sub-claims mid-loop, and
strengthens retrieval by having the LLM emit synonym-rich queries with
time/date/number terms boosted.
1. Avoid the outer re-loop — keep all multi-hop cycles inside agentic
RAG
2. Dynamic claims in medium mode — keep querying newly discovered
sub-questions
medium now enables allows_dynamic_claims. During orchestration, when
claim analysis discovers a new required sub-question
(discovered_claims), the loop spawns it as a new ClaimTarget and
continues searching it in subsequent cycles (bounded by the
dynamic-claim budget) instead of stopping. Also added:
3. Stronger query strategy — synonym-rich queries + time/date/number
weighting
LLM-generated synonyms: the claim-analysis prompt now instructs the
model to write each next_queries entry as a retrieval-boosted query that
actively folds in entity aliases, DATE/TIME synonyms (e.g. 1994 → 1994,
66th Academy Awards), and number/unit variants (e.g. 1.95 m → 6 ft 5
in).
Time/date/number boosting: query.py boosts numeric/date tokens to a high
weight (_NUM_DATE_TOKEN_RE).
