zlei6/ragflow
1
0
Fork 0
mirror of https://github.com/infiniflow/ragflow.git synced 2026-08-23 00:31:06 +08:00
Code Issues Packages Projects Releases Wiki Activity
8,395 Commits 2 Branches 115 Tags
fa386ddf101c03be7258c19cd898b8ca07f00045
Commit Graph

51 Commits

Author SHA1 Message Date
Lynn
344bd4d57a Fix: correct chunk order for non‑PDF documents (#18412) 2026-08-18 13:06:13 +08:00
Lynn
2368c54689 Fix: handle title and truncate document preview by sentence (#18407) 2026-08-18 11:54:55 +08:00
S
c6ba54bc72 fix: stop three recurring errors on doc delete (#17685) (#17686) 
Follow-up to #17526 ("Refactor: merge dataset scope graph"), which introduced two code paths that touch Infinity columns the deployed schema does not declare. This PR makes the runtime robust against the old schema while also adding the new column to the new schema so freshly created tables are correct.
 2026-08-16 09:25:51 +08:00
qinling0210
554fb1133a Search in navigation tree & store graph content in nav_doc (#18303) 
### Summary

Search in navigation tree & store graph content in nav_doc during parse
 2026-08-14 22:26:57 +08:00
buua436
024c35f085 fix: track wiki chunk changes incrementally (#18285) 2026-08-14 20:11:55 +08:00
buua436
15a63bc15c fix: exclude disabled documents from wiki compilation (#18261) 2026-08-14 13:45:14 +08:00
buua436
a4e819504c feat: support entity and topic wiki modes (#18216) 2026-08-13 19:03:24 +08:00
buua436
649b631cac fix: improve incremental wiki compilation (#18164) 2026-08-12 19:12:39 +08:00
Wang Qi
3e4a1fa887 Add doc_name for compilation generated chunk (#18126) 2026-08-12 09:37:25 +08:00
buua436
0cfd8f41e4 fix: improve incremental wiki compilation (#18130) 
### What problem does this PR solve?

Incremental Wiki compilation could lose provenance for claim-light
entities, produce unstable page groups across embedding models, route
entities to unrelated pages, and assign topics without sufficient
page-level context. Document removals and page membership changes could
also leave stale Wiki state.

This PR:

- preserves source document and chunk provenance throughout entity
matching, reduction, page generation, and deletion;
- uses embeddings to retrieve candidates and the LLM to make final page
grouping and incremental routing decisions;
- batches embedding and LLM operations with bounded concurrency and
deterministic fallbacks;
- selects source-scoped topic candidates with embeddings before the page
LLM chooses the final topic;
- rebuilds Wiki state when the compilation mode or embedding model
changes;
- normalizes Wiki array fields returned by the API and retains entities
without relations in graph responses.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
 2026-08-11 20:13:04 +08:00
Lynn
15ab2b3ee2 Fix: update tree when delete skill node (#18063) 2026-08-11 09:49:22 +08:00
buua436
6f50e478e0 fix: exclude disabled documents from dataset structures (#18041) 2026-08-10 17:50:34 +08:00
buua436
c0582b8e18 fix: handle disabled documents in dataset structures (#18033) 2026-08-10 13:52:49 +08:00
buua436
8379165c12 fix: record generated wiki page versions (#17931) 2026-08-06 16:49:50 +08:00
buua436
97f9ae5896 refa: unify wiki example naming (#17910) 2026-08-06 13:24:27 +08:00
buua436
d16b2556a2 fix: split wiki template instruction and example (#17891) 2026-08-06 09:41:57 +08:00
Jin Hai
cf13082a1a Revert "feat: Go knowledge compiler with scheduler-driven dataset compilation" (#17897) 
Reverts infiniflow/ragflow#17881
 2026-08-05 21:50:28 +08:00
Zhichang Yu
14b943a04a feat: Go knowledge compiler with scheduler-driven dataset compilation (#17881) 
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
 2026-08-05 20:00:42 +08:00
buua436
502921159a fix: use updated RAPTOR options for skill generation (#17859) 2026-08-05 19:27:17 +08:00
buua436
1d141aff18 fix: optimize dataflow indexing and logs (#17737) 2026-08-03 19:15:44 +08:00
buua436
3e7cfbe052 refa: simplify RAPTOR tree clustering and configuration (#17614) 2026-08-03 17:46:50 +08:00
Kevin Hu
3b4a96d421 Refactor: refine wiki plan procedure. (#17579) 
### Summary

Refine wiki plan procedure.

---------

Co-authored-by: Yingfeng Zhang <yingfeng.zhang@gmail.com>
Co-authored-by: buua436 <sz_buua@foxmail.com>
 2026-08-03 16:03:17 +08:00
Lynn
51af9e2eae Fix: set chunk_num before set_progress, avoid race condition (#17724) 2026-08-03 15:52:23 +08:00
buua436
44e13d1cb6 feat: add LLM-guided semantic rechunking for knowledge compilation (#17546) 2026-07-31 13:33:30 +08:00
Kevin Hu
3f8a3dfcff Feat: Compilation benefit naive rag. (#17555) 
### Summary

Compilation benifit naive rag

---------

Co-authored-by: Yingfeng Zhang <yingfeng.zhang@gmail.com>
 2026-07-30 18:39:01 +08:00
Kevin Hu
9bb037271e Refact: Rename "artifact_" to "wiki_". (#17553) 2026-07-30 11:06:16 +08:00
Kevin Hu
48a3280eac Refactor: merge dataset scope graph. (#17526) 
### Summary

merge dataset scope graph.

---------

Co-authored-by: Yingfeng Zhang <yingfeng.zhang@gmail.com>
 2026-07-29 18:23:51 +08:00
Kevin Hu
742837ce56 Feat: Add graph keyword search and fix dataset synthesizing issue. (#17342) 
### Summary

 Add graph keyword search and fix dataset synthesizing issue.
 2026-07-24 18:00:43 +08:00
buua436
2ba7ccecaf fix: stabilize knowledge compilation navigation updates (#17345) 2026-07-24 17:48:27 +08:00
Yingfeng
08e5d9c6be Fix infinity for compilation (#17313) 2026-07-23 21:43:56 +08:00
Kevin Hu
3e4c6dfc0a Feat: refine the tree navigation during compilations (#17140) 2026-07-23 18:39:16 +08:00
buua436
d4a8c91f3c refa: improve tree clustering (#17285) 2026-07-23 17:49:13 +08:00
buua436
285f3e4c7e refa: optimize wiki compilation concurrency (#17192) 2026-07-23 09:56:51 +08:00
Wang Qi
2223a514de Fix cancel ingest task, it will be stilling running and show internal server error (#836) (#16945) 2026-07-15 18:33:51 +08:00
buua436
4391e03886 refa: optimize knowledge compilation concurrency (#16933) 
### What problem does this PR solve?

Improve concurrency in the knowledge compilation pipeline:

- Run Compile LLM requests concurrently while preserving ordered
commits.
- Run merge flush tasks concurrently while keeping ES writes ordered.
- Improve concurrency for local deduplication, chain validation, and ES
deduplication.
- Remove temporary debugging instrumentation and unused timing
variables.

### Type of change

- [x] Refactor (no functional change)
 2026-07-15 16:24:35 +08:00
Lynn
cc94639555 Fix: get_by_id (#16765) 2026-07-09 14:52:41 +08:00
buua436
6a77523bf0 refa: resolve tenant model refs consistently (#16744) 2026-07-09 14:02:08 +08:00
Lynn
1430d0e431 Fix: provider name (#16733) 2026-07-09 10:19:10 +08:00
Kevin Hu
2c59d07bdb Feat: add wiki folder (#16749) 
### Summary

Add wiki folders.
 2026-07-08 20:08:14 +08:00
Wang Qi
c8d1b21ae3 Fix Build-in metadata not working (#788) (#16748) 2026-07-08 19:06:54 +08:00
Lynn
0ae5961e1c Feat: v0.27.0 model provider (#16604) 2026-07-08 09:47:29 +08:00
Wang Qi
705754ea8b Fix PageIndex is not working (#16704) 
Follow on PR #16515
 2026-07-07 18:09:05 +08:00
Wang Qi
2de5940325 Fix cannot run raptor (#16694) 2026-07-07 17:02:04 +08:00
Kevin Hu
52f985f43e Refactor: Remove redundant functions. (#16671) 
### Summary

Remove redundant functions.
 2026-07-06 19:02:25 +08:00
Yingfeng
706fa4e87a Feat: add gbrain compile template for session/memory data (#16613) 2026-07-03 18:22:29 +08:00
Wang Qi
6a4b9be426 Refactor: reformat all code for lefthook using ruff and gofmt (#16585) 2026-07-03 12:53:39 +08:00
Kevin Hu
62f94cd59b Feat: Add knowledge compilation workflows (#16515) 
## Summary
- Add knowledge compilation template APIs, services, and builtin
template seed data
- Add advanced knowledge compile structure/artifact/RAPTOR workflow
support
- Update parsing, dataset/document APIs, and supporting services for
compilation workflows
 2026-07-02 23:22:07 +08:00
euvre
d9a04ef702 fix: support auto mode in table parser document metadata aggregation (#15780) 
### What problem does this PR solve?

Table parser metadata aggregation previously only ran when
`table_column_mode` was set to `manual`. In auto mode (default), all
columns default to `"both"` role, meaning they should also be aggregated
into document-level metadata for UI/chat filters. Additionally, the task
snapshot could be stale — `table_column_names` are written to KB
`parser_config` during `chunk()` but the task may have been created
before that.

Changes:
- Renames `aggregate_table_manual_doc_metadata` →
`aggregate_table_doc_metadata`
- Supports both `"manual"` and `"auto"` `table_column_mode` (defaults to
`"auto"`)
- Reloads `table_column_names` from KB DB when missing from task
snapshot
- Removes the manual-only guard in `task_executor` and refactored
`post_processor`
- Updates all tests with new function name and adds auto mode test cases

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
 2026-06-08 19:08:23 +08:00
Jack
b363146997 refactor: overhaul task executor with layered architecture and comprehensive test suite (#15471) 
## Summary

Decomposes the monolithic `task_executor.py` (1945 lines) into a 6-layer
architecture with clear separation of concerns. The refactored code is
functionally equivalent to the original, verified through 400 passing
tests and a production-vs-dry-run comparison framework.

## Architecture

```
entry (task_manager)
  └─ orchestration (task_handler)
       ├─ services (chunk_service, embedding_service, dataflow_service, raptor_service, post_processor)
       │    └─ utilities (chunk_builder, chunk_post_processor, embedding_utils)
       └─ infrastructure (task_context, recording_context, interceptor)
```

Key design decisions:
- **TaskContext** — typed facade over raw task dict, injects rate
limiters + callbacks via composition
- **RecordingContext + Comparator** — enables side-by-side production vs
dry-run execution for safe migration
- **NullRecordingContext** — zero-allocation no-op for production, uses
`__slots__`
- **WriteOperationInterceptor** — FIFO replay of previous runs function
returns for comparison mode

## Migration Strategy

The original `handle_task()` in `task_executor.py` uses a 3-way switch
via `TE_RUN_MODE`:
- `TE_RUN_MODE=0` (default) → runs refactored code
- `TE_RUN_MODE=1` → runs both original + refactored, compares all
intermediate results
- `TE_RUN_MODE=2` → runs original code (fallback)

The comparison mode (`TE_RUN_MODE=1`) records ~40 intermediate values
(chunks, vectors, token counts, func return values) from the production
run and replays them during dry-run, then uses `ContextComparator` to
report mismatches.

## Functional Equivalence Fixes

All divergences between original and refactored code were identified and
fixed:
- Timeout decorators (handle/build_chunks/raptor/embedding)
- NullRecordingContext leak in finally block causing RuntimeError
- MinIO None-binary check with proper FileNotFoundError
- Dataflow dispatch after embedding binding + init_kb
- Memory task missing return after processing
- RAPTOR checkpoint progress reporting
- Tag cache (get_tags_from_cache/set_tags_to_cache) restoration
- dataflow_id correction in _load_dsl
- Language default Chinese, dead code guard removal
- embed_chunks made async with proper thread_pool_exec
- Full GraphRAG default configuration (10 parameters)
- Hardcoded q_768_vec fallback removal in RAPTOR

## Test Changes

- 20 new tests covering table parser manual mode, tag cache, embedding
edge cases, RAPTOR checkpoint, dataflow_id correction, storage binary
None, cancel cleanup, metadata=None boundary
- Unified `make_task_context`/`make_task_dict` factories eliminated 10+
duplicated helpers
- DataflowService tests migrated from internal method mocks to IO
boundary mocks (real orchestration code executes)
- Parametrized duplicate build_chunks post-processor tests
- 7 raptor tests modernized to @pytest.mark.asyncio
- Mock count per test reduced through boundary-level mocking strategy

**Test count: 400 passing, 0 warnings, 0 skips**

## Files Changed

| File | Change |
|------|--------|
| `rag/svr/task_executor.py` | +1 line (NullRecordingContext fix) |
| `rag/svr/task_executor_refactor/task_handler.py` | Orchestration
layer, 8 logic fixes |
| `rag/svr/task_executor_refactor/chunk_service.py` | +timeout +
None-check |
| `rag/svr/task_executor_refactor/embedding_service.py` | sync→async
rewrite |
| `rag/svr/task_executor_refactor/dataflow_service.py` | dataflow_id fix
+ timeout |
| `rag/svr/task_executor_refactor/raptor_service.py` | checkpoint fix +
assert |
| `rag/svr/task_executor_refactor/chunk_post_processor.py` | tag cache
restore |
| `rag/svr/task_executor_refactor/task_context.py` | language default
fix |
| `test/.../conftest.py` | +294 lines shared helpers |
| `test/.../*.py` | 15 test files refactored, 20 new tests |

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 17:18:31 +08:00
Lynn
dc4b82523b Feat: tenant llm provider (#14595) 
### What problem does this PR solve?

Python implementation of the Go-based model_provider API suite.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: bill <yibie_jingnian@163.com>
 2026-05-29 17:39:41 +08:00