Follow-up to #17526 ("Refactor: merge dataset scope graph"), which introduced two code paths that touch Infinity columns the deployed schema does not declare. This PR makes the runtime robust against the old schema while also adding the new column to the new schema so freshly created tables are correct.
Re-materialize wiki page graph from merged wiki_page rows after each
batch merge. Adds ProjectWikiGraph/DropWikiGraph, full page_type/slug
identity, delete-then-insert, tests.
## Problem
`kg_test.go`'s `getTestConfig()` returned `map[string]interface{}`,
which no longer matches `NewEngine`'s `config.ElasticsearchConfig`
signature after the config package moved to `internal/server/config`.
This broke compilation of the `elasticsearch` package under `-tags
integration` (kg_test.go:38).
## Fix
Make `getTestConfig()` return the typed `config.ElasticsearchConfig` so
the integration tier builds again.
## Scope
Fixes test compilation only; no production code changed. Unrelated to PR
#17802 (kb_id single-string + T0 read-back baseline), so it is tracked
in its own PR to keep that refactor focused.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
## Summary
- `ProcessChunksForPipeline` now sets `kb_id` to a plain string instead
of `[]string{kbID}`, removing an index-physical array shape from the
ingestion domain.
- Stored documents are byte-identical: Elasticsearch overrides `kb_id`
with `datasetID` on write, and Infinity's `transformChunkFields` already
accepts a plain string.
- Infinity is intentionally left unchanged — `service/chunk` paths still
feed `kb_id` as `[]string`, and Infinity handles both forms. The
`dataset` artifact merge (`dataset_artifact_service.go`) is out of scope
for this step.
- Unit assertion updated to expect a string.
## Scope / non-goals
This is the smallest first step (T1) of the index-schema leak cleanup
tracked in #17371. It does **not** move the other leaks (`docnm_kwd`,
`create_timestamp_flt`, position ints) to the engine boundary — those
are later steps behind a read-back golden test.
## Test plan
- `go test ./internal/ingestion/task/indexdoc/...` passes.
- The two `task` "Real" integration tests fail identically on a clean
tree (environment lacks real embedding/parsing); they are pre-existing,
unrelated to this change.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.
- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
(integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
`AGENTS.md`.
## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |
## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
`-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
component packages directly.
## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
### Summary
Aligns Go and Python error codes/messages so both backends honor the
same RESTful API contract, removing implementation-specific error leaks
(MySQL errors, `ValueError`, `AttributeError`, Gin validator format) in
favor of clean business error codes.
**Chat list** — invalid `orderby` now returns code 101 (was: raw Python
`AttributeError` code 100); invalid `page`/`page_size` values fall back
to defaults (was: raw `ValueError`/`ProgrammingError` code 100).
**Dataset create/update/delete** — adds UUID validation (101),
extra-field rejection (101), duplicate-id detection (101), content-type
/ JSON-syntax / object-shape checks (101), and "lacks permission" for
nonexistent datasets (IDOR). Create auto-deduplicates dataset names.
Pagerank updates tolerate a missing ES index. List response includes
`parser_config` and `pagerank`.
**Session list/update** — adds filtering, sorting, and pagination
support. Empty payloads are valid no-ops. Authorization errors map to
code 109.
**Chunk list** — doc object uses Python key names (`chunk_count`,
`dataset_id`, `chunk_method`, run text status). Add validates list
element types.
**Document update** — adds `chunk_method` alias, pydantic-style Field
error messages, metadata index auto-create with refresh, and "These
documents do not belong to dataset" messages. List validates
`metadata_condition` and reports ownership errors for unmatched name/id
filters.
**Search completion** — `kb_ids` ownership failure returns code 102
instead of 109.
Released 31 contract tests from `GO_ONLY_SKIPS` (all verified passing on
both Go and Python backends with real LLM keys).
### Summary
As title:
Some operations on datasets have been fixed as they previously lacked
authentication, preventing team members from proper or correct usage.
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
## Summary
Fix Elasticsearch-backed skill search by mapping skill search fields to
their indexed token fields.
`name`, `tags`, `description`, and `content` are stored for display but
are not searchable in the skill ES mapping. Search queries now target
`name_tks`, `tags_tks`, `description_tks`, and `content_tks`.
## Testing
- Ran Go unit tests:
```bash
/usr/local/go/bin/go test -count=1 ./internal/engine/elasticsearch
```
- Frontend verification:
1. Open /files/skills.
2. Enter a skill space.
3. Reindex the skill space if existing skills were created before this
fix.
4. Search by skill name or description keyword.
5. Confirm matching skills are returned.
### Summary
Plan to start api_server, admin_server and ingestor in one binary:
- ./ragflow_main --admin
- ./ragflow_main --api
- ./ragflow_main --ingestor
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### Summary
As title
Before, it return `update success` but never insert or update any
metadata
fixed:
```go
_, err = s.docEngine.InsertMetadata(nil, []map[string]interface{}{
{
"id": docID,
"kb_id": doc.KbID,
"meta_fields": meta,
},
}, tenantID)
```
Replace fragile wall-clock timeout assertions with semantic checks for
deadline errors, retry suppression, and event ordering. Keep only
lower-bound timing checks where they prove backoff behavior. This
reduces CPU-load flakes without weakening regression coverage.
### Summary
```
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id> DELETE
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id>/feedback PUT
```
Migrates the chat session message delete and feedback APIs to the Go
server, matching the Python behavior for authorization, session
ownership checks, message/reference updates, and feedback validation.
### Summary
```
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id> DELETE
/api/v1/chats/<chat_id>/sessions/<session_id>/messages/<msg_id>/feedback PUT
```
Migrates the chat session message delete and feedback APIs to the Go
server, matching the Python behavior for authorization, session
ownership checks, message/reference updates, and feedback validation.
### Testing
- `/usr/local/go/bin/go test ./internal/service ./internal/handler`
- Verified through the frontend page for deleting chat messages and
updating message feedback
### What problem does this PR solve?
As title:
implement:
```
/api/v1/messages/search GET
/api/v1/messages GET
/api/v1/messages/<memory_id>:<message_id>/content GET
/api/v1/memories/<memory_id>/config GET
/api/v1/messages/<memory_id>:<message_id> PUT
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Implement OpenAI chat completions in GO
POST /api/v1/openai/<chat_id>/chat/completions
OpenAI chat cli: internal/development.md
### Type of change
- [x] Refactoring
## Summary
- Implemented the Go API endpoint for Memory message forgetting:
- `DELETE /api/v1/messages/{memory_id}:{message_id}`
- Added route registration for the Memory message DELETE endpoint only.
- Added request path validation for `memory_id:message_id`.
- Added service logic to mark a message as forgotten by setting
`forget_at`.
- Preserved Python-compatible response behavior:
- Success returns `code: 0`, `message: true`, `data: null`.
- Added focused unit tests for message path parsing and invalid message
ID handling.
- Fixed Linux cgo linker config to use the installed shared PCRE2
library so Go tests/builds can run in this environment.
## Related Issue
Closes: #15240
## Change Type
- [x] Feature
- [x] Test
- [x] Build / CI compatibility
## Implemented API
- `DELETE /api/v1/messages/{memory_id}:{message_id}`
## Real Behavior Proof
Validated with targeted Go tests:
```bash
/tmp/go1.25.0/bin/go test ./internal/handler ./internal/router
```
Result:
```text
ok ragflow/internal/handler
? ragflow/internal/router [no test files]
```
Validated server entrypoint build:
```bash
/tmp/go1.25.0/bin/go build -o /tmp/ragflow-server-main ./cmd/server_main.go
```
Result:
```text
build succeeded
```
Validated patch formatting:
```bash
git diff --check
```
Result:
```text
no whitespace errors
```
## Checklist
- [x] Implemented only `DELETE
/api/v1/messages/{memory_id}:{message_id}`.
- [x] Did not implement unrelated Memory message APIs.
- [x] Added route registration.
- [x] Added handler validation.
- [x] Added service-level memory access check.
- [x] Added tests.
- [x] Ran targeted Go tests.
- [x] Ran server build validation.
- [x] Ran `git diff --check`.
## Summary
- Complete the Go Elasticsearch result functions that remained stubbed
after #15160.
- Add focused unit coverage for field mapping, aggregation, IDs, and
highlighting behavior.
- Update a stale query-builder test type import discovered during
validation.
## What changed
- Keep the Elasticsearch Go implementation merged in #15160 and fill in
`GetFields`, `GetAggregation`, `GetHighlight`, and `GetDocIDs` in
`internal/engine/elasticsearch/chunk.go`.
- Add regression and invariant coverage in
`internal/engine/elasticsearch/chunk_helpers_test.go`.
- Update `internal/service/nlp/query_builder_test.go` to use the current
`types.MatchTextExpr` type.
## Why
- #15160 implemented the main Go Elasticsearch surface, but
retrieval/tag flows still call result functions that returned stubs.
- Completing these functions keeps Elasticsearch result processing
aligned with the expected document-engine behavior for field extraction,
tag aggregation, doc ID extraction, and snippet highlighting.
## Validation
- `go test ./internal/engine/elasticsearch`
- `GOARCH=arm64 CGO_ENABLED=1 go test ./internal/service/nlp -run
TestQueryBuilder`
- `git diff --check`
- CodeRabbit review reported 0 issues after follow-up fixes.
- Codex Security diff scan found no reportable issues.
## Notes
- This PR is now a follow-up to #15160 rather than a competing
implementation.
- A full local `go test ./internal/service/nlp` run is blocked by local
WordNet resource prerequisites; the query-builder tests touched by this
PR pass with the arm64 CGO path.
### What problem does this PR solve?
```
RAGFlow(user)> ocr with 'hunyuanocr@test@gitee' file './picture.png'
+----------------------------------------------------------+
| text |
+----------------------------------------------------------+
| 生活不是等待风暴过去，而是学会在雨中翩翩起舞。
——佚名 |
+----------------------------------------------------------+
RAGFlow(user)> list 'test@gitee' tasks;
+---------+----------------------------------+
| status | task_id |
+---------+----------------------------------+
| success | C3FX4MQNKY5MGC6ZFMIXIAMJKHCEBQB5 |
+---------+----------------------------------+
RAGFlow(user)> show 'test@gitee' task 'C3FX4MQNKY5MGC6ZFMIXIAMJKHCEBQB5';
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| content | index |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| # PDF 1: Purpose of RAGFlow
RAGFlow is an open source Retrieval-Augmented Generation (RAG) engine designed to turn raw documents into reliable context for large language models.Its purpose is to make it practical to build an Al assistant that can ans... | 1 |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
```
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
---------
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Implement Delete in GO and refactor functions
### Type of change
- [x] Refactoring
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a remove_chunks command to delete specific or all chunks from a
document.
* Added new endpoints for chunk removal and chunk update.
* **Refactor**
* Renamed index commands to dataset/metadata table terminology and
updated REST routes accordingly.
* Updated chunk update flow to a JSON POST style and improved metadata
error messages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
### What problem does this PR solve?
Implement UpdateDataset and UpdateMetadata in GO
Add cli:
UPDATE CHUNK <chunk_id> OF DATASET <dataset_name> SET <update_fields>
REMOVE TAGS 'tag1', 'tag2' from DATASET 'dataset_name';
SET METADATA OF DOCUMENT <doc_id> TO <meta>
### Type of change
- [ ] Refactoring
