### Summary
Add Querit Contents as a built-in page content tool for RAGFlow Agents
and Canvas workflows.
Querit Contents crawls one or more known URLs and returns their page
content and optional metadata. It complements the existing Querit Search
tool: Search discovers relevant pages, while Contents retrieves pages
already selected by an Agent or workflow.
This integration supports two usage modes:
- A standalone `QueritContents` node in Canvas workflows.
- An embedded content tool available to RAGFlow Agents.
Make `ResolveModelContentLength` honor the per-model custom **context window length** (`content_length`) — stored in the Python-legacy `tenant_model.extra["max_tokens"]` field, whose semantic meaning is the context window, NOT the generation cap — **before** any provider-catalog read, and remove the parallel service-layer implementation so every consumer shares one resolution path.
Switch the agent LLM component's `fitMessages` from the local map-based `messageFitInRaw`/`countAllTokens`/`stringContent`/`setContent` helpers to the new shared `internal/component/messagefit` package (#18091).
Re-materialize wiki page graph from merged wiki_page rows after each
batch merge. Adds ProjectWikiGraph/DropWikiGraph, full page_type/slug
identity, delete-then-insert, tests.
Ports dataset knowledge compilation (wiki/graph/tree/mindmap) to the Go
scheduler with a status contract, aligns wiki storage/retrieval with
Python, sizes prompts by content_length, and resolves embedding batch
size from provider capability.
Ports the dataset knowledge compilation (wiki/graph/tree/mindmap) to the
Go scheduler with a status contract, aligns wiki storage/retrieval with
Python, and sizes prompts by content_length.
- Prefer canonical `dataset_ids` over legacy `kb_ids` in Canvas
retrieval components.
- Cover selected and explicitly cleared dataset IDs with regression
tests.
- Add cross-language retrieval with dataset-specific embedding and
rerank models.
- Support vector and keyword similarity controls, metadata filtering,
TOC enhancement, and child-chunk expansion.
- Route Canvas retrieval across datasets and memories with compatible
embedding validation.
- Start the Agent model-stream collector before ReAct execution.
- Preserve all streamed reasoning/content deltas and drain the collector
on errors.
- Add coverage for delayed thinking streams and tool-call execution.
## Summary
- Normalize Canvas component names before resolving Go Agent tools and
parameters.
- Add regression coverage for CodeExec and other Canvas tool mappings.
## Testing
- `CGO_ENABLED=0 go test -count=1 ./internal/agent/tool
./internal/agent/component`
Port Python rag/advanced_rag agentic search to Go: ES-backed dataset-nav
service, agentic-search harness, and agent tools.
Includes agentic-search port plan and self-review docs.
## Summary
- Align Go agent logs access with the Python behavior by checking tenant
access to the target agent instead of requiring the token to be bound
to that agent.
- Marshal canvas lifecycle event payloads safely so non-serializable
inputs do not break Thinking expansion in the UI.
## Testing
- `CGO_ENABLED=0 go test -count=1 -v -run
"Handler|Attachment|Logs|Chatbot|Agentbot|BuildWorkflow|NodeLifecycle|VarRef|Resolve|Extract"./internal/handler/
./internal/agent/canvas/...` PASS
## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.
- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
(integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
`AGENTS.md`.
## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |
## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
`-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
component packages directly.
## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
### Summary
Adds a `tenki` sandbox provider that runs each agent code execution in a
disposable Tenki (https://tenki.cloud) microVM (create → exec → destroy,
no volumes or snapshots).
Registration mirrors PR #15039, configure `api_key` and `project_id` in
Admin > Sandbox Settings.
Both runtimes are covered:
- Python: `agent/sandbox/providers/tenki.py` (structured results +
artifact collection).
- Go: `internal/agent/sandbox/tenki.go`, mirroring the e2b provider and
wired into the provider manager.
`tenki-sandbox` is an optional dependency (it requires `protobuf>=6.31`,
which differs from RAGFlow's pinned gRPC stack), lazily imported with a
clear error when missing; installation is documented in the sandbox
quickstart.
Unit tests cover execution, structured results, artifacts
(symlink/size/extension limits), non-zero exit, timeout, error mapping,
and idempotent destroy.
---------
Co-authored-by: yiming.wang <yiming.wang@luxor.com>
## Summary
This PR hardens the Go ingestion **Extractor** and **LLM retry** paths
and closes several Python->Go parity gaps in the keyword/question/tag
extraction flow.
- **Generic retry utility** (`internal/common/retry.go`):
`RetryWithBackoff` with exponential backoff (default 3 retries, 2s
initial delay, capped at 1m), context-aware sleep, and a `maxRetries<=0`
fast path. Covered by `internal/common/retry_test.go`.
- **LLM retry reuse**: `agent/component/llm_retry.go` now delegates to
`common.RetryWithBackoff` instead of an inline loop (behavior preserved:
ctx cancellation short-circuits the backoff).
- **Extractor LLM calls** (`extractor.go`):
- `call()` now retries transient LLM failures via `RetryWithBackoff`
(retry exhaustion fails the chunk instead of silently skipping).
- Sets `temperature = 0.2`, matching Python `generator.py:230,245`.
- Runs keyword and question extraction **concurrently** per chunk when
both are enabled (`task_executor.py:444-448`), with mutex-guarded map
writes to avoid data races.
- Substitutes `{field_name}` placeholders (including `{chunks}` -> chunk
text) in `prompt`/`system_prompt` before the call, mirroring Python
`string_format` (`extractor.py:102-103`); unmatched placeholders are
left as-is.
- Falls back to the **tenant default chat model** when `llm_id` is empty
(`task_executor.py:573-574`).
- Strips `` **greedily** (`strings.LastIndex`) in
`cleanExtractionResult`.
- **Auto-tagging** (`extractor_tag.go`): drops the `in.llmID != ""`
guards so an empty `llm_id` no longer skips tagging (uses the tenant
default model), and strips `` greedily in `parseTaggerResponse`.
- **Docs**: fixes a misleading `PresentationChunker` docstring that
claimed per-slide `image`/`position` output (the PPTX path emits none —
unlike PDF), and removes a stale `docs/migration_python_go_diff.md`
reference in `media_dispatch.go`.
## Test plan
- `bash build.sh --test ./internal/common/...` — passes (new retry
utility + tests).
- `bash build.sh --test ./internal/ingestion/component/...` — passes
(extractor/chunker/schema).
- `gofmt` and lefthook pre-commit checks pass.
Note: the personal `docs/migration_python_go_diff.md` working notebook
in the tree is intentionally **not** part of this PR.
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
## Summary
- Return the configured `empty_response` when retrieval has no query or
no chunks.
- Preserve `formalized_content` for downstream Message nodes.
## Testing
- `bash build.sh --go`
- `ok ragflow/internal/agent/tool`
- `ok ragflow/internal/agent/component`
## Summary
- Route ReAct graph step-limit errors through the Agent `_ERROR` output.
- Align the Go Agent default max rounds with Python by using five
rounds.
- Preserve cancellation, timeout, and configuration errors as real
failures.
## Testing
- `bash build.sh --test ./internal/agent/component/...`
<img width="2050" height="1409" alt="image"
src="https://github.com/user-attachments/assets/108e2d27-e115-4595-91a2-21f90c2531ae"
/>
## Summary
- Register `TavilySearch` and `TavilyExtract` Canvas component names in
the Go Agent tool registry.
- Add regression coverage for building both tools by their Python
component names.
## Testing
- `bash build.sh --test ./internal/agent/tool/...`
- `bash build.sh --test ./internal/agent/component/...`
## Summary
Three groups of changes across the Go ingestion pipeline:
### 1. DOCX parsing improvements
- **docx_parser.go**: Enhanced DOCX parsing with better structure
extraction and media handling
- **docx_parser_cgo_test.go**, **docx_parser_test.go**: Companion tests
- **office_parsers_no_cgo.go**: Stub sync for non-CGO builds
### 2. Email (.eml) parsing: base64 Content-Transfer-Encoding decoding
- **email_parser.go** (`decodeCTE`): Added Content-Transfer-Encoding
decoding for base64 and quoted-printable. Go's `mime/multipart.Reader`
does not decode Content-Transfer-Encoding automatically, so attachments
with `Content-Transfer-Encoding: base64` remained base64-encoded in the
output. The new `decodeCTE` helper is called after reading each
multipart part's raw bytes in `readMailBody`, mirroring Python's
`part.get_payload(decode=True)`.
- **email_parser_test.go**: Two new tests — simple base64 attachment and
nested multipart/alternative with base64 attachment.
### 3. Extractor LLM driver fix + ModelDriver consolidation
- **extractor.go**: Fixed a bug where the Extractor component used
`ModelFactory.CreateModelDriver()`, which creates bare model instances
without API keys or provider configuration. Switched to
`models.GetPreconfiguredDriver()` which resolves the actual
pre-configured driver from `ProviderManager`, matching the codepath used
by `llm.go`. This fixes auto keyword/question extraction in DSL
pipelines that require LLM calls.
- **get_driver.go** (new): Extracted shared `GetPreconfiguredDriver()`
from `llm.go:newChatModelDriver()` so both `llm.go` and `extractor.go`
use the same codepath.
- **get_driver_test.go** (new): Tests for the shared driver resolution.
- **llm.go**: Replaced inline driver resolution with
`models.GetPreconfiguredDriver()`.
### 4. Chunker fixes and observability
- **group.go** (`extractLineRecords`): Fixed to also read `markdown` and
`html` payload keys — previously it only read `text`/`content`, causing
GroupTitleChunker to silently return empty results for markdown-format
parser output.
- **common.go** (`compileDelimPattern`): Aligned with Python's
`_compile_delimiter_pattern` — only backtick-wrapped delimiters produce
an active regex pattern; plain delimiters are not compiled into the
split regex.
- **token.go** (`applyChildrenDelim`): Set `DocType` and `CKType` to
`"text"` on created ChunkDocs so the token-size merge path correctly
identifies and merges text segments.
- **parser.go**, **extractor.go**, **tokenizer.go**, **group.go**,
**hierarchy.go**: Added debug-level logging for pipeline diagnostics.
- **parser_dispatch_test.go**, **group_test.go**: New tests.
## Verification
- All Go tests pass: `bash build.sh --test ./internal/parser/parser/...`
and `bash build.sh --test ./internal/ingestion/component/...`
- Build succeeds: `bash build.sh --go`
