Commit Graph

1102 Commits

Author SHA1 Message Date
peewee92
89f3611248 fix(agent): make tag metadata searchable and sortable in agent list (#18313) 2026-08-18 17:17:03 +08:00
Lynn
b2baddfe77 Fix: verification error msg (#18437)
### Summary

As title.
2026-08-18 17:14:52 +08:00
Jack
06463f6a35 fix(deepdoc): preserve cross-page continuation rows on column-count mismatch (#18416) 2026-08-18 16:16:02 +08:00
Jack
5aeba40d99 chore(deepdoc): remove dead standalone DLA client, converge on inference.Client (#18384) 2026-08-18 16:14:37 +08:00
Jack
fb788ee7dc fix(deepdoc): assign each PDF box to a single table cell matching Python (#18410) 2026-08-18 15:13:41 +08:00
mkaaad
94a93748db Feat: connector go webdav (#18379)
as title
2026-08-18 15:01:53 +08:00
Jack
fa386ddf10 fix(deepdoc): drop per-cell OCR in Go table parser to align with Python (#18405) 2026-08-18 13:42:16 +08:00
Jack
97db1b4c38 chore(table): remove stale manual parity-issues test (#18403) 2026-08-18 13:18:57 +08:00
Jack
3a7e275155 fix(deepdoc): add orphan-row cleanup to Go table construction (parity with Python) (#18402) 2026-08-18 13:18:25 +08:00
Jack
d3ff25763c fix(deepdoc): align table orientation scoring with Python recognition confidence (#18401) 2026-08-18 13:18:09 +08:00
Jack
5ec11f7b07 fix(pdf): align TSR crop margin with Python (fixed 30px, not 3%) (#18387) 2026-08-18 13:11:41 +08:00
jay77721
c71991bb7a feat(ingestion,web): modularize extractor configuration with sub-tabs, independent prompts, and metadata integration (#18383)
This PR modularizes the **Extractor** component configuration with dedicated feature subtabs, adds independent system prompt configuration, fixes multi-node execution determinism and parameter persistence across save and page refresh, and ensures backward compatibility with legacy flat fields.
2026-08-18 11:38:52 +08:00
Haruko386
602f55deed feat[syncer]: add connection test with unified connector validation (#18281)
### Summary

as title
2026-08-18 11:30:29 +08:00
Jack
f94438adcc fix(parser): check every grid row, not just the first, before cross-page rebuild (#18378) 2026-08-18 09:45:46 +08:00
buua436
f5efff8d6d fix: exclude compiled artifacts from chunk lists (#18382) 2026-08-18 09:41:27 +08:00
Jin Hai
68f8260821 Go: fix context, part3 (#18390)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-08-18 00:07:37 +08:00
Lem0nTea2002
110ed630bd fix(harness): honor node and engine retry policies in Pregel (#18388)
### Summary

Pregel nodes created with `AddNodeWithOptions` retain a node-level
`RetryPolicy`, but task construction dropped that policy. When a task
had no policy, execution also bypassed the engine-level policy
configured with `WithRetryPolicy` and instantiated the default policy
directly.
2026-08-17 21:27:18 +08:00
Jack
37b945c94d fix(ingestor): stop dropping tasks under burst parse backpressure (#18369)
Fixes the defect where selecting many files (e.g. 20+) in one dataset and starting parsing at once leaves most of them stuck in `RUNNING` forever: a few parse, the rest never do.
2026-08-17 19:09:58 +08:00
Jack
cb012380fe fix(deepdoc): trim garbage-gate comments, fix subtest fact error, dedupe constants (#18377) 2026-08-17 19:08:47 +08:00
questfever
52519e2fd7 refactor: replace Split in loops with more efficient SplitSeq (#18248) 2026-08-17 19:04:04 +08:00
Haruko386
ebb7bd29c6 fix[syncer]: cannot run prune task (#18344)
### Summary

As title
2026-08-17 18:43:44 +08:00
Haruko386
e32723ee36 feat[syncer]: implement outlook data source (#18350)
### Summary

As title
2026-08-17 18:43:15 +08:00
Haruko386
5a5f600ae3 fix: unable to search agent log by ID (#18373)
### Summary

As title
2026-08-17 18:42:45 +08:00
Jack
68da250f8f Fix cross-page merged tables dropping continuation rows (#18362) 2026-08-17 17:58:19 +08:00
Jack
7ceafa07f8 fix(deepdoc): drop low-confidence garbage DLA regions to match Python (#18365)
- The Go DLA client (`inference.Client.DLA`) kept `footer`/`header`/`reference` regions whose detection confidence was below 0.4.
- Python's production path (`parse_into_bboxes` -> `LayoutRecognizer.__call__`, `deepdoc/vision/layout_recognizer.py:97` and `:379`) drops such regions via a 0.4 garbage gate over
`garbage_layouts=['footer','header','reference']`.
- The shared `/predict/dla` backend runs `LayoutRecognizer.forward`, which does **not** apply that gate, so the Go client was the only place the filter was missing — a real Go<->Python parity gap.
2026-08-17 17:50:28 +08:00
Jack
bd01c78d2b fix(pdf): drop text boxes only when replaced by a table HTML (#18363)
## Problem
In the Go PDF pipeline, `processTablesWithReplacements`
(`internal/deepdoc/parser/pdf/table/table_post.go`) marked **every** box
overlapping a DLA table region for removal, then `insertTableBoxes`
re-inserted an HTML box only when `buildTableHTMLs` produced one.

When DLA over-labels a **text** box as a table but TSR produces **no
cells** (so `buildTableHTMLs` skips the table and `htmls[ti]` is unset),
the box was removed with nothing re-inserted — the original text was
**silently dropped**.
2026-08-17 17:07:56 +08:00
buua436
1ef4ddddec fix: align Go ingestion progress and pipeline selection (#18366)
Fixes Go ingestion progress reporting and pipeline selection:
- Add timestamps to document progress logs.
- Keep document duration and status updated during parsing.
- Start frontend polling immediately after parsing begins.
- Prevent documents explicitly using General from inheriting an old
dataset pipeline.
- Populate missing pipeline operation log fields.
- Remove stale component progress logs between retries.
- Prevent progress values greater than `1`.
2026-08-17 16:58:44 +08:00
Wang Qi
57b907cd1e Refactor: do not include compliation chunks in retrieval testing (#18364)
Retrieval Testing: do not include compilation result.
Search: include compilation result.
2026-08-17 16:19:05 +08:00
jay77721
3e38b6cacb fix(ingestion): unify extractor prompt placeholder rendering and per-chunk substitution (#18355) 2026-08-17 15:19:47 +08:00
Jack
0517aebb80 fix(parser): align Go spreadsheet HTML with Python and harden header detection (#18304)
Brings the Go `XLSX`/`XLS`/`CSV` parsers in line with the Python deepdoc spreadsheet HTML contract and improves header detection beyond it.
2026-08-17 15:17:14 +08:00
Jack
8d20cbd0b3 deepdoc(pdf): WarpCrop de-skew + score-based layer-2 rotation (Go OCR parity with Python) (#18299)
Supersedes / folds in #18305. The score-based layer-2 rotation selection from #18305 now lives here, on top of `WarpCrop` (layer 1), applied to **all three** Go OCR paths, together with the Python score plumbing the Go side depends on. #18305 is closed in favor of this PR.
2026-08-17 15:13:56 +08:00
Jack
272645a27a fix(pdf): align DLA region handling with Python reference (#18295)
Aligns three Go DLA / PDF post-processing behaviors with the Python `deepdoc` reference so the Go PDF pipeline matches Python's DLA region / annotation semantics.
2026-08-17 14:33:44 +08:00
jay77721
452720a62d feat(ingestion): end-to-end Auto metadata in Go (extractor merge, builtin UI, built-in fields) (#18273) 2026-08-17 14:17:48 +08:00
euvre
9f358cbeaa fix(ingestion): accumulate component progress lines into document.progress_msg (#18356) 2026-08-17 13:27:40 +08:00
mkaaad
248a4e6960 Feat: connector go discord (#18349)
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-08-17 13:09:07 +08:00
mkaaad
c3b977f7b6 feat(syncer): Add REST API connector (#18256)
as title
2026-08-17 10:50:16 +08:00
S
c6ba54bc72 fix: stop three recurring errors on doc delete (#17685) (#17686)
Follow-up to #17526 ("Refactor: merge dataset scope graph"), which introduced two code paths that touch Infinity columns the deployed schema does not declare. This PR makes the runtime robust against the old schema while also adding the new column to the new schema so freshly created tables are correct.
2026-08-16 09:25:51 +08:00
euvre
00df872e03 fix(search): show honest empty state when mind map has no content (#18294) 2026-08-14 20:22:29 +08:00
Jin Hai
fa5762336b Go: fix context, part2 (#18296)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-08-14 19:42:41 +08:00
euvre
01d351b54a Go: align cancel-parse error with Python for un-started documents (#18245) 2026-08-14 19:22:36 +08:00
Jack
2678a0c5f5 deepdoc: send lossless PNG instead of JPEG to the DeepDoc inference service (#18282) 2026-08-14 18:49:33 +08:00
euvre
3d50f8bcfc fix(go-agent): forward sys.files uploads to vision LLMs (#18246) 2026-08-14 18:26:09 +08:00
Jin Hai
2c775d0549 Go: fix context (#18291)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2026-08-14 17:50:13 +08:00
Haruko386
5bb5ba2216 fix: return total numbers of the search result (#18071)
### Summary

As title
2026-08-14 17:44:38 +08:00
Haruko386
d77b514d3e feat[syncer]: add checkpoint resume run for Github (#18260)
### Summary

As title

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-08-14 17:44:25 +08:00
Haruko386
a2e6ddb56d feat[syncer]: add checkpoint resume run for RSS (#18257)
### Summary

As title

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-08-14 17:44:10 +08:00
Haruko386
cdbd82df4f fix: return total number of docs when sync has no update (#18255)
### Summary

As title

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-08-14 17:43:55 +08:00
euvre
fe963e69ba fix: route TokenChunker delimiter_mode "one" to OneChunker in Go ingestion (#18121) 2026-08-14 16:55:32 +08:00
Jack
ae256bcf59 feat(chunker): add ManualChunker for manual-layout PDFs (#18272)
Add `ManualChunker`, the Go port of Python's `manual` doc-type chunk method (`rag/app/manual.py`). Like `GroupTitleChunker` it merges adjacent text records into heading-bounded groups, but it first re-sorts the records into physical reading order before grouping.
2026-08-14 16:55:08 +08:00
euvre
e1330423ac fix(agent): keep think tags in persisted agent session messages (#18222) 2026-08-14 16:00:08 +08:00