Files
ragflow/internal/parser/parser/pptx_parser.go

145 lines
4.3 KiB
Go
Raw Normal View History

//go:build cgo
//
// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
package parser
import (
"context"
"fmt"
"strings"
officeOxide "github.com/yfedoseev/office_oxide/go"
)
Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111) ## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ```
2026-07-20 19:33:51 +08:00
// PPTXParser parses both .pptx (OOXML) and .ppt (OLE binary)
// files via the office_oxide backend. The format field controls
// the container format passed to OpenFromBytes — "pptx" for
// ZIP-based OOXML presentations and "ppt" for the legacy binary
// OLE format.
type PPTXParser struct {
format string
fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419) ## Summary Continuation of the Python→Go ingestion pipeline migration (File → Parser → Chunker → Extractor → Tokenizer). Fixes cover Parser, Chunker, and Tokenizer gaps identified. Fix page number (0-indexed and 1-index mixed before fix; use 1-indexed after fix) and chunk order issues. ### Parser - **Slides TCADP (1.7):** `pptx_tcadp.go` + TCADP branch in `pptx_parser.go`/`ppt_parser.go` — PowerPoint files now support `parse_method="tcadp"` via the TCADP cloud service, matching the spreadsheet-family TCADP pattern. PPT containers pass `"PPT"` as fileType (not hardcoded `"PPTX"`). - **Audio default output_format (2.11):** `defaultSetups()` audio default changed from `"text"` to `"json"`, aligning with Python `parser.py:232` and `AllowedOutputFormat["audio"]={"json"}`. - **PDF VLM enhancement (1.1):** `maybeDispatchPDFVisionEnhancement` in `pdf_vision_dispatch.go` enriches image/table items with IMAGE2TEXT model descriptions after PDF parsing, mirroring Python `enhance_media_sections_with_vision`. Semaphore fix: acquire before goroutine start to prevent unbounded goroutine creation. - **json family (2.3):** reclassified as Keep Go — `json_parser.go` is a functional enhancement, not a parity gap. - **page number:** changed from "mixed use of 1-indexed & 0-indexed" to "1-indexed" ### Chunker - **BULLET_PATTERN fallback (1.7):** 4th-level fallback in `resolveTitleLevels` (`title.go`) detects bullet/numbered-list patterns (Chinese legal, numbering, English) when outline + regex levels produce only bodyLevel. Guarded by `allBodyLevel` to never override existing structure. - **Tag/One chunker fields (1.8):** `tag.go` sets `TopInt` from source row index; `one.go` preserves `Positions`/`PDFPositions` from source items. TSV multi-line RowNum fix: tracks `contentStart` for correct row attribution. - **Overlapped_percent normalization (2.6):** `NormalizeOverlappedPercent` in `schema/chunker.go` mirrors Python `common/float_utils.py:50-58` — accepts `[0,1)` fraction or `[0,90]` percent, normalizes to canonical `[0,90]`. - **Paragraph splitting (2.7):** aligned to Python flow `naive_merge` — `CRLF` normalization, `splitKeepingDelimiter` preserves sentence delimiters, single-section merge with token-budget-governed chunking. - **chunk order:** sort by reading order ### Tokenizer - **Phantom chunk filtering (Omission 2):** `isPhantomChunk` + filter loop in `chunksFromTokenizerUpstream` skips zero-value ChunkDocs. - **Batch size env var (Omission 3):** `embeddingBatchSize()` reads `TOKENIZER_EMBEDDING_BATCH_SIZE`, defaults to 16. - **Summary empty check (Diff 5):** `TrimSpace(s) != ""` → `s != ""`, matching Python truthy check. - **chunk_order_int all paths (Diff 8):** set unconditionally before full_text/embedding branching. - **Timeout default (Diff 10):** `600s` → `60s`, matching Python `@timeout(60)`. - **Small maxTokens truncation (Diff 14):** `truncateForEmbedding` returns `""` when `maxTokens <= 10`, matching Python. ### Code review fixes - Semaphore acquire moved before goroutine in `pdf_vision_dispatch.go` (concurrency control) - Context propagation in `pptx_tcadp.go` (cancellation support) - Test resolver leak fix in `media_dispatch_test.go` (defer restore) - Migration history comments removed per AGENTS.md ## Test plan ``` bash build.sh --test ./internal/parser/parser/... ./internal/ingestion/component/... ``` ## Notes - Migration diff tracking: `docs/migration_python_go_diff.md` - Remaining gaps: Extractor component only (21 items)
2026-07-28 11:12:52 +08:00
// TCADP cloud-parsing configuration
ParseMethod string
TCADPAPIServer string
TCADPAPIKey string
TCADPTableResultType string
TCADPMarkdownImageResponseType string
OutputFormat string
Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111) ## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ```
2026-07-20 19:33:51 +08:00
}
feat(agent): Go ingestion pipeline progress mirroring and DeepDOC parser hardening (#16795) feat(ingestion): mirror Go pipeline progress into the document table; harden resume guards - pipeline: bind the owning document via WithDocumentID; after each TrackProgress event aggregate ingestion_task_log progress and mirror progress/run/progress_msg back into the document table, so GET /api/v1/datasets/{dataset_id}/documents reflects live Go pipeline progress without a bespoke endpoint. - canvas: extend the S3 resume guard to reject legacy no-op nodes (e.g. ExitLoop) so component_total equals the count of progress-reporting components and the aggregate percent can reach 100%. - runtime/canvas: route progress through TrackProgress; add interrupt test coverage (r3_interrupt_test.go). - dao/entity: add IngestionTask.DocumentID column and AggregateProgress support used by the mirror; IngestionTaskLog keeps a Checkpoint column alongside the progress fields. feat(deepdoc): cache DocAnalyzer inference results in Redis (1h TTL) - Redis-backed DocAnalyzerCache decorator over inference.Client; cache key = "ddoc:cache:<method>:" + sha256 of the JPEG-encoded image bytes (deterministic). - TTL = 1h; hits skip the inner HTTP call and return cached JSON; inner errors are not cached. refactor(deepdoc): align figure cropping with Python cropout + bounded page caches - CropSectionByDLA mirrors Python cropout: best-overlap DLA figure/equation region, fallback to section bbox per page, vertical concat on gray background. - sliding-window page-image cache bounds peak memory to the recent window instead of the whole PDF. - rename DLADebug -> DLARegions across parser/chunker/tests. refactor(parser): drop lib_type selector; align NewXxxParser with NewPDFParser - remove config["lib_type"] lookup and the libType param/field/switch from all nine constructors; surface the CGO-required error at ParseWithResult time instead of construction time; drop resolveLibType, its test, and the four lib_type constants. feat(utility): add a reusable workerpool for bounded concurrent execution - internal/utility/workerpool.go (+ tests). refactor: translate Chinese prose comments to English in non-harness Go files. chore: upgrade github.com/cloudwego/eino from v0.9.9 to v0.9.12.
2026-07-10 10:36:10 +08:00
func NewPPTXParser() *PPTXParser {
Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111) ## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ```
2026-07-20 19:33:51 +08:00
return &PPTXParser{format: "pptx"}
}
func (p *PPTXParser) String() string {
return "PPTXParser"
}
fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419) ## Summary Continuation of the Python→Go ingestion pipeline migration (File → Parser → Chunker → Extractor → Tokenizer). Fixes cover Parser, Chunker, and Tokenizer gaps identified. Fix page number (0-indexed and 1-index mixed before fix; use 1-indexed after fix) and chunk order issues. ### Parser - **Slides TCADP (1.7):** `pptx_tcadp.go` + TCADP branch in `pptx_parser.go`/`ppt_parser.go` — PowerPoint files now support `parse_method="tcadp"` via the TCADP cloud service, matching the spreadsheet-family TCADP pattern. PPT containers pass `"PPT"` as fileType (not hardcoded `"PPTX"`). - **Audio default output_format (2.11):** `defaultSetups()` audio default changed from `"text"` to `"json"`, aligning with Python `parser.py:232` and `AllowedOutputFormat["audio"]={"json"}`. - **PDF VLM enhancement (1.1):** `maybeDispatchPDFVisionEnhancement` in `pdf_vision_dispatch.go` enriches image/table items with IMAGE2TEXT model descriptions after PDF parsing, mirroring Python `enhance_media_sections_with_vision`. Semaphore fix: acquire before goroutine start to prevent unbounded goroutine creation. - **json family (2.3):** reclassified as Keep Go — `json_parser.go` is a functional enhancement, not a parity gap. - **page number:** changed from "mixed use of 1-indexed & 0-indexed" to "1-indexed" ### Chunker - **BULLET_PATTERN fallback (1.7):** 4th-level fallback in `resolveTitleLevels` (`title.go`) detects bullet/numbered-list patterns (Chinese legal, numbering, English) when outline + regex levels produce only bodyLevel. Guarded by `allBodyLevel` to never override existing structure. - **Tag/One chunker fields (1.8):** `tag.go` sets `TopInt` from source row index; `one.go` preserves `Positions`/`PDFPositions` from source items. TSV multi-line RowNum fix: tracks `contentStart` for correct row attribution. - **Overlapped_percent normalization (2.6):** `NormalizeOverlappedPercent` in `schema/chunker.go` mirrors Python `common/float_utils.py:50-58` — accepts `[0,1)` fraction or `[0,90]` percent, normalizes to canonical `[0,90]`. - **Paragraph splitting (2.7):** aligned to Python flow `naive_merge` — `CRLF` normalization, `splitKeepingDelimiter` preserves sentence delimiters, single-section merge with token-budget-governed chunking. - **chunk order:** sort by reading order ### Tokenizer - **Phantom chunk filtering (Omission 2):** `isPhantomChunk` + filter loop in `chunksFromTokenizerUpstream` skips zero-value ChunkDocs. - **Batch size env var (Omission 3):** `embeddingBatchSize()` reads `TOKENIZER_EMBEDDING_BATCH_SIZE`, defaults to 16. - **Summary empty check (Diff 5):** `TrimSpace(s) != ""` → `s != ""`, matching Python truthy check. - **chunk_order_int all paths (Diff 8):** set unconditionally before full_text/embedding branching. - **Timeout default (Diff 10):** `600s` → `60s`, matching Python `@timeout(60)`. - **Small maxTokens truncation (Diff 14):** `truncateForEmbedding` returns `""` when `maxTokens <= 10`, matching Python. ### Code review fixes - Semaphore acquire moved before goroutine in `pdf_vision_dispatch.go` (concurrency control) - Context propagation in `pptx_tcadp.go` (cancellation support) - Test resolver leak fix in `media_dispatch_test.go` (defer restore) - Migration history comments removed per AGENTS.md ## Test plan ``` bash build.sh --test ./internal/parser/parser/... ./internal/ingestion/component/... ``` ## Notes - Migration diff tracking: `docs/migration_python_go_diff.md` - Remaining gaps: Extractor component only (21 items)
2026-07-28 11:12:52 +08:00
// ConfigureFromSetup reads the slides-family setup map. Mirrors the
// XLSXParser ConfigureFromSetup pattern
func (p *PPTXParser) ConfigureFromSetup(setup map[string]any) {
if p == nil || setup == nil {
return
}
if v, ok := setup["parse_method"].(string); ok {
p.ParseMethod = v
}
if v, ok := setup["tcadp_apiserver"].(string); ok {
p.TCADPAPIServer = v
}
if v, ok := setup["tcadp_api_key"].(string); ok {
p.TCADPAPIKey = v
}
if v, ok := setup["table_result_type"].(string); ok {
p.TCADPTableResultType = v
}
if v, ok := setup["markdown_image_response_type"].(string); ok {
p.TCADPMarkdownImageResponseType = v
}
if v, ok := setup["output_format"].(string); ok {
p.OutputFormat = v
}
if p.OutputFormat == "" {
p.OutputFormat = "json"
}
}
// ParseWithResult emits one JSON item per slide with the slide's
// plain text. Mirrors the python parser.py:slides branch which
// forces output_format="json" for the slide family.
func (p *PPTXParser) ParseWithResult(ctx context.Context, filename string, data []byte) ParseResult {
fix: Go ingestion migration batch 5 (Parser 1.1/1.7/2.11, Chunker 1.7/1.8/2.6/2.7, Tokenizer 6x fixes) (#17419) ## Summary Continuation of the Python→Go ingestion pipeline migration (File → Parser → Chunker → Extractor → Tokenizer). Fixes cover Parser, Chunker, and Tokenizer gaps identified. Fix page number (0-indexed and 1-index mixed before fix; use 1-indexed after fix) and chunk order issues. ### Parser - **Slides TCADP (1.7):** `pptx_tcadp.go` + TCADP branch in `pptx_parser.go`/`ppt_parser.go` — PowerPoint files now support `parse_method="tcadp"` via the TCADP cloud service, matching the spreadsheet-family TCADP pattern. PPT containers pass `"PPT"` as fileType (not hardcoded `"PPTX"`). - **Audio default output_format (2.11):** `defaultSetups()` audio default changed from `"text"` to `"json"`, aligning with Python `parser.py:232` and `AllowedOutputFormat["audio"]={"json"}`. - **PDF VLM enhancement (1.1):** `maybeDispatchPDFVisionEnhancement` in `pdf_vision_dispatch.go` enriches image/table items with IMAGE2TEXT model descriptions after PDF parsing, mirroring Python `enhance_media_sections_with_vision`. Semaphore fix: acquire before goroutine start to prevent unbounded goroutine creation. - **json family (2.3):** reclassified as Keep Go — `json_parser.go` is a functional enhancement, not a parity gap. - **page number:** changed from "mixed use of 1-indexed & 0-indexed" to "1-indexed" ### Chunker - **BULLET_PATTERN fallback (1.7):** 4th-level fallback in `resolveTitleLevels` (`title.go`) detects bullet/numbered-list patterns (Chinese legal, numbering, English) when outline + regex levels produce only bodyLevel. Guarded by `allBodyLevel` to never override existing structure. - **Tag/One chunker fields (1.8):** `tag.go` sets `TopInt` from source row index; `one.go` preserves `Positions`/`PDFPositions` from source items. TSV multi-line RowNum fix: tracks `contentStart` for correct row attribution. - **Overlapped_percent normalization (2.6):** `NormalizeOverlappedPercent` in `schema/chunker.go` mirrors Python `common/float_utils.py:50-58` — accepts `[0,1)` fraction or `[0,90]` percent, normalizes to canonical `[0,90]`. - **Paragraph splitting (2.7):** aligned to Python flow `naive_merge` — `CRLF` normalization, `splitKeepingDelimiter` preserves sentence delimiters, single-section merge with token-budget-governed chunking. - **chunk order:** sort by reading order ### Tokenizer - **Phantom chunk filtering (Omission 2):** `isPhantomChunk` + filter loop in `chunksFromTokenizerUpstream` skips zero-value ChunkDocs. - **Batch size env var (Omission 3):** `embeddingBatchSize()` reads `TOKENIZER_EMBEDDING_BATCH_SIZE`, defaults to 16. - **Summary empty check (Diff 5):** `TrimSpace(s) != ""` → `s != ""`, matching Python truthy check. - **chunk_order_int all paths (Diff 8):** set unconditionally before full_text/embedding branching. - **Timeout default (Diff 10):** `600s` → `60s`, matching Python `@timeout(60)`. - **Small maxTokens truncation (Diff 14):** `truncateForEmbedding` returns `""` when `maxTokens <= 10`, matching Python. ### Code review fixes - Semaphore acquire moved before goroutine in `pdf_vision_dispatch.go` (concurrency control) - Context propagation in `pptx_tcadp.go` (cancellation support) - Test resolver leak fix in `media_dispatch_test.go` (defer restore) - Migration history comments removed per AGENTS.md ## Test plan ``` bash build.sh --test ./internal/parser/parser/... ./internal/ingestion/component/... ``` ## Notes - Migration diff tracking: `docs/migration_python_go_diff.md` - Remaining gaps: Extractor component only (21 items)
2026-07-28 11:12:52 +08:00
// p == nil guard: the struct is embedded by value in PPTParser and
// always created via NewPPTXParser or the "ppt"-format constructor in
// PPTParser, so this branch is unreachable from normal call paths.
// Kept as defensive guard — a nil dereference here would obscure the
// root cause behind a nil-pointer panic.
if p == nil {
return ParseResult{Err: fmt.Errorf("PPTXParser is nil")}
}
method := strings.ToLower(strings.TrimSpace(p.ParseMethod))
switch method {
case "tcadp":
return parsePresentationWithTCADP(ctx,
filename, data, strings.ToUpper(p.format),
p.TCADPAPIServer, p.TCADPAPIKey,
p.TCADPTableResultType, p.TCADPMarkdownImageResponseType,
p.OutputFormat,
)
case "", "deepdoc":
// Continue with the local office_oxide parser.
default:
// PDF-specific methods like "paddleocr" / "mineru" are
// meaningless for PPTX; treat as default path.
}
Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111) ## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ```
2026-07-20 19:33:51 +08:00
doc, err := officeOxide.OpenFromBytes(data, p.format)
if err != nil {
Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111) ## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ```
2026-07-20 19:33:51 +08:00
return ParseResult{Err: fmt.Errorf("presentation open: %w", err)}
}
defer doc.Close()
text, err := doc.PlainText()
if err != nil {
return ParseResult{Err: fmt.Errorf("pptx plain-text: %w", err)}
}
// Split on form-feed (the python TxtParser convention used by
Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111) ## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ```
2026-07-20 19:33:51 +08:00
// RAGFlow's slide parser) — each block becomes a JSON item.
var items []map[string]any
for i, raw := range strings.Split(text, "\f") {
trimmed := strings.TrimSpace(raw)
if trimmed == "" {
continue
}
items = append(items, map[string]any{
"text": trimmed,
"doc_type_kwd": "text",
"slide_number": i + 1,
})
}
if items == nil {
items = []map[string]any{{"text": strings.TrimSpace(text), "doc_type_kwd": "text"}}
}
return ParseResult{
OutputFormat: "json",
Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111) ## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ```
2026-07-20 19:33:51 +08:00
File: map[string]any{"name": filename, "format": p.format},
JSON: items,
}
}