zlei6/openserp
1
0
Fork 0
mirror of https://github.com/karust/openserp.git synced 2026-08-12 20:03:29 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix: /extract/batch per-URL errors, param parity, proxy headers.

Browse Source 
Update readme
This commit is contained in:
Rustem Kamalov
2026-07-19 04:18:50 +03:00
parent 01223e35d3
commit 2a57bd4248
6 changed files with 367 additions and 358 deletions
Download Patch File Download Diff File Expand all files Collapse all files

248
README.md
View File

@@ -2,30 +2,28 @@
# OpenSERP
[![Go Reference](https://pkg.go.dev/badge/github/karust/openserp?style=for-the-badge)](https://pkg.go.dev/github.com/karust/openserp)
[![Go Reference](https://pkg.go.dev/badge/github.com/karust/openserp.svg)](https://pkg.go.dev/github.com/karust/openserp)
[![release](https://img.shields.io/github/v/release/karust/openserp)](https://github.com/karust/openserp/releases)
[![Docker Pulls](https://img.shields.io/docker/v/karust/openserp)](https://hub.docker.com/r/karust/openserp)
[![CI](https://github.com/karust/openserp/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/karust/openserp/actions/workflows/ci.yml)
[![Telegram](https://img.shields.io/badge/Telegram-openserp__cloud-26A5E4?logo=telegram&logoColor=white)](https://t.me/openserp_cloud)
**OpenSERP** is a free, open-source SERP API and CLI for live search data from **Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia**.
**OpenSERP** is a free, open-source SERP API and CLI for Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia.
Use it as a search tool for **LLMs, agents, and RAG pipelines**, or as a scraper backend for **SEO rank tracking across Google, Yandex, Baidu, and more**. It is especially useful when your workflow needs RU/CN web coverage instead of another Google-only API.
No API keys, no per-search billing: one command gives you live, structured search results on localhost - including engines the paid APIs don't cover. Use it as a search tool for LLMs and agents, or as a backend for SEO rank tracking. If you'd rather not run infrastructure, there is a [hosted version](https://openserp.org/cloud) with the same API.
Run it locally, self-host it, or use the [cloud version](https://openserp.org/cloud) when you want the same public API shape without operating the server. Cloud is also the simplest way to support OpenSERP: it is where the technology gets tested at scale, and that work flows straight back into this open-source project.
![OpenSERP CLI demo](./docs/demo.gif)
## Features
- 🔍 **Multi-engine** - dedicated endpoints for Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia, with stable JSON for SEO rank pipelines
- 🌐 **Megasearch** - `/mega/search` runs one query across every selected engine, then merges and dedupes results
- 📄 **URL extraction** - return search results plus clean markdown/text target-page content in one call, for grounding and automation
- ✨ **SERP features** - AI summaries, answer boxes, people-also-ask, and related searches in a response
- 🖼 **Images** - image search is also available
- 🎯 **Advanced filters** - language, date range, file type, and site queries
- 📝 **Data formats** - JSON, Markdown, Text, NdJSON response formats
- 🌍 **Configurable** - proxy, cache, and resilient mode
- 🐳 **Docker-ready** - local and container deployment
- Dedicated endpoints for six engines, same JSON schema across all of them
- **Megasearch** - one query across several engines at once, merged and deduped
- **URL extraction** - search results plus clean markdown of the target pages in one call
- SERP features: AI summaries, answer boxes, people-also-ask, related searches
- Image search, filters (language, date range, file type, site), **JSON/Markdown/Text/NdJSON** output
- Proxies, cache, resilient mode, prebuilt Docker images
## ⚡ Quick Start
## Quick Start
### Docker
@@ -58,6 +56,7 @@ go build -o openserp .
### First request
```sh
# mode=any returns the first engine that responds
curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust&extract=1&mode=any"
```
@@ -86,7 +85,7 @@ curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust&
"title": "The Go Programming Language",
"url": "https://go.dev/",
"display_url": "go.dev",
"snippet": "Get Started Playground Tour Stack Overflow Help Packages Standard Library About Go Packages About Download Blog Issue Tracker Release Notes Brand Guidelines Code of Conduct Connect …",
"snippet": "Get Started Playground Tour Stack Overflow Help Packages Standard Library …",
"domain": "go.dev",
"favicon": "https://go.dev/favicon.ico",
"position": {
@@ -101,7 +100,7 @@ curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust&
"extracted": {
"title": "Build simple, secure, scalable systems with Go",
"format": "markdown",
"content": "## Build simple, secure, scalable systems with Go\n\n![Go Gopher climbing a ladder.](https://go.dev/images/gophers/ladder.svg)\n\n- “At the time, no single team member knew Go, but **within a month, everyone was writing in Go** and we were building out the endpoints. It was the flexibility, how easy it was to use, and the really cool concept behind Go (how Go handles native concurrency, garbage collection, and of course safety+speed.) that helped engage us during the build. Also, who can beat that cute mascot!”\n ........",
"content": "## Build simple, secure, scalable systems with Go\n\n![Go Gopher climbing a ladder.](https://go.dev/images/gophers/ladder.svg)\n\n- “At the time, no single team member knew Go, but **within a month, everyone was writing in Go** and we were building out the endpoints. ........",
"mode_used": "fast",
"fetched_at": "2026-06-16T00:06:56Z"
}
@@ -113,7 +112,7 @@ curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust&
"title": "Go (programming language) - Wikipedia",
"url": "https://en.wikipedia.org/wiki/Go_(programming_language)",
"display_url": "en.wikipedia.org › wiki › Go_(programming_language)",
"snippet": "In Go's package system, each package has a path (e.g., \"compress/bzip2\" or \"golang.org/x/net/html\") and a name (e.g., bzip2 or html). By default other packages' definitions must always be prefixed with …",
"snippet": "In Go's package system, each package has a path (e.g., \"compress/bzip2\" or \"golang.org/x/net/html\") and a name (e.g., bzip2 or html). …",
"domain": "en.wikipedia.org",
"favicon": "https://en.wikipedia.org/favicon.ico",
"position": {
@@ -162,24 +161,6 @@ curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust&
</details>
## Deployment Options
- **Self-hosted (this repo)** - free, MIT-licensed, with full control over runtime, proxies, cache, and scaling.
- **[OpenSERP Cloud](https://openserp.org/cloud)** - the same public API, run and maintained for you by the people who build OpenSERP.
Same endpoints, same response schema, and client code migrates either direction, so you are never locked in.
If OpenSERP is useful to you and you would rather not run the infrastructure yourself, using Cloud (or topping up an account) is a direct way to support the project. It is where the technology gets exercised against real traffic at scale, which is how the engines get hardened, new features get tested, and the open-source core keeps improving. Every request you send through Cloud helps fund and shape what lands back in this repository.
## API Docs
Once the server is running, the interactive docs are available locally:
- Swagger UI: `http://127.0.0.1:7000/docs`
- OpenAPI YAML: `http://127.0.0.1:7000/openapi.yaml`
To browse the spec without running the server, see [docs/openapi.yaml](./docs/openapi.yaml). For a higher-level overview of how OpenSERP works internally, see the [architecture docs](https://openserp.org/docs/architecture/).
## SDKs & Examples
Official client packages. Each works against your self-hosted server (set `baseUrl`) or the [hosted API](https://openserp.org/cloud) (set `apiKey`):
@@ -223,6 +204,8 @@ Megasearch:
curl "http://127.0.0.1:7000/mega/search?text=golang&limit=10"
```
`/mega/search` returns the same envelope as engine endpoints plus `clusters`: results are deduplicated by normalized URL, and clusters keep the per-engine occurrences and ranks.
| Mode | Best for | Behavior |
| ---------- | ------------------------------------ | ---------------------------------------------- |
| `balanced` | Most multi-engine SERP workflows | Queries engines in parallel and merges results |
@@ -266,43 +249,57 @@ curl "http://127.0.0.1:7000/extract?url=https://example.com&mode=auto"
# Return clean page markdown
curl "http://127.0.0.1:7000/extract?url=https://example.com&format=markdown"
# Extract several URLs at once - returns a bare [{page_content, metadata}] array
# (Open WebUI external loader compatible); failed URLs become items with metadata.error
curl -X POST "http://127.0.0.1:7000/extract/batch" \
-H "Content-Type: application/json" \
-d '{"urls":["https://example.com","https://go.dev"],"mode":"fast"}'
# Embed extracted content under the top search results
curl "http://127.0.0.1:7000/google/search?text=llm+observability&extract=2&format=markdown"
```
## 🖥 CLI Search
## CLI Search
No server required - query an engine straight from the terminal. The CLI shares the same engines, formats, and filters as the API.
```sh
openserp search duckduckgo "free open source serp" --format markdown
openserp search ecosia "weather in london" --format markdown
```
<details>
<summary>CLI output and more examples</summary>
```markdown
# Search results for "free open source serp"
# Search results for "weather in london"
**Query:** free open source serp - **Engines:** duckduckgo - **Took:** 1794ms
**Query:** weather in london - **Engines:** ecosia - **Took:** 866ms
## Results
### 1. OpenSERP: Open-Source, Self-Hosted & Free SERP API
### 1. London - BBC Weather
**openserp.org** - organic
**bbc.com › weather › 2643743** - organic
OpenSERP is a free, open-source and self-hosted SERP API for Google, Bing, Yandex, Baidu, DuckDuckGo and Ecosia, with an optional managed Cloud path.
Latest forecast for London ... Tonight will continue dry, and there will be mainly clear skies. Just a few patches of cloud drifting in from the north at times.
-> https://openserp.org/
-> https://www.bbc.com/weather/2643743
### 2. GitHub - karust/openserp: Open-source SERP API for AI, SEO & automation ...
### 2. London (Greater London) weather - Met Office
**github.com › karust › openserp** - organic
**weather.metoffice.gov.uk › forecast › gcpvj0v07** - organic
OpenSERP is a free, open-source API and CLI for accessing normalized search engine results from Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia. Run it locally, self-host it, or use the optional hosted API when you do not want to manage infrastructure.
Remaining warm with light winds and dry. Possibly cloudy at times Monday and Tuesday, then Wednesday sunnier conditions are likely.
-> https://github.com/karust/openserp
-> https://weather.metoffice.gov.uk/forecast/gcpvj0v07
### 3. London, London, United Kingdom Weather Forecast
**accuweather.com › en › gb › london › ec4a-2 › wea…** - organic
London, London, United Kingdom Weather Forecast, with current conditions, wind, air quality, and what to expect for the next 3 days.
-> https://www.accuweather.com/en/gb/london/ec4a-2/weather-forecast/328328
```
More CLI examples:
@@ -321,14 +318,15 @@ openserp search bing "release notes" --site github.com --format ndjson
openserp search google "llm observability" --extract 2 --format markdown
# Browserless (raw HTTP) mode through a proxy
openserp search duckduckgo "free open source serp" --raw --proxy http://user:pass@127.0.0.1:8080
# (raw mode: google, yandex, baidu, ecosia)
openserp search ecosia "weather in london" --raw --proxy http://user:pass@127.0.0.1:8080
```
</details>
Run `openserp search --help` for the full flag list. Engine names: `google`, `yandex`, `baidu`, `bing`, `duckduckgo`, `ecosia`.
## 🔍 Query Parameters
## Query Parameters
Common parameters:
@@ -353,132 +351,7 @@ Engine-specific parameters:
| `filter` | `google` | Duplicate filter: `true` hides similar results, `false` includes them. |
| `features` | browser `Search` | Populate `serp_features[]` from the live page. Defaults to `true`. |
## Search Response Example
<details>
<summary>Search response example</summary>
```json
{
"query": {
"text": "golang",
"engines_requested": ["google"]
},
"meta": {
"request_id": "019dc6c1-da45-706e-a57c-d671fa2862ee",
"requested_at": "2026-04-25T22:27:52Z",
"took_ms": 6410,
"engines_failed": [],
"version": "2.1"
},
"results": [
{
"id": "s_78341aa47c336101",
"rank": 1,
"type": "organic",
"title": "Documentation - The Go Programming Language",
"url": "https://go.dev/doc/",
"display_url": "go.dev > doc",
"snippet": "Official Go documentation, tutorials, references, and release notes.",
"domain": "go.dev",
"favicon": "https://go.dev/favicon.ico",
"position": {
"absolute": 1
},
"engine": "google",
"domain_info": {
"tld": "dev",
"sld": "go",
"category": ""
}
}
],
"pagination": {
"page": 1,
"has_more": true,
"next_start": 25
}
}
```
</details>
## Mega Response Notes
`/mega/search` returns the same envelope plus `clusters`. Results are deduplicated by normalized URL; clusters keep the per-engine occurrences.
<details>
<summary>Cluster example</summary>
```json
{
"id": "c_a1b2c3d4e5f6a1b2",
"canonical_url": "https://go.dev/",
"domain": "go.dev",
"title": "The Go Programming Language",
"occurrences": [
{ "engine": "google", "rank": 1, "result_id": "s_78341aa47c336101" },
{ "engine": "bing", "rank": 2, "result_id": "s_20f9f15f0c3d9f6d" }
],
"engines_count": 2,
"best_rank": 1,
"score": 0.75
}
```
</details>
## Image Response Example
<details>
<summary>Image result example</summary>
```json
{
"id": "i_a1b2c3d4e5f6a1b2",
"rank": 1,
"type": "image",
"title": "Go Gopher Logo",
"image": {
"url": "https://example.com/images/go-logo.png",
"thumbnail": "https://example.com/images/go-logo-thumb.png",
"width": 1200,
"height": 800
},
"source": {
"page_url": "https://go.dev/brand/",
"domain": "go.dev"
},
"engine": "bing"
}
```
</details>
## Error Responses
`400 Bad Request`:
```json
{
"error": "bad_request",
"code": 400,
"message": "EMPTY_QUERY: query cannot be empty: provide text, site, or file parameter",
"reason": "EMPTY_QUERY"
}
```
`503 Service Unavailable`:
```json
{
"error": "service_unavailable",
"code": 503,
"message": "captcha found, please stop sending requests for a while: captcha detected"
}
```
## 🌍 Proxy Support
## Proxy Support
OpenSERP supports HTTP and SOCKS5 proxies.
@@ -491,18 +364,21 @@ Simple global proxy:
Advanced proxy configuration is available in [config.yaml](./config.yaml). You can enable tagged proxy pools and per-request override via `X-Use-Proxy: <tag>` or `X-Use-Proxy: direct`.
A [managed API](https://openserp.org/cloud) is also available for teams that do not want to operate infrastructure.
## API Docs
## Health & Stats
Once the server is running, the interactive docs are available locally:
```bash
curl -i "http://127.0.0.1:7000/health"
curl "http://127.0.0.1:7000/ready"
curl "http://127.0.0.1:7000/stats"
curl "http://127.0.0.1:7000/stats/cache"
curl "http://127.0.0.1:7000/stats/proxy"
curl "http://127.0.0.1:7000/stats/cb"
```
- Swagger UI: `http://127.0.0.1:7000/docs` - full schemas, error shapes, and the `/health`, `/ready`, `/stats/*` endpoints
- OpenAPI YAML: `http://127.0.0.1:7000/openapi.yaml`
To browse the spec without running the server, see [docs/openapi.yaml](./docs/openapi.yaml). For a higher-level overview of how OpenSERP works internally, see the [architecture docs](https://openserp.org/docs/architecture/).
## Self-Hosted or Cloud
- **Self-hosted (this repo)** - free, MIT-licensed, full control over runtime, proxies, cache, and scaling.
- **[OpenSERP Cloud](https://openserp.org/cloud)** - same endpoints and response schema, no infrastructure to run.
Client code migrates in either direction, so you are never locked in.
## License
@@ -516,6 +392,6 @@ Contributions are welcome. See [docs/CONTRIBUTING.md](./docs/CONTRIBUTING.md).
- [GitHub Issues](https://github.com/karust/openserp/issues) - bugs, feature ideas, and reproducible issues.
- [feedback@openserp.org](mailto:feedback@openserp.org) - private notes, longer feedback, or anything that does not fit GitHub Issues.
- [Telegram Channel](https://t.me/+RJEKspw3mUlhZDMy) - OpenSERP news, release notes, and project updates.
- [Telegram](https://t.me/openserp_cloud) - OpenSERP news, release notes, and project updates.
> OpenSERP is free and open-source. Only links listed in this repository and on [openserp.org](https://openserp.org) are associated with the project.

2
cmd/root.go
View File

@@ -17,7 +17,7 @@ import (
)
const (
version = "0.8.10"
version = "0.8.11"
defaultConfigFilename = "config"
envPrefix = "OPENSERP"
)

183
core/server_extract.go
View File

@@ -7,6 +7,7 @@ import (
"fmt"
"io"
"net/http"
"strconv"
"strings"
"sync"
"time"
@@ -20,9 +21,10 @@ type extractPayload struct {
Mode string `json:"mode"`
// Clean defaults to true (article-only). Pointer so we can tell "omitted"
// (use default) from an explicit false (full-page extraction).
Clean *bool `json:"clean"`
UseLLMSTxt bool `json:"use_llms_txt"`
MinRunes int `json:"min_runes"`
Clean *bool `json:"clean"`
UseLLMSTxt bool `json:"use_llms_txt"`
MinRunes int `json:"min_runes"`
Lang string `json:"lang"`
}
func (s *Server) handleExtract(c *fiber.Ctx) error {
@@ -57,11 +59,10 @@ func (s *Server) handleExtract(c *fiber.Ctx) error {
return sendExtractResult(c, format, result)
}
func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (extractpkg.ExtractRequest, error) {
var body extractPayload
if len(c.Body()) > 0 {
_ = c.BodyParser(&body)
}
// baseExtractRequest builds the URL-independent part of an extract request
// from proxy headers, query params, and the parsed body. Shared by /extract
// and /extract/batch so both accept the same knobs.
func (s *Server) baseExtractRequest(c *fiber.Ctx, body extractPayload, cfg extractpkg.Config) (extractpkg.ExtractRequest, error) {
proxyOverride, err := NormalizeProxyRequestOverride(c.Get("X-Use-Proxy"))
if err != nil {
return extractpkg.ExtractRequest{}, errInvalidParam(fmt.Sprintf("X-Use-Proxy: %v", err))
@@ -78,7 +79,12 @@ func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (e
if err := s.validateRequestProxyURL(&q); err != nil {
return extractpkg.ExtractRequest{}, err
}
mode := firstNonEmpty(c.Query("mode"), body.Mode, cfg.DefaultMode)
mode := extractpkg.Mode(strings.ToLower(firstNonEmpty(c.Query("mode"), body.Mode, cfg.DefaultMode)))
switch mode {
case extractpkg.ModeAuto, extractpkg.ModeFast, extractpkg.ModeRendered:
default:
return extractpkg.ExtractRequest{}, errInvalidParam("mode must be one of auto, fast, rendered")
}
// Default clean=true (article-only). FullPage is the inverse: full-readable-body
// extraction, opted in via clean=false on the query string or body.
bodyClean := true
@@ -90,15 +96,10 @@ func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (e
if err != nil {
return extractpkg.ExtractRequest{}, errInvalidParam("min_runes must be a non-negative integer")
}
targetURL := extractpkg.NormalizeURL(strings.TrimSpace(firstNonEmpty(c.Query("url"), body.URL)))
if err := validateExtractTargetURL(c.UserContext(), targetURL, cfg.AllowPrivateNetworks); err != nil {
return extractpkg.ExtractRequest{}, errInvalidParam(err.Error())
}
return extractpkg.ExtractRequest{
URL: targetURL,
Mode: extractpkg.Mode(mode),
Mode: mode,
ProxyURL: proxyURL,
LangCode: strings.TrimSpace(c.Query("lang")),
LangCode: firstNonEmpty(body.Lang, c.Query("lang")),
Timeout: cfg.Timeout,
MaxBytes: cfg.MaxBytes,
FullPage: !clean,
@@ -107,6 +108,22 @@ func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (e
}, nil
}
func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (extractpkg.ExtractRequest, error) {
var body extractPayload
if len(c.Body()) > 0 {
_ = c.BodyParser(&body)
}
req, err := s.baseExtractRequest(c, body, cfg)
if err != nil {
return extractpkg.ExtractRequest{}, err
}
req.URL = extractpkg.NormalizeURL(strings.TrimSpace(firstNonEmpty(c.Query("url"), body.URL)))
if err := validateExtractTargetURL(c.UserContext(), req.URL, cfg.AllowPrivateNetworks); err != nil {
return extractpkg.ExtractRequest{}, errInvalidParam(err.Error())
}
return req, nil
}
func (s *Server) newExtractor() extractpkg.Extractor {
return extractpkg.Extractor{
RawFetch: s.rawExtractFetch,
@@ -422,12 +439,13 @@ func SanitizeExtractError(err error) string {
const maxBatchExtractURLs = 20
type batchExtractPayload struct {
extractPayload
URLs []string `json:"urls"`
Mode string `json:"mode"`
}
// batchExtractItem is a response item for a single extracted URL, using
// page_content/metadata keys
// batchExtractItem is one entry of the bare-array /extract/batch response.
// The {page_content, metadata} shape is the Open WebUI ExternalWebLoader
// contract - do not wrap it in the Envelope.
type batchExtractItem struct {
PageContent string `json:"page_content"`
Metadata map[string]string `json:"metadata"`
@@ -444,49 +462,34 @@ func (s *Server) handleBatchExtract(c *fiber.Ctx) error {
return &APIError{HTTPStatus: fiber.StatusNotFound, ErrorCode: "not_found", Message: "Extraction is disabled"}
}
var body batchExtractPayload
if len(c.Body()) == 0 {
return errInvalidParam("request body is required")
}
var body batchExtractPayload
if err := c.BodyParser(&body); err != nil {
return errInvalidParam("invalid JSON body")
}
// Deduplicate and normalize URLs.
seen := make(map[string]struct{}, len(body.URLs))
var urls []string
for _, raw := range body.URLs {
u := extractpkg.NormalizeURL(strings.TrimSpace(raw))
if u == "" {
continue
}
if _, dup := seen[u]; dup {
continue
}
seen[u] = struct{}{}
urls = append(urls, u)
}
urls := dedupeBatchURLs(body.URLs)
if len(urls) == 0 {
return errInvalidParam("urls array is required and must contain at least one valid URL")
}
if len(urls) > maxBatchExtractURLs {
return errInvalidParam(fmt.Sprintf("urls array exceeds maximum of %d", maxBatchExtractURLs))
}
// Validate all URLs upfront.
for _, u := range urls {
if err := validateExtractTargetURL(c.UserContext(), u, cfg.AllowPrivateNetworks); err != nil {
return &APIError{HTTPStatus: fiber.StatusBadRequest, ErrorCode: "invalid_extract_url", Message: err.Error()}
}
baseReq, err := s.baseExtractRequest(c, body.extractPayload, cfg)
if err != nil {
return err
}
mode := firstNonEmpty(body.Mode, cfg.DefaultMode)
// Target URLs are validated in the fetch path, inside the workers - a bad
// URL becomes an error item instead of failing the whole batch (Open WebUI
// drops every doc on a non-2xx). 400 is reserved for malformed requests.
extractor := s.newExtractor()
results := make([]batchExtractItem, len(urls))
// Concurrent extraction with bounded parallelism (same pattern as
// EnrichEnvelopeWithExtraction).
ctx, cancel := context.WithTimeout(c.UserContext(), cfg.BatchTimeout(len(urls)))
// Bounded parallelism plus an aggregate deadline, same pattern as
// EnrichEnvelopeWithExtraction.
ctx, cancel := context.WithTimeout(requestCtx, cfg.BatchTimeout(len(urls)))
defer cancel()
sem := make(chan struct{}, cfg.MaxConcurrent)
@@ -497,50 +500,58 @@ func (s *Server) handleBatchExtract(c *fiber.Ctx) error {
go func(idx int, url string) {
defer wg.Done()
defer func() { <-sem }()
if err := ctx.Err(); err != nil {
results[idx] = batchExtractItem{
PageContent: "",
Metadata: map[string]string{"source": url, "error": "batch timeout"},
}
return
}
req := extractpkg.ExtractRequest{
URL: url,
Mode: extractpkg.Mode(mode),
ProxyURL: "",
LangCode: strings.TrimSpace(c.Query("lang")),
Timeout: cfg.Timeout,
MaxBytes: cfg.MaxBytes,
}
result, err := extractor.Extract(ctx, req)
if err != nil {
results[idx] = batchExtractItem{
PageContent: "",
Metadata: map[string]string{
"source": url,
"error": SanitizeExtractError(err),
},
}
return
}
results[idx] = batchExtractItem{
PageContent: result.Markdown,
Metadata: map[string]string{
"source": url,
"title": result.Title,
"description": result.Description,
"lang": result.Lang,
"canonical": result.Canonical,
"mode_used": result.Meta.ModeUsed,
"fetched_at": result.Meta.FetchedAt,
"took_ms": fmt.Sprintf("%d", result.Meta.TookMs),
},
}
results[idx] = batchExtractOne(ctx, extractor, baseReq, url)
}(i, u)
}
wg.Wait()
return c.JSON(results)
}
// dedupeBatchURLs normalizes, drops empties, and keeps first occurrence order.
func dedupeBatchURLs(raw []string) []string {
seen := make(map[string]struct{}, len(raw))
var urls []string
for _, r := range raw {
u := extractpkg.NormalizeURL(strings.TrimSpace(r))
if u == "" {
continue
}
if _, dup := seen[u]; dup {
continue
}
seen[u] = struct{}{}
urls = append(urls, u)
}
return urls
}
// batchExtractOne extracts a single URL, folding failures into the item.
func batchExtractOne(ctx context.Context, extractor extractpkg.Extractor, req extractpkg.ExtractRequest, url string) batchExtractItem {
fail := func(err error) batchExtractItem {
WithRequest(ctx).WithError(err).WithField("url", url).Warn("Batch extract failed")
return batchExtractItem{Metadata: map[string]string{"source": url, "error": SanitizeExtractError(err)}}
}
// Skip the fetch once the batch budget is spent.
if ctx.Err() != nil {
return fail(errors.New("batch timeout"))
}
req.URL = url
result, err := extractor.Extract(ctx, req)
if err != nil {
return fail(err)
}
return batchExtractItem{
PageContent: result.Markdown,
Metadata: map[string]string{
"source": url,
"title": result.Title,
"description": result.Description,
"lang": result.Lang,
"canonical": result.Canonical,
"mode_used": result.Meta.ModeUsed,
"fetched_at": result.Meta.FetchedAt,
"took_ms": strconv.FormatInt(result.Meta.TookMs, 10),
},
}
}

239
core/server_extract_test.go
View File

@@ -6,8 +6,8 @@ import (
"fmt"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
@@ -115,61 +115,48 @@ func TestValidateExtractTargetURLNormalizesBarePublicIP(t *testing.T) {
func TestBatchExtractSingleURL(t *testing.T) {
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`<html><body><article><h1>Test Page</h1><p>This is a test page with enough content to pass the minimum runes threshold for extraction in batch mode.</p></article></body></html>`))
}))
defer target.Close()
opts := DefaultServerOptions()
opts.Extract = extractpkg.Config{
Enabled: true,
DefaultMode: string(extractpkg.ModeFast),
Timeout: time.Second,
MaxBytes: 256 * 1024,
MaxConcurrent: 2,
AllowPrivateNetworks: true,
}
s := NewServerWithOptions("127.0.0.1", 0, opts)
body := fmt.Sprintf(`{"urls":["%s"]}`, target.URL)
req, err := http.NewRequest(http.MethodPost, "/extract/batch", strings.NewReader(body))
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req)
if err != nil {
t.Fatal(err)
}
s := batchExtractTestServer(true)
resp := postBatchExtract(t, s, fmt.Sprintf(`{"urls":["%s"]}`, target.URL))
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
}
var results []map[string]interface{}
if err := json.NewDecoder(resp.Body).Decode(&results); err != nil {
t.Fatalf("decode response: %v", err)
items := decodeBatchItems(t, resp)
if len(items) != 1 {
t.Fatalf("items count = %d, want 1", len(items))
}
if len(results) != 1 {
t.Fatalf("results count = %d, want 1", len(results))
if items[0].PageContent == "" {
t.Fatal("expected non-empty page_content")
}
if _, ok := results[0]["page_content"]; !ok {
t.Fatalf("expected page_content key, got keys: %v", reflect.ValueOf(results[0]).MapKeys())
}
if _, ok := results[0]["metadata"]; !ok {
t.Fatalf("expected metadata key, got keys: %v", reflect.ValueOf(results[0]).MapKeys())
if items[0].Metadata["title"] != "Test Page" {
t.Fatalf("metadata = %v, want title 'Test Page'", items[0].Metadata)
}
}
func TestBatchExtractHandlesMultipleURLs(t *testing.T) {
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`<html><body><article><h1>Multi</h1><p>Page with sufficient content for batch extraction test that verifies concurrent processing works correctly.</p></article></body></html>`))
}))
defer target.Close()
s := batchExtractTestServer(true)
resp := postBatchExtract(t, s, fmt.Sprintf(`{"urls":["%s/1","%s/2","%s/3"]}`, target.URL, target.URL, target.URL))
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
}
if items := decodeBatchItems(t, resp); len(items) != 3 {
t.Fatalf("items count = %d, want 3", len(items))
}
}
func batchExtractTestServer(allowPrivate bool) *Server {
opts := DefaultServerOptions()
opts.Extract = extractpkg.Config{
Enabled: true,
@@ -177,51 +164,163 @@ func TestBatchExtractHandlesMultipleURLs(t *testing.T) {
Timeout: time.Second,
MaxBytes: 256 * 1024,
MaxConcurrent: 2,
AllowPrivateNetworks: true,
AllowPrivateNetworks: allowPrivate,
}
s := NewServerWithOptions("127.0.0.1", 0, opts)
return NewServerWithOptions("127.0.0.1", 0, opts)
}
body := fmt.Sprintf(`{"urls":["%s/1","%s/2","%s/3"]}`, target.URL, target.URL, target.URL)
func postBatchExtract(t *testing.T, s *Server, body string, header ...[2]string) *http.Response {
t.Helper()
req, err := http.NewRequest(http.MethodPost, "/extract/batch", strings.NewReader(body))
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", "application/json")
for _, h := range header {
req.Header.Set(h[0], h[1])
}
resp, err := s.app.Test(req)
if err != nil {
t.Fatal(err)
}
return resp
}
func decodeBatchItems(t *testing.T, resp *http.Response) []batchExtractItem {
t.Helper()
var items []batchExtractItem
if err := json.NewDecoder(resp.Body).Decode(&items); err != nil {
t.Fatalf("decode response: %v", err)
}
return items
}
func TestBatchExtractReturnsPerURLErrors(t *testing.T) {
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`<html><body><article><h1>Good page</h1><p>Enough body text to extract something meaningful from this page in batch mode.</p></article></body></html>`))
}))
defer target.Close()
s := batchExtractTestServer(true)
resp := postBatchExtract(t, s, fmt.Sprintf(`{"urls":["%s","ftp://example.com/x"]}`, target.URL))
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
}
var results []map[string]interface{}
if err := json.NewDecoder(resp.Body).Decode(&results); err != nil {
t.Fatalf("decode response: %v", err)
items := decodeBatchItems(t, resp)
if len(items) != 2 {
t.Fatalf("items count = %d, want 2", len(items))
}
if len(results) != 3 {
t.Fatalf("results count = %d, want 3", len(results))
if !strings.Contains(items[0].PageContent, "Good page") || items[0].Metadata["error"] != "" {
t.Fatalf("first item = %+v, want successful extraction", items[0])
}
if items[1].PageContent != "" || items[1].Metadata["error"] == "" {
t.Fatalf("second item = %+v, want error item", items[1])
}
if items[1].Metadata["source"] != "ftp://example.com/x" {
t.Fatalf("second item source = %q", items[1].Metadata["source"])
}
}
func TestBatchExtractKeepsPrivateNetworkGuardPerItem(t *testing.T) {
s := batchExtractTestServer(false)
resp := postBatchExtract(t, s, `{"urls":["http://169.254.169.254/latest/meta-data/"]}`)
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
}
items := decodeBatchItems(t, resp)
if len(items) != 1 {
t.Fatalf("items count = %d, want 1", len(items))
}
if items[0].PageContent != "" || !strings.Contains(items[0].Metadata["error"], "not allowed") {
t.Fatalf("item = %+v, want blocked-target error", items[0])
}
}
func TestBatchExtractDedupesURLs(t *testing.T) {
var hits int32
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
_, _ = w.Write([]byte(`<html><body><article><h1>Dedup</h1><p>Some body text long enough for the extractor to produce markdown content here.</p></article></body></html>`))
}))
defer target.Close()
s := batchExtractTestServer(true)
body := fmt.Sprintf(`{"urls":["%s/a","%s/a","%s/b"]}`, target.URL, target.URL, target.URL)
resp := postBatchExtract(t, s, body)
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
}
if items := decodeBatchItems(t, resp); len(items) != 2 {
t.Fatalf("items count = %d, want 2", len(items))
}
if got := atomic.LoadInt32(&hits); got != 2 {
t.Fatalf("target hits = %d, want 2", got)
}
}
func TestBatchExtractDisabledReturns404(t *testing.T) {
opts := DefaultServerOptions()
opts.Extract = extractpkg.DefaultConfig()
opts.Extract.Enabled = false
s := NewServerWithOptions("127.0.0.1", 0, opts)
resp := postBatchExtract(t, s, `{"urls":["https://example.com"]}`)
defer resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusNotFound)
}
}
func TestBatchExtractRejectsInvalidMode(t *testing.T) {
s := batchExtractTestServer(true)
resp := postBatchExtract(t, s, `{"urls":["https://example.com"],"mode":"turbo"}`)
defer resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusBadRequest)
}
}
func TestBatchExtractRejectsBadProxyHeader(t *testing.T) {
s := batchExtractTestServer(true)
resp := postBatchExtract(t, s, `{"urls":["https://example.com"]}`, [2]string{"X-Proxy-URL", "not-a-proxy"})
defer resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusBadRequest)
}
}
func TestBatchExtractBodyLangReachesFetch(t *testing.T) {
var acceptLanguage string
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
acceptLanguage = r.Header.Get("Accept-Language")
_, _ = w.Write([]byte(`<html><body><article><h1>Lang</h1><p>Body text long enough to satisfy extraction thresholds for this test case.</p></article></body></html>`))
}))
defer target.Close()
s := batchExtractTestServer(true)
resp := postBatchExtract(t, s, fmt.Sprintf(`{"urls":["%s"],"lang":"de"}`, target.URL))
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
}
if !strings.HasPrefix(acceptLanguage, "de") {
t.Fatalf("Accept-Language = %q, want de-prefixed", acceptLanguage)
}
}
func TestBatchExtractRejectsEmptyURLs(t *testing.T) {
opts := DefaultServerOptions()
opts.Extract = extractpkg.DefaultConfig()
s := NewServerWithOptions("127.0.0.1", 0, opts)
req, err := http.NewRequest(http.MethodPost, "/extract/batch", strings.NewReader(`{"urls":[]}`))
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req)
if err != nil {
t.Fatal(err)
}
s := batchExtractTestServer(true)
resp := postBatchExtract(t, s, `{"urls":[]}`)
defer resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
@@ -230,27 +329,15 @@ func TestBatchExtractRejectsEmptyURLs(t *testing.T) {
}
func TestBatchExtractRejectsURLsOverLimit(t *testing.T) {
opts := DefaultServerOptions()
opts.Extract = extractpkg.DefaultConfig()
s := NewServerWithOptions("127.0.0.1", 0, opts)
// Build 21 URLs (limit is 20)
// 21 URLs, limit is 20
urls := make([]string, 21)
for i := 0; i < 21; i++ {
for i := range urls {
urls[i] = fmt.Sprintf("https://example.com/%d", i)
}
body, _ := json.Marshal(map[string][]string{"urls": urls})
req, err := http.NewRequest(http.MethodPost, "/extract/batch", strings.NewReader(string(body)))
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req)
if err != nil {
t.Fatal(err)
}
s := batchExtractTestServer(true)
resp := postBatchExtract(t, s, string(body))
defer resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {

BIN
docs/demo.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 137 KiB

53
docs/openapi.yaml
View File

@@ -536,6 +536,11 @@ paths:
extracted-text runes than this, escalate to a browser render.
`0` (default) uses the built-in floor. Ignored in `fast` and
`rendered` modes.
lang:
type: string
description: >
Language hint sent as `Accept-Language` on the target fetch.
The `lang` query parameter works as a fallback.
responses:
"200":
description: Extracted URL content
@@ -550,13 +555,19 @@ paths:
/extract/batch:
post:
tags: [Extract]
tags: [Search]
operationId: extractBatch
summary: Extract content from multiple URLs
description: >
Accepts an array of URLs and returns extracted page content for each
one. Each result contains `page_content` (markdown) and `metadata`
(title, source, lang, etc.).
Accepts up to 20 URLs (deduplicated) and returns a bare JSON array of
`{page_content, metadata}` items - the shape Open WebUI's external web
loader expects. A URL that fails validation or extraction yields an
item with `metadata.error` and an empty `page_content` instead of
failing the batch. `400` is reserved for malformed requests: empty or
oversized `urls`, unknown `mode`, invalid proxy headers.
parameters:
- $ref: "#/components/parameters/UseProxyHeader"
- $ref: "#/components/parameters/ProxyURLHeader"
requestBody:
required: true
content:
@@ -565,7 +576,7 @@ paths:
$ref: "#/components/schemas/BatchExtractRequest"
responses:
"200":
description: Batch extraction results
description: Batch extraction results, one item per unique URL
content:
application/json:
schema:
@@ -2076,20 +2087,44 @@ components:
items:
type: string
maxItems: 20
description: URLs to extract content from (max 20)
description: URLs to extract content from (max 20 after deduplication)
mode:
type: string
enum: [auto, fast, rendered]
description: "Extraction mode (default: auto)"
default: auto
description: Extraction mode
clean:
type: boolean
default: true
description: >
Article-only extraction (default). Set `false` for
whole-readable-body extraction.
use_llms_txt:
type: boolean
default: false
description: >
Probe `/llms-full.txt` then `/llms.txt` for site-root URLs and use
that markdown instead of scraping HTML.
min_runes:
type: integer
minimum: 0
description: Auto-mode escalation floor, same semantics as `/extract`
lang:
type: string
description: >
Language hint sent as `Accept-Language` on target fetches. The
`lang` query parameter works as a fallback.
BatchExtractItem:
type: object
properties:
page_content:
type: string
description: Extracted markdown content
description: Extracted markdown content, empty when the item failed
metadata:
type: object
additionalProperties:
type: string
description: Page metadata (title, source, lang, etc.)
description: >
Page metadata (source, title, lang, canonical, mode_used, etc.).
Failed items carry only `source` and `error`.