mirror of
https://github.com/karust/openserp.git
synced 2026-08-05 16:53:54 +08:00
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution) - Unify engines behind a single engineSpec registry (CLI + serve share it) - Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers - Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks - Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
This commit is contained in:
Rustem Kamalov
255
README.md
255
README.md
@@ -1,6 +1,6 @@
|
||||
|
||||
|
||||
# OpenSERP (Search Engine Results)
|
||||
# OpenSERP
|
||||
|
||||
[](https://goreportcard.com/report/github.com/karust/openserp)
|
||||
[](https://pkg.go.dev/github.com/karust/openserp)
|
||||
@@ -8,42 +8,47 @@
|
||||
[](https://hub.docker.com/r/karust/openserp)
|
||||
[](https://github.com/karust/openserp/actions/workflows/ci.yml)
|
||||
|
||||
**OpenSERP** is a free, open-source API and CLI for accessing normalized search engine results from **Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia**.
|
||||
**OpenSERP** is a free, open-source SERP API and CLI for live search data from **Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia**.
|
||||
|
||||
Run it locally, self-host it, or use the optional hosted API when you do not want to manage infrastructure.
|
||||
Use it as a search tool for **LLMs, agents, and RAG pipelines**, or as a scraper backend for **SEO rank tracking across Google, Yandex, Baidu, and more**. It is especially useful when your workflow needs RU/CN web coverage instead of another Google-only API.
|
||||
|
||||
Run it locally, self-host it, or use [OpenSERP Cloud](https://openserp.org/cloud) when you want the same public API shape without operating the server.
|
||||
|
||||
**Official website:** [openserp.org](https://openserp.org)
|
||||
|
||||
**Feedback:** [GitHub Issues](https://github.com/karust/openserp/issues) or [feedback@openserp.org](mailto:feedback@openserp.org)
|
||||
|
||||
**Latest updates, usage examples**: [Telegram](https://t.me/+RJEKspw3mUlhZDMy)
|
||||
|
||||
> 💡 OpenSERP is free and open-source. Only links listed in this repository and on the official website are associated with the project.
|
||||
|
||||
## Features
|
||||
|
||||
- 🔍 **Multi-engine** - search with dedicated endpoints for each engine
|
||||
- 🌐 **Megasearch** - cross-engine aggregation with deduplication
|
||||
- 🔍 **Multi-engine** - dedicated endpoints for Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia, with stable JSON for SEO rank pipelines
|
||||
- 🌐 **Megasearch** - `/mega/search` runs one query across every selected engine, then merges and dedupes results
|
||||
- 📄 **URL extraction** - return search results plus clean markdown/text target-page content in one call, for grounding and automation
|
||||
- ✨ **SERP features** - AI summaries, answer boxes, people-also-ask, and related searches in a response
|
||||
- 🖼 **Images** - image search is also available
|
||||
- 🎯 **Advanced filters** - language, date range, file type, and site queries
|
||||
- ✨ **SERP features** - AI summaries, answer boxes, people-also-ask, and related searches in a response
|
||||
- 📄 **URL extraction** - turn target pages into clean markdown/text for grounding and automation
|
||||
- 📝 **Data formats** - JSON, Markdown, Text, NdJSON response formats
|
||||
- 🌍 **Configurable** - proxy, cache, and resilient mode
|
||||
- 🐳 **Docker-ready** - local and container deployment
|
||||
- 📝 **Data Formats** - JSON, Markdown, Text, NdJSON response formats
|
||||
|
||||
## ⚡ Quick Start
|
||||
|
||||
### Docker
|
||||
|
||||
Prebuilt images are published to [docker hub: `karust/openserp`](https://hub.docker.com/r/karust/openserp).
|
||||
|
||||
```sh
|
||||
# Run the API server via prebuilt image
|
||||
docker run --rm -p 127.0.0.1:7000:7000 karust/openserp:latest serve -a 0.0.0.0 -p 7000
|
||||
|
||||
# Or use docker-compose (pulls the prebuilt image)
|
||||
# Or
|
||||
docker compose up
|
||||
```
|
||||
|
||||
### Go install
|
||||
|
||||
```sh
|
||||
go install github.com/karust/openserp@latest
|
||||
openserp search duckduckgo "open source serp api" --format markdown
|
||||
```
|
||||
|
||||
### From Source
|
||||
|
||||
```sh
|
||||
@@ -53,10 +58,117 @@ go build -o openserp .
|
||||
./openserp serve
|
||||
```
|
||||
|
||||
### First request
|
||||
|
||||
```sh
|
||||
curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust&extract=1&mode=any"
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Example JSON response</summary>
|
||||
|
||||
```json
|
||||
{
|
||||
"query": {
|
||||
"text": "golang vs rust",
|
||||
"engines_requested": ["bing", "google"]
|
||||
},
|
||||
"meta": {
|
||||
"request_id": "019ecdc0-a66d-79a4-9d2b-9e9b480d495e",
|
||||
"requested_at": "2026-06-16T00:06:55Z",
|
||||
"took_ms": 720,
|
||||
"engines_responded": ["bing"],
|
||||
"engines_failed": [],
|
||||
"version": "2.1"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"id": "s_5a8273f16b19ab64",
|
||||
"rank": 1,
|
||||
"type": "organic",
|
||||
"title": "The Go Programming Language",
|
||||
"url": "https://go.dev/",
|
||||
"display_url": "go.dev",
|
||||
"snippet": "Get Started Playground Tour Stack Overflow Help Packages Standard Library About Go Packages About Download Blog Issue Tracker Release Notes Brand Guidelines Code of Conduct Connect …",
|
||||
"domain": "go.dev",
|
||||
"favicon": "https://go.dev/favicon.ico",
|
||||
"position": {
|
||||
"absolute": 1
|
||||
},
|
||||
"engine": "bing",
|
||||
"domain_info": {
|
||||
"tld": "dev",
|
||||
"sld": "go",
|
||||
"category": ""
|
||||
},
|
||||
"extracted": {
|
||||
"title": "Build simple, secure, scalable systems with Go",
|
||||
"format": "markdown",
|
||||
"content": "## Build simple, secure, scalable systems with Go\n\n\n\n- “At the time, no single team member knew Go, but **within a month, everyone was writing in Go** and we were building out the endpoints. It was the flexibility, how easy it was to use, and the really cool concept behind Go (how Go handles native concurrency, garbage collection, and of course safety+speed.) that helped engage us during the build. Also, who can beat that cute mascot!”\n ........",
|
||||
"mode_used": "fast",
|
||||
"fetched_at": "2026-06-16T00:06:56Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "s_1a364ebcb3035539",
|
||||
"rank": 2,
|
||||
"type": "organic",
|
||||
"title": "Go (programming language) - Wikipedia",
|
||||
"url": "https://en.wikipedia.org/wiki/Go_(programming_language)",
|
||||
"display_url": "en.wikipedia.org › wiki › Go_(programming_language)",
|
||||
"snippet": "In Go's package system, each package has a path (e.g., \"compress/bzip2\" or \"golang.org/x/net/html\") and a name (e.g., bzip2 or html). By default other packages' definitions must always be prefixed with …",
|
||||
"domain": "en.wikipedia.org",
|
||||
"favicon": "https://en.wikipedia.org/favicon.ico",
|
||||
"position": {
|
||||
"absolute": 2
|
||||
},
|
||||
"engine": "bing",
|
||||
"domain_info": {
|
||||
"tld": "org",
|
||||
"sld": "wikipedia",
|
||||
"category": ""
|
||||
},
|
||||
"classification": {
|
||||
"content_type": "article",
|
||||
"source_hint": "encyclopedia"
|
||||
}
|
||||
},
|
||||
...
|
||||
],
|
||||
"serp_features": [],
|
||||
"pagination": {
|
||||
"page": 1,
|
||||
"has_more": false,
|
||||
"next_start": 10
|
||||
},
|
||||
"clusters": [
|
||||
{
|
||||
"id": "c_f20b23a020101dce",
|
||||
"canonical_url": "https://go.dev/",
|
||||
"domain": "go.dev",
|
||||
"title": "The Go Programming Language",
|
||||
"occurrences": [
|
||||
{
|
||||
"engine": "bing",
|
||||
"rank": 1,
|
||||
"result_id": "s_5a8273f16b19ab64"
|
||||
}
|
||||
],
|
||||
"engines_count": 1,
|
||||
"best_rank": 1,
|
||||
"score": 0.5
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Deployment Options
|
||||
|
||||
- **Self-hosted (this repo)** - free, MIT-licensed, with full control over runtime, proxies, cache, and scaling.
|
||||
- **[Hosted API](https://openserp.org/cloud)** - optional managed version from the project maintainers, with the same API shape.
|
||||
- **[OpenSERP Cloud](https://openserp.org/cloud)** - optional managed version from the project maintainers, with the same API shape.
|
||||
|
||||
The hosted API helps fund continued development of the open-source project. Same endpoints, same response schema, and client code can migrate either direction.
|
||||
|
||||
@@ -109,16 +221,26 @@ curl "http://127.0.0.1:7000/bing/image?text=golang+logo&limit=10"
|
||||
Megasearch:
|
||||
|
||||
```bash
|
||||
# Search all configured engines
|
||||
curl "http://127.0.0.1:7000/mega/search?text=golang&limit=10"
|
||||
```
|
||||
|
||||
# Fast mode: only one fastest engine is queried
|
||||
| Mode | Best for | Behavior |
|
||||
| ---------- | ------------------------------------ | ---------------------------------------------- |
|
||||
| `balanced` | Most multi-engine SERP workflows | Queries engines in parallel and merges results |
|
||||
| `fast` | Lowest latency | Uses the fastest available engine |
|
||||
| `any` | Fallback-style availability checking | Tries engines sequentially until one responds |
|
||||
|
||||
<details>
|
||||
<summary>More megasearch examples</summary>
|
||||
|
||||
```bash
|
||||
# Fast mode
|
||||
curl "http://127.0.0.1:7000/mega/search?text=golang&mode=fast&engines=google,bing,yandex"
|
||||
|
||||
# Any mode: sequential fallback in provided order (default order if none provided)
|
||||
# Any mode
|
||||
curl "http://127.0.0.1:7000/mega/search?text=golang&mode=any&engines=google,yandex,bing"
|
||||
|
||||
# Balanced mode (default): parallel all engines with aggregation controls
|
||||
# Balanced mode with aggregation controls
|
||||
curl "http://127.0.0.1:7000/mega/search?text=golang&mode=balanced&dedupe=true&merge=true"
|
||||
|
||||
# Advanced filtering
|
||||
@@ -128,6 +250,8 @@ curl "http://127.0.0.1:7000/mega/search?text=golang&engines=google,bing&limit=20
|
||||
curl "http://127.0.0.1:7000/mega/image?text=golang+logo&limit=20"
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
List engines:
|
||||
|
||||
```bash
|
||||
@@ -144,27 +268,84 @@ curl "http://127.0.0.1:7000/extract?url=https://example.com&mode=auto"
|
||||
curl "http://127.0.0.1:7000/extract?url=https://example.com&format=markdown"
|
||||
|
||||
# Embed extracted content under the top search results
|
||||
curl "http://127.0.0.1:7000/google/search?text=llm+observability&extract=true&extract_top=2&format=markdown"
|
||||
curl "http://127.0.0.1:7000/google/search?text=llm+observability&extract=2&format=markdown"
|
||||
```
|
||||
|
||||
## 🖥 CLI Search
|
||||
|
||||
No server required - query an engine straight from the terminal. The CLI shares the same engines, formats, and filters as the API.
|
||||
|
||||
```sh
|
||||
openserp search duckduckgo "free open source serp" --format markdown
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>CLI output and more examples</summary>
|
||||
|
||||
```markdown
|
||||
# Search results for "free open source serp"
|
||||
|
||||
**Query:** free open source serp - **Engines:** duckduckgo - **Took:** 1794ms
|
||||
|
||||
## Results
|
||||
|
||||
### 1. OpenSERP: Open-Source, Self-Hosted & Free SERP API
|
||||
|
||||
**openserp.org** - organic
|
||||
|
||||
OpenSERP is a free, open-source and self-hosted SERP API for Google, Bing, Yandex, Baidu, DuckDuckGo and Ecosia, with an optional managed Cloud path.
|
||||
|
||||
-> https://openserp.org/
|
||||
|
||||
### 2. GitHub - karust/openserp: Open-source SERP API for AI, SEO & automation ...
|
||||
|
||||
**github.com › karust › openserp** - organic
|
||||
|
||||
OpenSERP is a free, open-source API and CLI for accessing normalized search engine results from Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia. Run it locally, self-host it, or use the optional hosted API when you do not want to manage infrastructure.
|
||||
|
||||
-> https://github.com/karust/openserp
|
||||
```
|
||||
|
||||
More CLI examples:
|
||||
|
||||
```sh
|
||||
# JSON is the default format
|
||||
openserp search google "golang generics" --limit 20
|
||||
|
||||
# Plain text, German results
|
||||
openserp search yandex "wetter berlin" --format text --lang DE --region DE
|
||||
|
||||
# Restrict to a site and stream NdJSON
|
||||
openserp search bing "release notes" --site github.com --format ndjson
|
||||
|
||||
# Embed clean page content from the top 2 results
|
||||
openserp search google "llm observability" --extract 2 --format markdown
|
||||
|
||||
# Browserless (raw HTTP) mode through a proxy
|
||||
openserp search duckduckgo "free open source serp" --raw --proxy http://user:pass@127.0.0.1:8080
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
Run `openserp search --help` for the full flag list. Engine names: `google`, `yandex`, `baidu`, `bing`, `duckduckgo`, `ecosia`.
|
||||
|
||||
## 🔍 Query Parameters
|
||||
|
||||
Common parameters:
|
||||
|
||||
| Parameter | Description | Example |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------ |
|
||||
| `text` | Search query | `golang programming` |
|
||||
| `lang` | Language code | `EN`, `DE`, `RU`, `ES` |
|
||||
| `region` | Market/location hint. Countries/locales work across engines; Google also accepts city names via `uule`; Yandex accepts numeric `lr`. | `DE`, `en-GB`, `Berlin`, `213` |
|
||||
| `date` | Date range | `20250101..20251231` |
|
||||
| `file` | File extension | `pdf`, `doc`, `xls` |
|
||||
| `site` | Site-specific search | `github.com` |
|
||||
| `limit` | Number of organic results, max 100. When omitted or `<=10`, only the first SERP page is parsed. | `25`, `50` |
|
||||
| `start` | Pagination offset | `0`, `10`, `20` |
|
||||
| `format` | Output format | `json`, `markdown`, `text`, `ndjson` |
|
||||
| `extract` | Fetch and embed target-page content for top web results | `true` |
|
||||
| `extract_top` | Number of top web results to extract, clamped to 1-5 | `3` |
|
||||
| `extract_mode` | Extraction strategy: raw HTTP first, raw only, or browser-rendered | `auto`, `fast`, `rendered` |
|
||||
| Parameter | Description | Example |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
|
||||
| `text` | Search query | `golang programming` |
|
||||
| `lang` | Language code | `EN`, `DE`, `RU`, `ES` |
|
||||
| `region` | Market/location hint. Countries/locales work across engines; Google also accepts city names via `uule`; Yandex accepts numeric `lr`. | `DE`, `en-GB`, `Berlin`, `213` |
|
||||
| `date` | Date range | `20250101..20251231` |
|
||||
| `file` | File extension | `pdf`, `doc`, `xls` |
|
||||
| `site` | Site-specific search | `github.com` |
|
||||
| `limit` | Number of organic results, max 100. When omitted or `<=10`, only the first SERP page is parsed. | `25`, `50` |
|
||||
| `start` | Pagination offset | `0`, `10`, `20` |
|
||||
| `format` | Output format | `json`, `markdown`, `text`, `ndjson` |
|
||||
| `extract` | Fetch and embed target-page content for top web results. Bool or int depth: `0`/`false` off, `true`/`1` top result, `N` top N (1-5). `extract_mode`/`min_runes` imply `extract=true` unless `extract=0` | `1`, `3`, `true` |
|
||||
| `extract_mode` | Extraction strategy: raw HTTP first, raw only, or browser-rendered | `auto`, `fast`, `rendered` |
|
||||
|
||||
Engine-specific parameters:
|
||||
|
||||
@@ -335,7 +516,7 @@ Contributions are welcome. See [docs/CONTRIBUTING.md](./docs/CONTRIBUTING.md).
|
||||
## Feedback & Updates
|
||||
|
||||
- [GitHub Issues](https://github.com/karust/openserp/issues) - bugs, feature ideas, and reproducible issues.
|
||||
- [Telegram channel](https://t.me/openserp_cloud) - OpenSERP news, release notes, and project updates. Direct messages are open for quick feedback and hosted API questions.
|
||||
- [feedback@openserp.org](mailto:feedback@openserp.org) - private notes, longer feedback, or anything that does not fit GitHub Issues.
|
||||
- [Telegram Channel](https://t.me/+RJEKspw3mUlhZDMy) - OpenSERP news, release notes, and project updates. Direct messages are open for quick feedback and hosted API questions.
|
||||
|
||||
###### _"OpenSERP" is the name of this open-source project. The official website is [openserp.org](https://openserp.org). Resources not linked on this page are not affiliated with the project._
|
||||
> OpenSERP is free and open-source. Only links listed in this repository and on [openserp.org](https://openserp.org) are associated with the project.
|
||||
|
||||
Reference in New Issue
Block a user