diff --git a/README.md b/README.md index 6e39a23..b0e2ea9 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ [![Docker Pulls](https://img.shields.io/docker/v/karust/openserp)](https://hub.docker.com/repository/docker/karust/openserp) [![CI](https://github.com/karust/openserp/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/karust/openserp/actions/workflows/ci.yml) -**OpenSERP** is an API and CLI for accessing search engine results from **Google, Yandex, Baidu, Bing, and DuckDuckGo**. +**OpenSERP** is an API and CLI for accessing search engine results from **Google, Yandex, Baidu, Bing, and DuckDuckGo**. A developer-friendly alternative to paid SERP API services! **Official website:** [openserp.org](https://openserp.org) @@ -23,10 +23,11 @@ A developer-friendly alternative to paid SERP API services! - 🎯 **Advanced filters** - language, date range, file type, and site queries - 🌍 **Configurable** - proxy, cache, and resilient mode support - 🐳 **Docker-ready** - local and container deployment +- πŸ“ **Data Formats** - JSON, Markdown, Text, NdJSON response format types are supported ## Quick Start⚑️ -### Docker (Recommended) +### Docker ```bash # Run the API server via prebuilt image @@ -36,7 +37,7 @@ docker run -p 127.0.0.1:7000:7000 -it karust/openserp serve -a 0.0.0.0 -p 7000 docker compose up --build ``` -### From source +### From Source ```bash git clone https://github.com/karust/openserp.git @@ -45,164 +46,180 @@ go build -o openserp . ./openserp serve ``` -## 🌐 Megasearch & Megaimage +## API Docs -Search all engines at once: +- Swagger UI: `http://127.0.0.1:7000/docs` +- OpenAPI YAML: `http://127.0.0.1:7000/openapi.yaml` + +## Search Endpoints + +Available engine names: `google`, `yandex`, `baidu`, `bing`, `duckduckgo`. + +Dedicated engine endpoints: ```bash -curl "http://127.0.0.1:7000/mega/search?text=golang&limit=10" -``` - -Search only selected engines: - -```bash -curl "http://127.0.0.1:7000/mega/search?text=golang&engines=duckduckgo,bing&limit=15" -``` - -Advanced filtering: - -```bash -curl "http://127.0.0.1:7000/mega/search?text=Donald+Trump&engines=duckduckgo,bing&limit=20&date=20251005..20251005&lang=EN" -``` - -API response example: - -```json -[ - { - "rank": 1, - "url": "https://en.wikipedia.org/wiki/Golden_Retriever", - "title": "Golden Retriever - Wikipedia", - "description": "The Golden Retriever is a Scottish breed of retriever dog of medium size. It is characterised by a gentle and affectionate nature and a striking golden coat.", - "ad": false, - "engine": "duckduckgo" - }, - { - "rank": 2, - "url": "https://www.bing.com/ck/a?!&&p=6f15ac4589858d0a104cd6f55cc8", - "title": "Golden Retriever Dog Forums", - "description": "Oct 20, 2024 Β· Back in the 1970s, Golden Retrievers routinely lived until 16 and 17 years old, they are now...", - "ad": false, - "engine": "bing" - }, - { - "rank": 3, - "url": "http://www.baidu.com/link?url==2544q3ugc68j0scVxdpWCSX-gl2AmuCy1l7uRR3loIfS1", - "title": "golden retrieversζ˜―δ»€δΉˆζ„ζ€", - "description": "2025εΉ΄9月21ζ—₯golden retrievers 读音:ηΎŽθ‹± golden retrieversεŸΊζœ¬θ§£ι‡Š ι‡‘ζ―›ηŒŽηŠ¬ εˆ†θ―θ§£ι‡Š golden金(ι»„)θ‰²ηš„...", - "ad": false, - "engine": "baidu" - } -] +curl "http://127.0.0.1:7000/google/search?text=golang&limit=10" ``` Image search: ```bash -curl "http://127.0.0.1:7000/mega/image?text=golang logo&limit=20" +curl "http://127.0.0.1:7000/bing/image?text=golang+logo&limit=10" ``` -List available engines: +Megasearch: + +```bash +# Search all configured engines +curl "http://127.0.0.1:7000/mega/search?text=golang&limit=10" + +# Search selected engines +curl "http://127.0.0.1:7000/mega/search?text=golang&engines=duckduckgo,bing&limit=15" + +# Advanced filtering +curl "http://127.0.0.1:7000/mega/search?text=golang&engines=google,bing&limit=20&date=20250101..20251231&lang=EN" + +# Image megasearch +curl "http://127.0.0.1:7000/mega/image?text=golang+logo&limit=20" +``` + +List engines: ```bash curl "http://127.0.0.1:7000/mega/engines" ``` -**Available engines:** `google`, `yandex`, `baidu`, `bing`, `duckduckgo` +## πŸ” Query Parameters -## πŸ” Individual Engine APIs +Common parameters: -Common query parameters: - -| Parameter | Description | Example | -| --------- | -------------------- | --------------------------------- | -| `text` | Search query | `golang programming` | -| `lang` | Language code | `EN`, `DE`, `RU`, `ES` | -| `date` | Date range | `20230101..20231231` | -| `file` | File extension | `PDF`, `DOC`, `XLS` | -| `site` | Site-specific search | `github.com`, `stackoverflow.com` | -| `limit` | Number of results | `10`, `25`, `50` | +| Parameter | Description | Example | +| --------- | -------------------------- | ------------------------------------ | +| `text` | Search query | `golang programming` | +| `lang` | Language code | `EN`, `DE`, `RU`, `ES` | +| `date` | Date range | `20250101..20251231` | +| `file` | File extension | `pdf`, `doc`, `xls` | +| `site` | Site-specific search | `github.com` | +| `limit` | Number of results, max 100 | `10`, `25`, `50` | +| `start` | Pagination offset | `0`, `10`, `20` | +| `format` | Output format | `json`, `markdown`, `text`, `ndjson` | Engine-specific parameters: -| Parameter | Supported engines | Notes | -| --------- | ----------------------------------- | ------------------------------------------------------------------ | -| `start` | `google`, `bing`, `yandex`, `baidu` | Web search pagination offset. | -| `filter` | `google` | Duplicate filter (`true` hides similar, `false` includes similar). | -| `answers` | `google` | Include Google answer boxes in output with negative ranks. | +| Parameter | Supported engines | Notes | +| --------- | ----------------- | ---------------------------------------------------------------------- | +| `filter` | `google` | Duplicate filter: `true` hides similar results, `false` includes them. | +| `answers` | `google` | Include Google answer boxes in output. | -Examples: - -```bash -curl "http://127.0.0.1:7000/duck/search?text=golang&limit=7" -curl "http://127.0.0.1:7000/google/search?text=golang&lang=EN&limit=10" -curl "http://127.0.0.1:7000/bing/search?text=golang&limit=10&start=20" -curl "http://127.0.0.1:7000/yandex/search?text=golang&limit=10&start=10" -curl "http://127.0.0.1:7000/bing/image?text=golang&limit=20" -``` - -## Response Examples - -Interactive docs (OpenAPI + Swagger UI) are available at: - -- `http://127.0.0.1:7000/docs` -- `http://127.0.0.1:7000/openapi.yaml` - -### Web Search Response (`//search`) +## Search Response Example ```json -[ - { - "rank": 1, - "url": "https://go.dev/doc/", - "title": "Documentation - The Go Programming Language", - "description": "Official Go documentation, tutorials, references, and release notes.", - "ad": false +{ + "query": { + "text": "golang", + "engines_requested": ["google"] }, - { - "rank": 2, - "url": "https://pkg.go.dev/", - "title": "pkg.go.dev", - "description": "Go package discovery and API documentation.", - "ad": false + "meta": { + "request_id": "019dc6c1-da45-706e-a57c-d671fa2862ee", + "requested_at": "2026-04-25T22:27:52Z", + "took_ms": 6410, + "engines_failed": [], + "version": "1.0" + }, + "results": [ + { + "id": "s_78341aa47c336101", + "rank": 1, + "type": "organic", + "title": "Documentation - The Go Programming Language", + "url": "https://go.dev/doc/", + "display_url": "go.dev > doc", + "snippet": "Official Go documentation, tutorials, references, and release notes.", + "domain": "go.dev", + "favicon": "https://go.dev/favicon.ico", + "is_ad": false, + "position": { + "absolute": 1, + "page": 1, + "on_page": 1 + }, + "engine": "google", + "domain_info": { + "tld": "dev", + "sld": "go", + "is_gov": false, + "is_edu": false, + "is_social": false + }, + "classification": { + "content_type": "webpage" + } + } + ], + "pagination": { + "page": 1, + "has_more": true, + "next_start": 25 } -] +} ``` -### Image Search Response (`//image`) +## Mega Response Notes + +`/mega/search` returns the same envelope plus `clusters`. Results are deduplicated by normalized URL; clusters keep the per-engine occurrences: ```json -[ - { - "rank": 1, - "url": "https://golang.org/lib/godoc/images/go-logo-blue.svg", - "title": "Go Gopher Logo", - "description": "Source: https://go.dev/brand/", - "ad": false - }, - { - "rank": 2, - "url": "https://example.com/images/go-mascot.png", - "title": "Go mascot", - "description": "Height:800, Width:1200, Source Page: https://example.com/post", - "ad": false - } -] +{ + "id": "c_a1b2c3d4e5f6a1b2", + "canonical_url": "https://go.dev/", + "domain": "go.dev", + "title": "The Go Programming Language", + "occurrences": [ + { "engine": "google", "rank": 1, "result_id": "s_78341aa47c336101" }, + { "engine": "bing", "rank": 2, "result_id": "s_20f9f15f0c3d9f6d" } + ], + "engines_count": 2, + "best_rank": 1, + "score": 0.75 +} ``` -### Error Responses +## Image Response Example -`400 Bad Request` (invalid/missing query): +```json +{ + "id": "i_a1b2c3d4e5f6a1b2", + "rank": 1, + "type": "image", + "title": "Go Gopher Logo", + "image": { + "url": "https://example.com/images/go-logo.png", + "thumbnail": "https://example.com/images/go-logo-thumb.png", + "width": 1200, + "height": 800 + }, + "source": { + "page_url": "https://go.dev/brand/", + "domain": "go.dev" + }, + "engine": "bing" +} +``` + +## Error Responses + +`400 Bad Request`: ```json { "error": "bad_request", "code": 400, - "message": "Query cannot be empty" + "message": "EMPTY_QUERY: query cannot be empty: provide text, site, or file parameter", + "reason": "EMPTY_QUERY" } ``` -`503 Service Unavailable` (engine unavailable, captcha, timeout, or proxy path failure): +`503 Service Unavailable`: ```json { @@ -212,16 +229,6 @@ Interactive docs (OpenAPI + Swagger UI) are available at: } ``` -### Response Headers - -| Header | Values/Examples | Meaning | -| ------------------- | ------------------------------- | ------------------------------------------------------------- | -| `X-Cache` | `HIT`, `MISS`, `BYPASS` | Cache result for this response. | -| `X-Fallback-Engine` | `google`, `bing`, `duckduckgo` | Present when dedicated endpoint used fallback engine. | -| `X-Proxy-Mode` | `off`, `single`, `pool` | Proxy policy mode applied by resilient search. | -| `X-Proxy-Tag` | `residential`, `datacenter`, `` | Selected proxy pool tag. Empty when proxy mode is off/direct. | -| `X-Proxy-Used` | `direct`, `socks5://host:port` | Actual upstream route used to execute request. | - ## 🌍 Proxy Support OpenSERP supports HTTP and SOCKS5 proxies. @@ -233,35 +240,25 @@ Simple global proxy: ./openserp search bing "query" --proxy http://user:pass@127.0.0.1:8080 ``` -Advanced proxy configuration is available in [config.yaml](./config.yaml). -You can enable tagged proxy pools and per-request override via `X-Use-Proxy: ` or `X-Use-Proxy: direct`. +Advanced proxy configuration is available in [config.yaml](./config.yaml). You can enable tagged proxy pools and per-request override via `X-Use-Proxy: ` or `X-Use-Proxy: direct`. ## Health & Stats ```bash curl -i "http://127.0.0.1:7000/health" +curl "http://127.0.0.1:7000/ready" curl "http://127.0.0.1:7000/stats" curl "http://127.0.0.1:7000/stats/cache" curl "http://127.0.0.1:7000/stats/proxy" curl "http://127.0.0.1:7000/stats/cb" ``` -Useful response headers in server mode: `X-Cache`, `X-Fallback-Engine`,`X-Proxy-Mode`, `X-Proxy-Tag`, `X-Proxy-Used` - ## License This project is licensed under the MIT License. See [LICENSE](LICENSE). ## 🀝 Contributing -Contributions are welcome. See [CONTRIBUTE](./docs/CONTRIBUTING.md). Please feel free to submit your improvements! +Contributions are welcome. See [docs/CONTRIBUTING.md](./docs/CONTRIBUTING.md). -## πŸ‘Ύ Issues & Support - -If you encounter issues or have questions: - -- Open an issue on GitHub -- Check existing issues for similar reports -- Review the documentation and example config - -###### _"OpenSerp" is the name of this open-source project. Use of the name in a way that implies affiliation, endorsement, or official status is not permitted._ +###### _"OpenSERP" is the name of this open-source project. Use of the name in a way that implies affiliation, endorsement, or official status is not permitted._ diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7727384..43286c9 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,265 +1,253 @@ # OpenSERP Architecture -## 1. Overview +## Overview -OpenSERP is a Go API + CLI for search results extraction from Google, Yandex, Baidu, Bing, and DuckDuckGo. +OpenSERP is a Go API + CLI for search result extraction from Google, Yandex, Baidu, Bing, and DuckDuckGo. -It supports two execution modes: +Execution modes: -- Browser mode (default): headless Chromium via `go-rod`, with engine-specific DOM parsing. -- Raw HTTP mode: direct requests + HTML parsing (`goquery`) for engines that implement raw parsing. +- **Browser mode**: default path, headless Chromium via `go-rod`, supported by all engines. +- **Raw HTTP mode**: direct HTTP + `goquery`, currently supported by Google, Yandex, and Baidu. -Browser mode is the primary path and supports all engines. Raw mode currently supports Google, Yandex, and Baidu only. +Browser mode is the primary compatibility path. -## 2. Directory Structure +## Project Layout ```text openserp/ -β”œβ”€β”€ main.go # Entry point, executes cmd.RootCmd -β”œβ”€β”€ AGENTS.md # Contributor + agent project guidance -β”œβ”€β”€ README.md # User-facing quickstart and API overview -β”œβ”€β”€ config.yaml # Runtime configuration (loaded by Viper) +β”œβ”€β”€ main.go +β”œβ”€β”€ README.md +β”œβ”€β”€ config.yaml β”œβ”€β”€ docs/ -β”‚ β”œβ”€β”€ ARCHITECTURE.md # This architecture reference -β”‚ β”œβ”€β”€ openapi.yaml # OpenAPI 3.0 specification -β”‚ └── embed.go # Embeds openapi.yaml for /openapi.yaml endpoint +β”‚ β”œβ”€β”€ ARCHITECTURE.md +β”‚ β”œβ”€β”€ CONTRIBUTING.md +β”‚ β”œβ”€β”€ openapi.yaml +β”‚ └── embed.go β”œβ”€β”€ cmd/ -β”‚ β”œβ”€β”€ root.go # Cobra root command + Viper config binding/defaults -β”‚ β”œβ”€β”€ serve.go # HTTP server bootstrap, engine wiring, browser pooling -β”‚ β”œβ”€β”€ search.go # CLI one-shot search command -β”‚ └── proxy_policy.go # Proxy policy mapping from config to runtime +β”‚ β”œβ”€β”€ root.go +β”‚ β”œβ”€β”€ serve.go +β”‚ β”œβ”€β”€ search.go +β”‚ └── proxy_policy.go β”œβ”€β”€ core/ -β”‚ β”œβ”€β”€ common.go # Shared domain types: Query, SearchResult, SearchEngine -β”‚ β”œβ”€β”€ server.go # Fiber routes, request handlers, cache/proxy headers -β”‚ β”œβ”€β”€ middleware.go # CORS, request logging, JSON error envelope -β”‚ β”œβ”€β”€ browser.go # Chromium navigation lifecycle and page orchestration -β”‚ β”œβ”€β”€ http_client.go # Raw HTTP client (uTLS fingerprinting) -β”‚ β”œβ”€β”€ resilient.go # Retry + CB + rate limiting + proxy orchestration -β”‚ β”œβ”€β”€ retry.go # Backoff retry runner and retry conditions -β”‚ β”œβ”€β”€ circuit_breaker.go # Per-engine circuit breaker state machine -β”‚ β”œβ”€β”€ cache.go # In-memory TTL cache for API responses -β”‚ β”œβ”€β”€ proxy.go # Proxy normalization, pools, health/rotation, stats -β”‚ β”œβ”€β”€ logger.go # Logging setup helpers -β”‚ └── captcha.go # Captcha-related helpers/errors -β”œβ”€β”€ google/ # Google engine implementation -β”‚ β”œβ”€β”€ url.go # URL builders -β”‚ β”œβ”€β”€ search.go # Browser mode parser -β”‚ └── search_raw.go # Raw HTTP parser -β”œβ”€β”€ yandex/ # Yandex engine implementation -β”‚ β”œβ”€β”€ url.go -β”‚ β”œβ”€β”€ search.go -β”‚ └── search_raw.go -β”œβ”€β”€ baidu/ # Baidu engine implementation -β”‚ β”œβ”€β”€ url.go -β”‚ β”œβ”€β”€ search.go -β”‚ └── search_raw.go -β”œβ”€β”€ bing/ # Bing engine implementation (browser-only) -β”‚ β”œβ”€β”€ url.go -β”‚ └── search.go -β”œβ”€β”€ duckduckgo/ # DuckDuckGo engine implementation (browser-only) -β”‚ β”œβ”€β”€ url.go -β”‚ └── search.go -β”œβ”€β”€ testutil/ # Integration gating and shared test fixtures/helpers -└── .github/workflows/ci.yml # CI checks (test/vet/build/lint/openapi lint) +β”‚ β”œβ”€β”€ common.go +β”‚ β”œβ”€β”€ server.go +β”‚ β”œβ”€β”€ response.go +β”‚ β”œβ”€β”€ result.go +β”‚ β”œβ”€β”€ response_builder.go +β”‚ β”œβ”€β”€ clusters.go +β”‚ β”œβ”€β”€ format_markdown.go +β”‚ β”œβ”€β”€ format_text.go +β”‚ β”œβ”€β”€ enrichment_domain.go +β”‚ β”œβ”€β”€ enrichment_domains.yaml +β”‚ β”œβ”€β”€ middleware.go +β”‚ β”œβ”€β”€ browser.go +β”‚ β”œβ”€β”€ http_client.go +β”‚ β”œβ”€β”€ resilient.go +β”‚ β”œβ”€β”€ retry.go +β”‚ β”œβ”€β”€ circuit_breaker.go +β”‚ β”œβ”€β”€ cache.go +β”‚ β”œβ”€β”€ proxy.go +β”‚ β”œβ”€β”€ logger.go +β”‚ └── captcha.go +β”œβ”€β”€ google/ +β”œβ”€β”€ yandex/ +β”œβ”€β”€ baidu/ +β”œβ”€β”€ bing/ +β”œβ”€β”€ duckduckgo/ +└── testutil/ ``` -## 3. Key Interfaces and Types +## Core Interfaces ### `core.SearchEngine` -Contract for all engines: +All engines implement: -- `Search(Query) ([]SearchResult, error)` for web results -- `SearchImage(Query) ([]SearchResult, error)` for image results -- `IsInitialized() bool` for health readiness -- `Name() string` for endpoint and stats identity -- `GetRateLimiter() *rate.Limiter` for per-engine throttling +- `Search(context.Context, Query) ([]SearchResult, error)` +- `SearchImage(context.Context, Query) ([]SearchResult, error)` +- `IsInitialized() bool` +- `Name() string` +- `GetRateLimiter() *rate.Limiter` ### `core.Query` -Parsed from query parameters and request headers: +Parsed from query parameters (`text`, `lang`, `date`, `file`, `site`, `limit`, `start`, `filter`, `answers`) and the `X-Use-Proxy` request header. At least one of `text`, `site`, or `file` must be non-empty. -- `Text` (`text`) -- `LangCode` (`lang`) -- `DateInterval` (`date`, format `YYYYMMDD..YYYYMMDD`) -- `Filetype` (`file`) -- `Site` (`site`) -- `Limit` (`limit`, default `25`) -- `Start` (`start`, default `0`) -- `Filter` (`filter`, default `true`) -- `Answers` (`answers`, default `false`) -- `ProxyOverride` (`X-Use-Proxy` header: `` or `direct`) -- Internal runtime fields: `ProxyURL`, `Insecure` +### Internal `core.SearchResult` -Validation summary: +Engine parsers return the older internal shape: -- `start` must be `>= 0` -- At least one of `text`, `site`, or `file` must be non-empty -- Invalid query parsing is returned as JSON error response +- `Rank` +- `URL` +- `Title` +- `Description` +- `Ad` -### `core.SearchResult` +HTTP handlers convert this into the public v1 response through `core/response_builder.go`. -Single SERP item shape: - -- `rank` (int) -- `url` (string) -- `title` (string) -- `description` (string) -- `ad` (bool) - -Mega endpoints return `core.MegaSearchResult`, which extends `SearchResult` with: - -- `engine` (string) - -## 4. Request Flow +## HTTP Request Flow ```text HTTP request - -> Fiber router + -> Fiber middleware + -> RequestContextMiddleware + -> CORS + -> RequestLoggerMiddleware -> handleDedicatedEndpoint / handleMegaEndpoint -> Query.InitFromContext - -> ResilientSearcher.SearchPrimary/SearchWithFallback (or mega parallel search) - -> CircuitBreaker.AllowRequest - -> RateLimiter.Wait - -> Proxy policy resolution and proxy selection - -> RetryableSearch (backoff/retry loop) - -> Engine.Search / Engine.SearchImage - Browser path: Browser.Navigate(url) -> DOM parse -> []SearchResult - Raw path: raw HTTP request -> goquery parse -> []SearchResult - -> De-duplication (mega endpoints) - -> Cache.Set (if enabled and cacheable) - -> JSON response + X-Cache/X-Proxy-*/X-Fallback-Engine headers + -> resolveFormat + -> cache lookup for JSON responses only + -> ResilientSearcher + -> circuit breaker + -> rate limiter + -> proxy policy resolution + -> retry loop + -> engine.Search / engine.SearchImage + -> browser path: Browser.Navigate -> DOM parse -> []SearchResult + -> raw path: HTTP client -> goquery parse -> []SearchResult + -> response enrichment + -> stable IDs + -> normalized URL/display URL + -> pagination position + -> domain_info/classification + -> image metadata extraction + -> mega-only normalized URL dedupe + clusters + -> cache write for eligible JSON responses + -> output serializer: JSON, Markdown, text, or NDJSON ``` -## 5. Browser vs Raw Mode +## Public API Response -### Browser Mode (default) +JSON endpoints return a v1 envelope. -- Enabled when `server.raw_requests: false` -- Uses Chromium + `go-rod` navigation and page parsing -- Supported engines: Google, Yandex, Baidu, Bing, DuckDuckGo -- Best compatibility, but heavier resource usage +Top-level fields: -### Raw HTTP Mode +- `query`: request echo, including `engines_requested` +- `meta`: `request_id`, `requested_at`, `took_ms`, `engines_failed`, `version` +- `results`: normalized web or image results +- `pagination`: `page`, `has_more`, `next_start` +- `clusters`: only on `/mega/search` -- Enabled when `server.raw_requests: true` -- Uses direct HTTP + HTML parsing without launching a browser -- Supported engines: Google, Yandex, Baidu -- Faster/lighter, but less reliable for anti-bot protected pages and missing image support +Stable ID prefixes: -Mode switch options: +- `s_`: web search result +- `i_`: image result +- `c_`: mega search URL cluster -- Config: `server.raw_requests` -- CLI flag: `--raw` +`meta.engines_failed` is the only engine status list in the body. Clients can derive responded engines as: -## 6. Resilience Stack +```text +query.engines_requested - meta.engines_failed +``` -The effective request protection sequence is: +Dedicated endpoint fallback is represented by: -1. Rate limiter (`engine.GetRateLimiter().Wait`) -2. Retry with exponential backoff (`core/retry.go`) -3. Circuit breaker per engine (`core/circuit_breaker.go`) -4. Proxy selection/rotation + health tracking (`core/proxy.go`) -5. Response cache (API-level TTL cache in `core/cache.go`) +- `X-Fallback-Engine` +- `results[].engine` +- `meta.engines_failed` containing the primary engine + +## Mega Search + +`/mega/search` and `/mega/image` run selected engines in parallel. + +`/mega/search` behavior: + +- Uses `engines` query parameter if provided; otherwise uses all configured engines. +- Skips duplicate engine names. +- Allows partial success; failed engines are listed in `meta.engines_failed`. +- Deduplicates flat results by normalized URL. +- Builds `clusters` from all enriched results before flat dedupe. +- Sorts clusters by score descending, then best rank ascending. + +Cluster score: + +```text +sum(1 / rank for each occurrence) / engines_queried +``` + +The score is capped at `1.0` and rounded to two decimals. + +## Response Formatting + +`resolveFormat` supports: + +- `json` (default) +- `markdown` +- `text` +- `ndjson` + +The format can be selected with `?format=` or by `Accept` header: + +- `text/markdown` +- `text/plain` +- `application/x-ndjson` + +Only JSON responses use the response cache. Cached JSON refreshes request-scoped metadata before sending: + +- `meta.request_id` +- `meta.requested_at` +- `meta.took_ms` + +## Domain Enrichment + +`core/enrichment_domain.go` derives: + +- `domain_info`: public suffix, SLD, and category booleans +- `classification`: content type and known source hint + +Public suffix parsing uses `golang.org/x/net/publicsuffix`. + +Mutable domain category data lives in: + +```text +core/enrichment_domains.yaml +``` + +It can be replaced at runtime: + +```bash +OPENSERP_ENRICHMENT_DOMAINS_FILE=/path/to/enrichment_domains.yaml ./openserp serve +``` + +## Resilience Stack + +Request protection sequence: + +1. Engine rate limiter +2. Retry with backoff +3. Circuit breaker +4. Proxy policy and proxy health +5. Response cache Important behaviors: - `ErrCaptcha` is non-retryable. -- `ErrProxyUnavailable` does not record circuit-breaker failure. -- Dedicated endpoints are engine-pure by default (`allow_endpoint_fallback: false`). +- Proxy health is degraded only for proxy/network failures, not parser or captcha errors. +- Dedicated endpoints are engine-pure by default. +- Dedicated fallback is opt-in via `resilience.allow_endpoint_fallback`. - Fallback responses are not cached on dedicated endpoints. -## 7. Config Reference +## Proxy Model -Defaults below are the shipped defaults in `config.yaml` (if present). If the config file is missing, fallback defaults from `cmd/root.go` are applied. +Proxy policy can come from: -### `server` +- global config +- per-engine config +- per-request `X-Use-Proxy` -| Key | Default | Description | -| --------------------- | --------- | ---------------------------------- | -| `server.host` | `0.0.0.0` | API bind host | -| `server.port` | `7000` | API bind port | -| `server.debug` | `false` | Debug mode, forces headful browser | -| `server.verbose` | `true` | Info-level request logs | -| `server.raw_requests` | `false` | `true` = raw HTTP mode | -| `server.insecure` | `true` | Allow insecure TLS connections | +Supported request override values: -### `app` +- `X-Use-Proxy: direct` +- `X-Use-Proxy: ` -| Key | Default | Description | -| ------------------ | ------- | ------------------------------ | -| `app.timeout` | `15` | Request timeout in seconds | -| `app.browser_path` | `""` | Custom browser binary path | -| `app.profiles` | `""` | Override browser profiles JSON | -| `app.head` | `false` | Headful browser UI | -| `app.leakless` | `false` | Force browser process cleanup | -| `app.leave_head` | `false` | Keep browser tabs open | +Response headers: -### `proxies` +- `X-Proxy-Mode`: `off` or `tag_pool` +- `X-Proxy-Tag` +- `X-Proxy-Used` -| Key | Default | Description | -| ---------------------------------- | ------- | ----------------------------------------- | -| `proxies.global` | unset | Force single proxy for all engines | -| `proxies.entries[]` | empty | Tagged proxy pool entries (`url`, `tags`) | -| `proxies.health.failure_threshold` | `3` | Disable proxy after N failures | +## Config Reference -Per-engine optional proxy tag: +Config priority: `CLI flags > OPENSERP_* env vars > config.yaml > defaults` (via Viper). -- `google.proxy` -- `yandex.proxy` -- `baidu.proxy` -- `bing.proxy` -- `duckduckgo.proxy` - -### `cache` - -| Key | Default | Description | -| ------------------- | ------- | ------------------------------------- | -| `cache.ttl_seconds` | `60` | Response cache TTL (0 disables cache) | -| `cache.max_size` | `1000` | Max cached entries | - -### `resilience` - -| Key | Default | Description | -| ------------------------------------ | ------- | ------------------------------------------------------ | -| `resilience.max_retries` | `2` | Retry attempts per request | -| `resilience.allow_endpoint_fallback` | `false` | Allow dedicated endpoints to fallback to other engines | - -### `circuit_breaker` - -| Key | Default | Description | -| ---------------------------------- | ------- | ------------------------------------ | -| `circuit_breaker.failures` | `5` | Failures before opening circuit | -| `circuit_breaker.recovery_seconds` | `60` | Open -> half-open wait time | -| `circuit_breaker.successes` | `2` | Half-open successes to close circuit | - -### `cors` - -| Key | Default | Description | -| -------------------- | ------------------------------------------------------------ | --------------------------------- | -| `cors.enabled` | `true` | Enable CORS middleware | -| `cors.allow_origins` | `"*"` | Allowed origins | -| `cors.allow_methods` | `"GET, POST, OPTIONS"` | Allowed methods | -| `cors.allow_headers` | `"Origin, Content-Type, Accept, Authorization, X-Use-Proxy"` | Allowed headers | -| `cors.max_age` | `86400` | Preflight cache max age (seconds) | - -### `2captcha` - -| Key | Default | Description | -| ----------------- | ------- | --------------------------- | -| `2captcha.apikey` | unset | Optional captcha solver key | - -### Engine rate-limit defaults - -For each engine (`google`, `yandex`, `baidu`, `bing`, `duckduckgo`): - -| Key | Default | Description | -| --------------------------- | --------------- | ----------------------------- | -| `.rate_requests` | `4` | Average requests per minute | -| `.rate_burst` | `2` | Burst capacity | -| `.rate_seconds` | `60` (implicit) | Rate window seconds | -| `.selector_timeout` | `5` (implicit) | Selector wait timeout seconds | - -Google-only additional toggle: - -- `google.captcha` (default: `true`) +See [config.yaml](../config.yaml) for all available sections and defaults. diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 6e99915..9fa9dde 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -50,8 +50,8 @@ Create a new folder (example: `myengine/`) with: Your engine type must implement: -- `Search(core.Query) ([]core.SearchResult, error)` -- `SearchImage(core.Query) ([]core.SearchResult, error)` +- `Search(context.Context, core.Query) ([]core.SearchResult, error)` +- `SearchImage(context.Context, core.Query) ([]core.SearchResult, error)` - `IsInitialized() bool` - `Name() string` - `GetRateLimiter() *rate.Limiter` @@ -60,14 +60,14 @@ Use the existing engines (for example `google/`) as the reference pattern. ### 3) Register the engine in server wiring -Update [`cmd/serve.go`](cmd/serve.go): +Update [`cmd/serve.go`](../cmd/serve.go): - Add engine spec in `browserEngineSpecs()` - Add raw-mode handling if raw support exists ### 4) Add config block -Update [`config.yaml`](config.yaml) with your engine section: +Update [`config.yaml`](../config.yaml) with your engine section: - `rate_requests` - `rate_burst` @@ -115,6 +115,6 @@ For each PR: If you change API behavior, update: -- [`docs/openapi.yaml`](docs/openapi.yaml) -- [`README.md`](README.md) -- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) when flow/design changes +- [`openapi.yaml`](openapi.yaml) +- [`../README.md`](../README.md) +- [`ARCHITECTURE.md`](ARCHITECTURE.md) when flow/design changes