feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs

- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
This commit is contained in:
Rustem Kamalov
2026-06-16 03:51:37 +03:00
parent ca3143ccd1
commit 9cc69da758
33 changed files with 1613 additions and 383 deletions

View File

@@ -2,7 +2,7 @@ server:
host: 0.0.0.0 # API host to bind
port: 7000 # API port to bind
debug: false # Enable debug logs and force browser UI mode
verbose: true # Enable info-level request logs
verbose: false # Enable debug-level request logs
raw_requests: false # true = raw HTTP mode, false = browser mode
insecure: true # Allow insecure TLS connections
@@ -18,8 +18,6 @@ app:
idle_ttl: 5m # close a Chrome that has not served traffic for this long
mega_timeout: 90s # max total wait for /mega/* requests; slow engines return partial results
block_trackers: true
block_resources: "image,font,css,media"
extract:
enabled: true