mirror of
https://github.com/karust/openserp.git
synced 2026-08-05 16:53:54 +08:00
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution) - Unify engines behind a single engineSpec registry (CLI + serve share it) - Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers - Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks - Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
This commit is contained in:
@@ -2,7 +2,7 @@ server:
|
||||
host: 0.0.0.0 # API host to bind
|
||||
port: 7000 # API port to bind
|
||||
debug: false # Enable debug logs and force browser UI mode
|
||||
verbose: true # Enable info-level request logs
|
||||
verbose: false # Enable debug-level request logs
|
||||
raw_requests: false # true = raw HTTP mode, false = browser mode
|
||||
insecure: true # Allow insecure TLS connections
|
||||
|
||||
@@ -18,8 +18,6 @@ app:
|
||||
idle_ttl: 5m # close a Chrome that has not served traffic for this long
|
||||
|
||||
mega_timeout: 90s # max total wait for /mega/* requests; slow engines return partial results
|
||||
block_trackers: true
|
||||
block_resources: "image,font,css,media"
|
||||
|
||||
extract:
|
||||
enabled: true
|
||||
|
||||
Reference in New Issue
Block a user