feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs

- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
This commit is contained in:
Rustem Kamalov
2026-06-16 03:51:37 +03:00
parent ca3143ccd1
commit 9cc69da758
33 changed files with 1613 additions and 383 deletions

View File

@@ -48,7 +48,6 @@ paths:
- $ref: "#/components/parameters/FilterQuery"
- $ref: "#/components/parameters/FeaturesQuery"
- $ref: "#/components/parameters/ExtractQuery"
- $ref: "#/components/parameters/ExtractTopQuery"
- $ref: "#/components/parameters/ExtractModeQuery"
- $ref: "#/components/parameters/MinRunesQuery"
- $ref: "#/components/parameters/FormatQuery"
@@ -330,7 +329,6 @@ paths:
- $ref: "#/components/parameters/MegaDedupeQuery"
- $ref: "#/components/parameters/MegaMergeQuery"
- $ref: "#/components/parameters/ExtractQuery"
- $ref: "#/components/parameters/ExtractTopQuery"
- $ref: "#/components/parameters/ExtractModeQuery"
- $ref: "#/components/parameters/MinRunesQuery"
- $ref: "#/components/parameters/FormatQuery"
@@ -834,25 +832,27 @@ components:
name: extract
in: query
required: false
description: Fetch and embed cleaned target-page content for top web results.
description: >
Fetch and embed cleaned target-page content for the top web results.
Accepts a boolean or an integer depth: `extract=0`/`false` disables
extraction; `extract=true`/`1` enriches the top result; `extract=N`
(1-5) enriches the top N results. The tuning params `extract_mode` and
`min_runes` imply `extract=true` (top 1) when present, unless
`extract=0` is set explicitly.
schema:
type: boolean
oneOf:
- type: boolean
- type: integer
minimum: 0
maximum: 5
default: false
ExtractTopQuery:
name: extract_top
in: query
required: false
description: Number of top organic results to enrich when `extract=true`.
schema:
type: integer
minimum: 1
maximum: 5
default: 3
ExtractModeQuery:
name: extract_mode
in: query
required: false
description: Extraction strategy for target pages.
description: >
Extraction strategy for target pages. Its presence implies
`extract=true` unless `extract=0` is set explicitly.
schema:
type: string
enum: [auto, fast, rendered]