mirror of
https://github.com/karust/openserp.git
synced 2026-08-12 11:53:29 +08:00
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution) - Unify engines behind a single engineSpec registry (CLI + serve share it) - Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers - Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks - Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
This commit is contained in:
@@ -48,7 +48,6 @@ paths:
|
||||
- $ref: "#/components/parameters/FilterQuery"
|
||||
- $ref: "#/components/parameters/FeaturesQuery"
|
||||
- $ref: "#/components/parameters/ExtractQuery"
|
||||
- $ref: "#/components/parameters/ExtractTopQuery"
|
||||
- $ref: "#/components/parameters/ExtractModeQuery"
|
||||
- $ref: "#/components/parameters/MinRunesQuery"
|
||||
- $ref: "#/components/parameters/FormatQuery"
|
||||
@@ -330,7 +329,6 @@ paths:
|
||||
- $ref: "#/components/parameters/MegaDedupeQuery"
|
||||
- $ref: "#/components/parameters/MegaMergeQuery"
|
||||
- $ref: "#/components/parameters/ExtractQuery"
|
||||
- $ref: "#/components/parameters/ExtractTopQuery"
|
||||
- $ref: "#/components/parameters/ExtractModeQuery"
|
||||
- $ref: "#/components/parameters/MinRunesQuery"
|
||||
- $ref: "#/components/parameters/FormatQuery"
|
||||
@@ -834,25 +832,27 @@ components:
|
||||
name: extract
|
||||
in: query
|
||||
required: false
|
||||
description: Fetch and embed cleaned target-page content for top web results.
|
||||
description: >
|
||||
Fetch and embed cleaned target-page content for the top web results.
|
||||
Accepts a boolean or an integer depth: `extract=0`/`false` disables
|
||||
extraction; `extract=true`/`1` enriches the top result; `extract=N`
|
||||
(1-5) enriches the top N results. The tuning params `extract_mode` and
|
||||
`min_runes` imply `extract=true` (top 1) when present, unless
|
||||
`extract=0` is set explicitly.
|
||||
schema:
|
||||
type: boolean
|
||||
oneOf:
|
||||
- type: boolean
|
||||
- type: integer
|
||||
minimum: 0
|
||||
maximum: 5
|
||||
default: false
|
||||
ExtractTopQuery:
|
||||
name: extract_top
|
||||
in: query
|
||||
required: false
|
||||
description: Number of top organic results to enrich when `extract=true`.
|
||||
schema:
|
||||
type: integer
|
||||
minimum: 1
|
||||
maximum: 5
|
||||
default: 3
|
||||
ExtractModeQuery:
|
||||
name: extract_mode
|
||||
in: query
|
||||
required: false
|
||||
description: Extraction strategy for target pages.
|
||||
description: >
|
||||
Extraction strategy for target pages. Its presence implies
|
||||
`extract=true` unless `extract=0` is set explicitly.
|
||||
schema:
|
||||
type: string
|
||||
enum: [auto, fast, rendered]
|
||||
|
||||
Reference in New Issue
Block a user