Rustem Kamalov
9f68086d47
refactor(engines): unify captcha/block/no-results detection across raw and browser paths; Fix google proxy block diagnostics
2026-07-02 00:47:21 +03:00
Rustem Kamalov
9cc69da758
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
...
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
2026-06-16 03:51:37 +03:00
Rustem Kamalov
04db5347a8
fix(core): harden extract and stabilize request protection
...
- breaker: stop counting client cancellations/deadlines (incl. bare
rate-limiter wait errors) and circuit-open as engine failures
- rate limiting: cache limiters in SearchEngineOptions and rawEngine so
pacing applies on raw/library paths; pool wrapper delegates
- /extract SSRF guard: public-IP-only policy with dial-time IP pinning,
redirect re-validation, rendered-mode preflight, http/https allow-list,
ErrTargetNotAllowed -> HTTP 400, extract.allow_private_networks
escape hatch (default off)
- browser: no rod Must* on the request path; CLI parses block_resources
without panicking
2026-06-12 18:09:40 +03:00
Rustem Kamalov
aa397eb2c1
Add new URL extraction endpoint and search feature
2026-06-04 02:22:38 +03:00
Rustem Kamalov
2b4a80fcb2
fix default search limits
2026-06-02 21:44:58 +03:00
Rustem Kamalov
d0006e30ff
dedup feature rendering, align default limit, simplify CSS guard
2026-06-02 04:21:56 +03:00
Rustem Kamalov
327b65869b
Fix yandex search limits
2026-06-02 04:01:14 +03:00
Rustem Kamalov
2b96a8330f
SERP features: AI summary, answer boxes, related searches/questions, etc - for supporting engines + other fixesa and improvements
2026-05-28 01:36:52 +03:00
Rustem Kamalov
cec16a08ed
Normalize query inputs and cache keys
2026-05-24 02:29:04 +03:00
Rustem Kamalov
25f4ff942b
feat(query): add region parameter for per-engine geo targeting (google, yandex, bing, duck)
2026-05-21 20:45:10 +03:00
Rustem Kamalov
82abe57aae
Improve SERP result classification and ad positioning
2026-05-19 00:32:11 +03:00
Rustem Kamalov
b3ca4e0803
feat(api): v2 response shape - absolute rank, ad/organic split, slimmer fields
2026-05-13 01:30:54 +03:00
Rustem Kamalov
00630a5d31
refactor: centralize engine prelude, page cleanup, and rate limiter
2026-05-11 11:35:09 +03:00
Rustem Kamalov
e8fb0c24fa
feat(proxy): support X-Proxy-URL via per-process browser pool
2026-04-28 03:46:08 +03:00
Rustem Kamalov
6992ad585d
feat: typed 400 validation errors with stable reason codes
...
Invalid limit/start/boolean params now return 400 bad_request with a
machine-readable reason field (INVALID_LIMIT, INVALID_START,
INVALID_PARAM, EMPTY_QUERY) instead of 500. Limit is validated in
range [1, 100].
2026-04-26 02:57:50 +03:00
Rustem Kamalov
0ccb13c36e
fix(proxy): isolate proxy health from captcha/parser errors, replace re-enable thrash with quarantine
2026-04-24 04:33:41 +03:00
Rustem Kamalov
d12a301cf8
Enhance error handling and logging across search engines
2026-04-21 03:48:52 +03:00
Rustem Kamalov
0b7fbd3a45
docs: add godoc for public api and readme response examples
2026-04-20 01:20:52 +03:00
Rustem Kamalov
3c07c7789a
Runtime proxy override
2026-04-01 17:38:13 +03:00
Rustem Kamalov
7cdd358cad
Add filter param to Google. Add start param for supporting engines
2026-03-17 01:54:25 +03:00
Pachakutiq
6691ebaadd
fix: Remove duplicated results for all search engines and both modes
2025-04-10 16:59:31 +08:00
Pachakutiq
84e2d46c4b
feat: Support http and socks5 proxy with auth for raw search
2025-04-08 16:06:41 +08:00
Rustem Kamalov
0a3c07b149
Collects google ads/answers + captcha solver test
2024-05-06 01:13:07 +03:00
Rustem Kamalov
38183b804a
Google images scroll parsing
2023-07-23 04:35:28 +03:00
Rustem Kamalov
df972585c1
Fix default ratelimiter
2023-06-30 22:04:26 +03:00
Rustem Kamalov
dd5e0a2069
Default ratelimeter
2023-06-28 22:20:19 +03:00
Rustem Kamalov
72bddf7fc9
SearchTimeout error handling and proper tabs close
2023-06-27 23:13:10 +03:00
Rustem Kamalov
3f84503ce3
Add captcha check and error
2023-06-25 20:41:47 +03:00
Rustem Kamalov
8b87e7b4ab
Init
2023-06-23 04:08:00 +03:00
