Rustem Kamalov
9cc69da758
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
...
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
2026-06-16 03:51:37 +03:00
Rustem Kamalov
04db5347a8
fix(core): harden extract and stabilize request protection
...
- breaker: stop counting client cancellations/deadlines (incl. bare
rate-limiter wait errors) and circuit-open as engine failures
- rate limiting: cache limiters in SearchEngineOptions and rawEngine so
pacing applies on raw/library paths; pool wrapper delegates
- /extract SSRF guard: public-IP-only policy with dial-time IP pinning,
redirect re-validation, rendered-mode preflight, http/https allow-list,
ErrTargetNotAllowed -> HTTP 400, extract.allow_private_networks
escape hatch (default off)
- browser: no rod Must* on the request path; CLI parses block_resources
without panicking
2026-06-12 18:09:40 +03:00
Rustem Kamalov
665f09266a
fix: derive a per-request deadline from the retry budget.
...
Add `RequestTimeout` config that bounds wall-clock time of any request that does not manage its own deadline budget. It is derived from the engine timeout and retry budget via RequestTimeoutForRetries.
2026-06-12 04:40:31 +03:00
Rustem Kamalov
45287112df
fix docs + google: gate organic results by matched selector, harden answer-box parse
2026-06-10 15:15:42 +03:00
Rustem Kamalov
11d02731b6
Fix extraction limits and strategy
2026-06-09 02:28:54 +03:00
Rustem Kamalov
aa397eb2c1
Add new URL extraction endpoint and search feature
2026-06-04 02:22:38 +03:00
Rustem Kamalov
2b4a80fcb2
fix default search limits
2026-06-02 21:44:58 +03:00
Rustem Kamalov
7ed0780192
Add google uule query param support for regional search
2026-06-02 03:54:55 +03:00
Rustem Kamalov
2b96a8330f
SERP features: AI summary, answer boxes, related searches/questions, etc - for supporting engines + other fixesa and improvements
2026-05-28 01:36:52 +03:00
Rustem Kamalov
cec16a08ed
Normalize query inputs and cache keys
2026-05-24 02:29:04 +03:00
Rustem Kamalov
82abe57aae
Improve SERP result classification and ad positioning
2026-05-19 00:32:11 +03:00
Rustem Kamalov
b3ca4e0803
feat(api): v2 response shape - absolute rank, ad/organic split, slimmer fields
2026-05-13 01:30:54 +03:00
Rustem Kamalov
88337cd7ce
refactor: Speed up Baidu. Fix raw client + reuse HTML parsers for raw search
2026-05-08 00:00:33 +03:00
Rustem Kamalov
471e5ad7fe
refactor: speed up SERP parsing and centralize selectors
2026-05-06 04:12:15 +03:00
Rustem Kamalov
ed23af7acc
feat: add POST /{engine}/parse HTML parser endpoints. One source of truth for browser and raw mode selectors
2026-05-05 05:07:59 +03:00
Rustem Kamalov
3eaa30118e
fix(ecosia): wire proxy policy, validation, raw mode, and OpenAPI docs
2026-05-05 02:47:47 +03:00
Martin Czygan
3964653e45
Add search engine ecosia
...
Ecosia's search results are [in parts] provided by its own search index,
Staan, developed by the co-founded European Search Perspective (EUSP) --
https://en.wikipedia.org/wiki/Ecosia
2026-05-03 13:56:14 +02:00
Rustem Kamalov
5d48690eb3
feat(mega): add mode fast/any/balanced with CB latency-based fast path
2026-05-02 16:30:56 +03:00
Rustem Kamalov
0864de9887
Rework browser profiles. Add x-use-profile header. Selectors fixes
2026-04-30 23:32:39 +03:00
Rustem Kamalov
063fb74f23
Add browser profile ID in logs and response headers
2026-04-30 20:46:08 +03:00
Rustem Kamalov
f9890ae10f
feat: add network bytes telemetry, mega timeout, and per-engine error details
2026-04-29 02:45:16 +03:00
Rustem Kamalov
e8fb0c24fa
feat(proxy): support X-Proxy-URL via per-process browser pool
2026-04-28 03:46:08 +03:00
Rustem Kamalov
7a0fb21daf
feat: propagate query locale to accept-language header and browser profile
2026-04-27 04:28:53 +03:00
Rustem Kamalov
2b10aca4ac
feat: enhance query handling with locale support and date filtering for Bing, DuckDuckGo, Google, and Yandex
2026-04-27 04:07:28 +03:00
Rustem Kamalov
d47bf08523
fix api response format issues
2026-04-26 02:58:13 +03:00
Rustem Kamalov
e4a84d8914
feat(browser): make configurable resource/tracker blocking
2026-04-24 04:02:23 +03:00
Rustem Kamalov
d052c30bd5
feat(browser): replace stealth with custom profile functionality
2026-04-23 03:06:49 +03:00
Rustem Kamalov
e02b374699
Add fingerprints debug tests and endpoint
2026-04-23 00:53:30 +03:00
Rustem Kamalov
47b5ad3b09
feat: graceful shutdown with drain signal and browser cleanup
2026-04-22 04:25:56 +03:00
Rustem Kamalov
0d4e2da960
Improve logging: add structured logs, enrich logs with useful info.
2026-04-22 03:39:30 +03:00
Rustem Kamalov
08edec5779
fix(captcha): exit non-zero on misconfig, tighten solver gate tests
2026-04-20 20:12:19 +03:00
Rustem Kamalov
e89b1a3a50
feat: propagate request context across search, retry, limiter, and browser navigation
2026-04-20 05:53:13 +03:00
Rustem Kamalov
ecacfab4d6
Add integration tests for multiple search engines and enhance error handling
...
- Implement integration tests for Baidu, Bing, DuckDuckGo, Google, and Yandex.
- Refactor test utility functions to avoid core import cycles.
2026-04-14 02:05:58 +03:00
Rustem Kamalov
bc59de46fb
Add CI and integration workflows
2026-04-14 01:04:32 +03:00
Rustem Kamalov
10decd5740
Fix config argument, update deault config.
2026-04-04 19:32:23 +03:00
Rustem Kamalov
3c07c7789a
Runtime proxy override
2026-04-01 17:38:13 +03:00
Rustem Kamalov
3daa48e6a3
Proxy policy wiring and fail-closed execution
...
- replace proxy wiring with global and per-engine tag policies
- split stats endpoints and lock fail-closed proxy behavior with tests
2026-04-01 00:54:21 +03:00
Rustem Kamalov
8bec5578c0
Implement proxy support with configuration, pooling, and integration tests
...
- Added ProxyConfig and ProxyPool structures to manage proxy settings and rotation.
- Implemented normalization functions for proxy URLs and configurations.
- Created a new ProxyPool that supports failure tracking and round-robin selection.
- Integrated proxy handling into the ResilientSearcher for search queries.
- Added integration tests for various proxy scenarios including SOCKS5 and HTTP proxies.
- Updated server and resilience stats to include proxy information.
- Refactored search functions to utilize the new proxy client.
- Removed direct proxy handling from search_raw.go and yandex/search_raw.go, using the new core proxy client instead.
- Added unit tests for proxy configuration and pool behavior.
2026-03-30 22:46:03 +03:00
Rustem Kamalov
0146891076
Add bounded TTL cache for dedicated and mega endpoints
2026-03-30 00:31:49 +03:00
Rustem Kamalov
f219c78d84
Add retry/circuit breaker, configurable fallback and CORS.
...
Based in part on work from PR #21 by @Sai-Prashanth123, adapted and integrated with project-specific fixes.
2026-03-25 23:31:41 +03:00
Rustem Kamalov
a4d459761e
Add custom browser path support
2026-03-17 02:18:51 +03:00
Rustem Kamalov
7cdd358cad
Add filter param to Google. Add start param for supporting engines
2026-03-17 01:54:25 +03:00
Rustem Kamalov
6c8e205af2
Fix duckduck date and limit filtering. Deduplicate results
2025-10-08 03:33:48 +03:00
Rustem Kamalov
0a347edc77
New megasearch and megaimage functionality. Update readme
2025-10-01 04:34:41 +03:00
Rustem Kamalov
de0a6a80d4
DuckDuck fixes. Non-stealth antibot fixes
2025-10-01 04:30:58 +03:00
Rustem Kamalov
74fc15a6e2
Add DuckDuckGo. Add custom logger. Fix arg priority and other issues
2025-09-30 05:27:54 +03:00
Rustem Kamalov
f8815b4560
Add Bing search+images
2025-09-19 02:11:55 +03:00
Rustem Kamalov
49e9837878
Fix build issues
2025-08-29 03:44:38 +03:00
Pachakutiq
c980ca4dc4
fix!: Allow insecure TLS connections
2025-08-28 16:35:53 +08:00
Rustem Kamalov
d8c3700252
Stealth browser as option. Update default config
2025-08-18 22:49:45 +03:00
