Rustem Kamalov
e7827cc03d
refactor(browser): native CDP fingerprinting, drop patch.js + localized profiles. Update docker chrome
2026-07-13 06:07:37 +03:00
Rustem Kamalov
cf21bf5150
fix google: detect /sorry/ block pages; scope proxy sessions to credentials. Update default config
2026-07-09 19:30:21 +03:00
Rustem Kamalov
9f68086d47
refactor(engines): unify captcha/block/no-results detection across raw and browser paths; Fix google proxy block diagnostics
2026-07-02 00:47:21 +03:00
Rustem Kamalov
3d5eddfeed
fix(engines): better classify captcha/blocked/no-results pages instead of returning empty results, also enable it to <engine>/parse endpoints. Enable allow_request_proxy_url by default
2026-07-01 01:38:03 +03:00
Rustem Kamalov
e7af939599
fix: strip features when disabled and normalize feature output. Improve google+bing SERP features extraction
2026-06-29 10:36:46 +03:00
Rustem Kamalov
15bab0295f
refactor: extract shared RankState, unify rod/goquery parse paths
2026-06-27 18:04:25 +03:00
Rustem Kamalov
ba27fcf1f4
remove dead code, pin golangci linters
2026-06-25 17:06:01 +03:00
Rustem Kamalov
9cc69da758
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
...
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
2026-06-16 03:51:37 +03:00
Rustem Kamalov
976ceeadf4
fix: report ErrSearchTimeout when results never hydrate; raise integration selector timeout to 15s
2026-06-12 19:06:46 +03:00
Rustem Kamalov
d20a7c53cd
fix: centralize engine panic recovery in resilient layer
...
A rod panic during SearchImage propagated uncaught into fasthttp and killed the whole process: per-engine recover blocks only covered Search (5 of 6 engines had no recovery on SearchImage), and there was no Fiber
recover middleware.
Regression test: a panicking SearchImage returns 502 engine_internal and the server keeps serving.
2026-06-12 04:24:06 +03:00
Rustem Kamalov
0c7edcd7b6
fix integration tests, dedupe engine smoke tests
2026-06-10 15:42:26 +03:00
Rustem Kamalov
45287112df
fix docs + google: gate organic results by matched selector, harden answer-box parse
2026-06-10 15:15:42 +03:00
Rustem Kamalov
2b4a80fcb2
fix default search limits
2026-06-02 21:44:58 +03:00
Rustem Kamalov
1573642de8
extract region/UULE resolution into dependency-free core/region subpackage
2026-06-02 05:02:51 +03:00
Rustem Kamalov
d0006e30ff
dedup feature rendering, align default limit, simplify CSS guard
2026-06-02 04:21:56 +03:00
Rustem Kamalov
7ed0780192
Add google uule query param support for regional search
2026-06-02 03:54:55 +03:00
Rustem Kamalov
2b96a8330f
SERP features: AI summary, answer boxes, related searches/questions, etc - for supporting engines + other fixesa and improvements
2026-05-28 01:36:52 +03:00
Rustem Kamalov
25f4ff942b
feat(query): add region parameter for per-engine geo targeting (google, yandex, bing, duck)
2026-05-21 20:45:10 +03:00
Rustem Kamalov
82abe57aae
Improve SERP result classification and ad positioning
2026-05-19 00:32:11 +03:00
Rustem Kamalov
b3ca4e0803
feat(api): v2 response shape - absolute rank, ad/organic split, slimmer fields
2026-05-13 01:30:54 +03:00
Rustem Kamalov
00630a5d31
refactor: centralize engine prelude, page cleanup, and rate limiter
2026-05-11 11:35:09 +03:00
Rustem Kamalov
88337cd7ce
refactor: Speed up Baidu. Fix raw client + reuse HTML parsers for raw search
2026-05-08 00:00:33 +03:00
Rustem Kamalov
471e5ad7fe
refactor: speed up SERP parsing and centralize selectors
2026-05-06 04:12:15 +03:00
Rustem Kamalov
ed23af7acc
feat: add POST /{engine}/parse HTML parser endpoints. One source of truth for browser and raw mode selectors
2026-05-05 05:07:59 +03:00
Rustem Kamalov
0864de9887
Rework browser profiles. Add x-use-profile header. Selectors fixes
2026-04-30 23:32:39 +03:00
Rustem Kamalov
7a0fb21daf
feat: propagate query locale to accept-language header and browser profile
2026-04-27 04:28:53 +03:00
Rustem Kamalov
2b10aca4ac
feat: enhance query handling with locale support and date filtering for Bing, DuckDuckGo, Google, and Yandex
2026-04-27 04:07:28 +03:00
Rustem Kamalov
24f9bdab30
perf(browser): smart waits, fix captcha detection selectors + selectors single source of truth
2026-04-23 22:04:28 +03:00
Rustem Kamalov
d052c30bd5
feat(browser): replace stealth with custom profile functionality
2026-04-23 03:06:49 +03:00
Rustem Kamalov
0d4e2da960
Improve logging: add structured logs, enrich logs with useful info.
2026-04-22 03:39:30 +03:00
Rustem Kamalov
d12a301cf8
Enhance error handling and logging across search engines
2026-04-21 03:48:52 +03:00
Rustem Kamalov
08edec5779
fix(captcha): exit non-zero on misconfig, tighten solver gate tests
2026-04-20 20:12:19 +03:00
Rustem Kamalov
1842c4bb07
feat: implement DrainAndCloseResponse for safe HTTP response handling and update search handlers
2026-04-20 19:39:27 +03:00
Rustem Kamalov
e89b1a3a50
feat: propagate request context across search, retry, limiter, and browser navigation
2026-04-20 05:53:13 +03:00
Rustem Kamalov
0b7fbd3a45
docs: add godoc for public api and readme response examples
2026-04-20 01:20:52 +03:00
Rustem Kamalov
e5e0c96303
Add URL builder edge-case tests and server endpoint coverage
2026-04-15 02:50:52 +03:00
Rustem Kamalov
ecacfab4d6
Add integration tests for multiple search engines and enhance error handling
...
- Implement integration tests for Baidu, Bing, DuckDuckGo, Google, and Yandex.
- Refactor test utility functions to avoid core import cycles.
2026-04-14 02:05:58 +03:00
Rustem Kamalov
de59d07d47
Fix linter errors. Enhance error handling in search functions across multiple engines
2026-04-14 01:12:17 +03:00
Rustem Kamalov
9e22f68c6f
Add sanitized real html pages (google,yandex,baidu) for tests. Raw search HTML parser tests.
2026-04-14 00:41:35 +03:00
Rustem Kamalov
8bec5578c0
Implement proxy support with configuration, pooling, and integration tests
...
- Added ProxyConfig and ProxyPool structures to manage proxy settings and rotation.
- Implemented normalization functions for proxy URLs and configurations.
- Created a new ProxyPool that supports failure tracking and round-robin selection.
- Integrated proxy handling into the ResilientSearcher for search queries.
- Added integration tests for various proxy scenarios including SOCKS5 and HTTP proxies.
- Updated server and resilience stats to include proxy information.
- Refactored search functions to utilize the new proxy client.
- Removed direct proxy handling from search_raw.go and yandex/search_raw.go, using the new core proxy client instead.
- Added unit tests for proxy configuration and pool behavior.
2026-03-30 22:46:03 +03:00
Rustem Kamalov
7cdd358cad
Add filter param to Google. Add start param for supporting engines
2026-03-17 01:54:25 +03:00
Rustem Kamalov
74fc15a6e2
Add DuckDuckGo. Add custom logger. Fix arg priority and other issues
2025-09-30 05:27:54 +03:00
Rustem Kamalov
49e9837878
Fix build issues
2025-08-29 03:44:38 +03:00
Pachakutiq
8fdefb6ed5
fix(google)!: Fix missing datas
2025-08-28 16:44:22 +08:00
Pachakutiq
81fc057c83
feat(raw search)!: Use utls to mimic chrome TLS clienthello fingerprint
2025-08-28 16:43:39 +08:00
Rustem Kamalov
7f0f32ac4e
Handle failed navigation
2025-07-03 02:29:12 +03:00
Pachakutiq
6691ebaadd
fix: Remove duplicated results for all search engines and both modes
2025-04-10 16:59:31 +08:00
Pachakutiq
8099dae3ce
fix: Remove duplicated results in google/search.go
2025-04-10 15:35:54 +08:00
Pachakutiq
21f12c5996
fix: Google updated DOM of search result page
2025-04-08 18:48:07 +08:00
Pachakutiq
84e2d46c4b
feat: Support http and socks5 proxy with auth for raw search
2025-04-08 16:06:41 +08:00
