Rustem Kamalov
29c7b0fbe0
expose proxy-attempts + use-profile CORS headers; simplify DDG url builders
2026-07-22 15:17:16 +03:00
Parash Subedi
5d5d135fbf
feat: update DuckDuckGo search logic, use "more results" button for next page results ( #41 )
...
* feat: implement windowOrganicResults function and update DuckDuckGo search logic
* Add fallback ddg "more results" selector. Update test HTML + add test
---------
Co-authored-by: Rustem Kamalov <karust13@gmail.com >
2026-07-22 00:11:20 +03:00
Rustem Kamalov
2a57bd4248
fix: /extract/batch per-URL errors, param parity, proxy headers.
...
Update readme
2026-07-19 04:18:50 +03:00
Rustem Kamalov
01223e35d3
Merge pull request #39 from AIWintermuteAI/main
...
Add `/extract/batch` path for batch URL extraction
2026-07-18 21:44:46 +03:00
Rustem Kamalov
00edd86c1c
fix: openapi yaml parse error + /extract/batch timeout exemption
2026-07-18 21:31:46 +03:00
AIWintermuteAI
e430584324
refactor: use errInvalidParam helper and add per-URL took_ms to batch extract
2026-07-18 09:56:20 +02:00
AIWintermuteAI
aec0d9c42a
cleanup
2026-07-18 09:49:38 +02:00
AIWintermuteAI
58fdf2b7a6
docs: add /extract/batch endpoint to OpenAPI spec
2026-07-18 00:19:15 +02:00
AIWintermuteAI
c3934bb75b
feat: add POST /extract/batch endpoint for WebUI integration
2026-07-18 00:18:05 +02:00
Rustem Kamalov
e7827cc03d
refactor(browser): native CDP fingerprinting, drop patch.js + localized profiles. Update docker chrome
2026-07-13 06:07:37 +03:00
Rustem Kamalov
cf21bf5150
fix google: detect /sorry/ block pages; scope proxy sessions to credentials. Update default config
2026-07-09 19:30:21 +03:00
Rustem Kamalov
9f68086d47
refactor(engines): unify captcha/block/no-results detection across raw and browser paths; Fix google proxy block diagnostics
2026-07-02 00:47:21 +03:00
Rustem Kamalov
3d5eddfeed
fix(engines): better classify captcha/blocked/no-results pages instead of returning empty results, also enable it to <engine>/parse endpoints. Enable allow_request_proxy_url by default
2026-07-01 01:38:03 +03:00
Rustem Kamalov
e7af939599
fix: strip features when disabled and normalize feature output. Improve google+bing SERP features extraction
2026-06-29 10:36:46 +03:00
Rustem Kamalov
15bab0295f
refactor: extract shared RankState, unify rod/goquery parse paths
2026-06-27 18:04:25 +03:00
Rustem Kamalov
4a81ab774c
core/http: adopt tls-client for raw+extract fingerprinting. Update docs
2026-06-27 06:10:11 +03:00
Rustem Kamalov
ba27fcf1f4
remove dead code, pin golangci linters
2026-06-25 17:06:01 +03:00
Rustem Kamalov
9cc69da758
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
...
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
2026-06-16 03:51:37 +03:00
Rustem Kamalov
ca3143ccd1
Merge pull request #32 from karust/stability-safety-improvements
...
Stability and safety improvements
2026-06-12 19:14:29 +03:00
Rustem Kamalov
976ceeadf4
fix: report ErrSearchTimeout when results never hydrate; raise integration selector timeout to 15s
2026-06-12 19:06:46 +03:00
Rustem Kamalov
04db5347a8
fix(core): harden extract and stabilize request protection
...
- breaker: stop counting client cancellations/deadlines (incl. bare
rate-limiter wait errors) and circuit-open as engine failures
- rate limiting: cache limiters in SearchEngineOptions and rawEngine so
pacing applies on raw/library paths; pool wrapper delegates
- /extract SSRF guard: public-IP-only policy with dial-time IP pinning,
redirect re-validation, rendered-mode preflight, http/https allow-list,
ErrTargetNotAllowed -> HTTP 400, extract.allow_private_networks
escape hatch (default off)
- browser: no rod Must* on the request path; CLI parses block_resources
without panicking
2026-06-12 18:09:40 +03:00
Rustem Kamalov
665f09266a
fix: derive a per-request deadline from the retry budget.
...
Add `RequestTimeout` config that bounds wall-clock time of any request that does not manage its own deadline budget. It is derived from the engine timeout and retry budget via RequestTimeoutForRetries.
2026-06-12 04:40:31 +03:00
Rustem Kamalov
d20a7c53cd
fix: centralize engine panic recovery in resilient layer
...
A rod panic during SearchImage propagated uncaught into fasthttp and killed the whole process: per-engine recover blocks only covered Search (5 of 6 engines had no recovery on SearchImage), and there was no Fiber
recover middleware.
Regression test: a panicking SearchImage returns 502 engine_internal and the server keeps serving.
2026-06-12 04:24:06 +03:00
Rustem Kamalov
c9e548eb3a
Add release build job
2026-06-12 04:00:12 +03:00
Rustem Kamalov
59d076616c
Add docker image build job
2026-06-12 03:29:57 +03:00
Rustem Kamalov
0c7edcd7b6
fix integration tests, dedupe engine smoke tests
2026-06-10 15:42:26 +03:00
Rustem Kamalov
45287112df
fix docs + google: gate organic results by matched selector, harden answer-box parse
2026-06-10 15:15:42 +03:00
Rustem Kamalov
b31e84316c
Add contributor onboarding kit
2026-06-10 14:54:45 +03:00
Rustem Kamalov
ffd3d4250c
Add OpenSERP SDK usage examples
2026-06-09 04:14:35 +03:00
Rustem Kamalov
11d02731b6
Fix extraction limits and strategy
2026-06-09 02:28:54 +03:00
Rustem Kamalov
5be448e220
Add client SDKs and integrations
2026-06-05 19:48:27 +03:00
Rustem Kamalov
aa397eb2c1
Add new URL extraction endpoint and search feature
2026-06-04 02:22:38 +03:00
Rustem Kamalov
2b4a80fcb2
fix default search limits
2026-06-02 21:44:58 +03:00
Rustem Kamalov
1573642de8
extract region/UULE resolution into dependency-free core/region subpackage
2026-06-02 05:02:51 +03:00
Rustem Kamalov
d0006e30ff
dedup feature rendering, align default limit, simplify CSS guard
2026-06-02 04:21:56 +03:00
Rustem Kamalov
327b65869b
Fix yandex search limits
2026-06-02 04:01:14 +03:00
Rustem Kamalov
7ed0780192
Add google uule query param support for regional search
2026-06-02 03:54:55 +03:00
Rustem Kamalov
2b96a8330f
SERP features: AI summary, answer boxes, related searches/questions, etc - for supporting engines + other fixesa and improvements
2026-05-28 01:36:52 +03:00
Rustem Kamalov
cec16a08ed
Normalize query inputs and cache keys
2026-05-24 02:29:04 +03:00
Rustem Kamalov
25f4ff942b
feat(query): add region parameter for per-engine geo targeting (google, yandex, bing, duck)
2026-05-21 20:45:10 +03:00
Rustem Kamalov
82abe57aae
Improve SERP result classification and ad positioning
2026-05-19 00:32:11 +03:00
Rustem Kamalov
b3ca4e0803
feat(api): v2 response shape - absolute rank, ad/organic split, slimmer fields
2026-05-13 01:30:54 +03:00
Rustem Kamalov
00630a5d31
refactor: centralize engine prelude, page cleanup, and rate limiter
2026-05-11 11:35:09 +03:00
Rustem Kamalov
88337cd7ce
refactor: Speed up Baidu. Fix raw client + reuse HTML parsers for raw search
2026-05-08 00:00:33 +03:00
Rustem Kamalov
471e5ad7fe
refactor: speed up SERP parsing and centralize selectors
2026-05-06 04:12:15 +03:00
Rustem Kamalov
ed23af7acc
feat: add POST /{engine}/parse HTML parser endpoints. One source of truth for browser and raw mode selectors
2026-05-05 05:07:59 +03:00
Rustem Kamalov
3eaa30118e
fix(ecosia): wire proxy policy, validation, raw mode, and OpenAPI docs
2026-05-05 02:47:47 +03:00
Rustem Kamalov
718fad0542
Merge pull request #26 from miku/miku/ecosia
...
Add search engine ecosia
2026-05-04 02:51:08 +03:00
Martin Czygan
3964653e45
Add search engine ecosia
...
Ecosia's search results are [in parts] provided by its own search index,
Staan, developed by the co-founded European Search Perspective (EUSP) --
https://en.wikipedia.org/wiki/Ecosia
2026-05-03 13:56:14 +02:00
Rustem Kamalov
5d48690eb3
feat(mega): add mode fast/any/balanced with CB latency-based fast path
2026-05-02 16:30:56 +03:00
