Rustem Kamalov
665f09266a
fix: derive a per-request deadline from the retry budget.
...
Add `RequestTimeout` config that bounds wall-clock time of any request that does not manage its own deadline budget. It is derived from the engine timeout and retry budget via RequestTimeoutForRetries.
2026-06-12 04:40:31 +03:00
Rustem Kamalov
d20a7c53cd
fix: centralize engine panic recovery in resilient layer
...
A rod panic during SearchImage propagated uncaught into fasthttp and killed the whole process: per-engine recover blocks only covered Search (5 of 6 engines had no recovery on SearchImage), and there was no Fiber
recover middleware.
Regression test: a panicking SearchImage returns 502 engine_internal and the server keeps serving.
2026-06-12 04:24:06 +03:00
Rustem Kamalov
c9e548eb3a
Add release build job
2026-06-12 04:00:12 +03:00
Rustem Kamalov
59d076616c
Add docker image build job
2026-06-12 03:29:57 +03:00
Rustem Kamalov
0c7edcd7b6
fix integration tests, dedupe engine smoke tests
2026-06-10 15:42:26 +03:00
Rustem Kamalov
45287112df
fix docs + google: gate organic results by matched selector, harden answer-box parse
2026-06-10 15:15:42 +03:00
Rustem Kamalov
b31e84316c
Add contributor onboarding kit
2026-06-10 14:54:45 +03:00
Rustem Kamalov
ffd3d4250c
Add OpenSERP SDK usage examples
2026-06-09 04:14:35 +03:00
Rustem Kamalov
11d02731b6
Fix extraction limits and strategy
2026-06-09 02:28:54 +03:00
Rustem Kamalov
5be448e220
Add client SDKs and integrations
2026-06-05 19:48:27 +03:00
Rustem Kamalov
aa397eb2c1
Add new URL extraction endpoint and search feature
2026-06-04 02:22:38 +03:00
Rustem Kamalov
2b4a80fcb2
fix default search limits
2026-06-02 21:44:58 +03:00
Rustem Kamalov
1573642de8
extract region/UULE resolution into dependency-free core/region subpackage
2026-06-02 05:02:51 +03:00
Rustem Kamalov
d0006e30ff
dedup feature rendering, align default limit, simplify CSS guard
2026-06-02 04:21:56 +03:00
Rustem Kamalov
327b65869b
Fix yandex search limits
2026-06-02 04:01:14 +03:00
Rustem Kamalov
7ed0780192
Add google uule query param support for regional search
2026-06-02 03:54:55 +03:00
Rustem Kamalov
2b96a8330f
SERP features: AI summary, answer boxes, related searches/questions, etc - for supporting engines + other fixesa and improvements
2026-05-28 01:36:52 +03:00
Rustem Kamalov
cec16a08ed
Normalize query inputs and cache keys
2026-05-24 02:29:04 +03:00
Rustem Kamalov
25f4ff942b
feat(query): add region parameter for per-engine geo targeting (google, yandex, bing, duck)
2026-05-21 20:45:10 +03:00
Rustem Kamalov
82abe57aae
Improve SERP result classification and ad positioning
2026-05-19 00:32:11 +03:00
Rustem Kamalov
b3ca4e0803
feat(api): v2 response shape - absolute rank, ad/organic split, slimmer fields
2026-05-13 01:30:54 +03:00
Rustem Kamalov
00630a5d31
refactor: centralize engine prelude, page cleanup, and rate limiter
2026-05-11 11:35:09 +03:00
Rustem Kamalov
88337cd7ce
refactor: Speed up Baidu. Fix raw client + reuse HTML parsers for raw search
2026-05-08 00:00:33 +03:00
Rustem Kamalov
471e5ad7fe
refactor: speed up SERP parsing and centralize selectors
2026-05-06 04:12:15 +03:00
Rustem Kamalov
ed23af7acc
feat: add POST /{engine}/parse HTML parser endpoints. One source of truth for browser and raw mode selectors
2026-05-05 05:07:59 +03:00
Rustem Kamalov
3eaa30118e
fix(ecosia): wire proxy policy, validation, raw mode, and OpenAPI docs
2026-05-05 02:47:47 +03:00
Rustem Kamalov
718fad0542
Merge pull request #26 from miku/miku/ecosia
...
Add search engine ecosia
2026-05-04 02:51:08 +03:00
Martin Czygan
3964653e45
Add search engine ecosia
...
Ecosia's search results are [in parts] provided by its own search index,
Staan, developed by the co-founded European Search Perspective (EUSP) --
https://en.wikipedia.org/wiki/Ecosia
2026-05-03 13:56:14 +02:00
Rustem Kamalov
5d48690eb3
feat(mega): add mode fast/any/balanced with CB latency-based fast path
2026-05-02 16:30:56 +03:00
Rustem Kamalov
0864de9887
Rework browser profiles. Add x-use-profile header. Selectors fixes
2026-04-30 23:32:39 +03:00
Rustem Kamalov
063fb74f23
Add browser profile ID in logs and response headers
2026-04-30 20:46:08 +03:00
Rustem Kamalov
4d760fab55
fix: browser request blocking cleanup panic
2026-04-29 05:03:30 +03:00
Rustem Kamalov
f9890ae10f
feat: add network bytes telemetry, mega timeout, and per-engine error details
2026-04-29 02:45:16 +03:00
Rustem Kamalov
e8fb0c24fa
feat(proxy): support X-Proxy-URL via per-process browser pool
2026-04-28 03:46:08 +03:00
Rustem Kamalov
7a0fb21daf
feat: propagate query locale to accept-language header and browser profile
2026-04-27 04:28:53 +03:00
Rustem Kamalov
2b10aca4ac
feat: enhance query handling with locale support and date filtering for Bing, DuckDuckGo, Google, and Yandex
2026-04-27 04:07:28 +03:00
Rustem Kamalov
94ef0b20b4
docs: update readme to match new api format and features, update architecture doc
2026-04-26 03:50:48 +03:00
Rustem Kamalov
d47bf08523
fix api response format issues
2026-04-26 02:58:13 +03:00
Rustem Kamalov
1bdb103b23
docs: rewrite OpenAPI spec to v1 schema
...
Envelopes replace bare arrays on all search endpoints. New schemas for
QueryEcho, ResponseMeta, Pagination, Result (with Position, DomainInfo,
Classification), Cluster, and image variants. ErrorResponse gains reason
field. limit parameter gets min/max bounds. FormatQuery parameter and
X-Request-ID header documented.
2026-04-26 02:58:13 +03:00
Rustem Kamalov
ee42231a09
feat: output format support via ?format= param
...
All search endpoints accept ?format=json|markdown|text|ndjson (default:
json). Accept header negotiation also works (text/markdown, text/plain,
application/x-ndjson). Non-JSON formats skip the response cache to
avoid polluting the JSON cache key. Markdown output is optimised for
n8n Slack/email nodes; text output reduces token count ~25-30% vs JSON
for LLM grounding pipelines; NDJSON emits one result per line.
2026-04-26 02:58:06 +03:00
Rustem Kamalov
16d3a844ad
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
...
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-26 02:57:59 +03:00
Rustem Kamalov
6992ad585d
feat: typed 400 validation errors with stable reason codes
...
Invalid limit/start/boolean params now return 400 bad_request with a
machine-readable reason field (INVALID_LIMIT, INVALID_START,
INVALID_PARAM, EMPTY_QUERY) instead of 500. Limit is validated in
range [1, 100].
2026-04-26 02:57:50 +03:00
Rustem Kamalov
0ccb13c36e
fix(proxy): isolate proxy health from captcha/parser errors, replace re-enable thrash with quarantine
2026-04-24 04:33:41 +03:00
Rustem Kamalov
e4a84d8914
feat(browser): make configurable resource/tracker blocking
2026-04-24 04:02:23 +03:00
Rustem Kamalov
24f9bdab30
perf(browser): smart waits, fix captcha detection selectors + selectors single source of truth
2026-04-23 22:04:28 +03:00
Rustem Kamalov
d052c30bd5
feat(browser): replace stealth with custom profile functionality
2026-04-23 03:06:49 +03:00
Rustem Kamalov
e02b374699
Add fingerprints debug tests and endpoint
2026-04-23 00:53:30 +03:00
Rustem Kamalov
cfd8c418dc
feat(browser): reuse persistent rod connection with isolated browser contexts per request
2026-04-22 22:31:20 +03:00
Rustem Kamalov
5d15335501
Fix CI
2026-04-22 05:32:59 +03:00
Rustem Kamalov
47b5ad3b09
feat: graceful shutdown with drain signal and browser cleanup
2026-04-22 04:25:56 +03:00
