Rustem Kamalov
e7af939599
fix: strip features when disabled and normalize feature output. Improve google+bing SERP features extraction
2026-06-29 10:36:46 +03:00
Rustem Kamalov
15bab0295f
refactor: extract shared RankState, unify rod/goquery parse paths
2026-06-27 18:04:25 +03:00
Rustem Kamalov
4a81ab774c
core/http: adopt tls-client for raw+extract fingerprinting. Update docs
2026-06-27 06:10:11 +03:00
Rustem Kamalov
ba27fcf1f4
remove dead code, pin golangci linters
2026-06-25 17:06:01 +03:00
Rustem Kamalov
9cc69da758
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
...
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
2026-06-16 03:51:37 +03:00
Rustem Kamalov
ca3143ccd1
Merge pull request #32 from karust/stability-safety-improvements
...
Stability and safety improvements
2026-06-12 19:14:29 +03:00
Rustem Kamalov
976ceeadf4
fix: report ErrSearchTimeout when results never hydrate; raise integration selector timeout to 15s
2026-06-12 19:06:46 +03:00
Rustem Kamalov
04db5347a8
fix(core): harden extract and stabilize request protection
...
- breaker: stop counting client cancellations/deadlines (incl. bare
rate-limiter wait errors) and circuit-open as engine failures
- rate limiting: cache limiters in SearchEngineOptions and rawEngine so
pacing applies on raw/library paths; pool wrapper delegates
- /extract SSRF guard: public-IP-only policy with dial-time IP pinning,
redirect re-validation, rendered-mode preflight, http/https allow-list,
ErrTargetNotAllowed -> HTTP 400, extract.allow_private_networks
escape hatch (default off)
- browser: no rod Must* on the request path; CLI parses block_resources
without panicking
2026-06-12 18:09:40 +03:00
Rustem Kamalov
665f09266a
fix: derive a per-request deadline from the retry budget.
...
Add `RequestTimeout` config that bounds wall-clock time of any request that does not manage its own deadline budget. It is derived from the engine timeout and retry budget via RequestTimeoutForRetries.
2026-06-12 04:40:31 +03:00
Rustem Kamalov
d20a7c53cd
fix: centralize engine panic recovery in resilient layer
...
A rod panic during SearchImage propagated uncaught into fasthttp and killed the whole process: per-engine recover blocks only covered Search (5 of 6 engines had no recovery on SearchImage), and there was no Fiber
recover middleware.
Regression test: a panicking SearchImage returns 502 engine_internal and the server keeps serving.
2026-06-12 04:24:06 +03:00
Rustem Kamalov
c9e548eb3a
Add release build job
2026-06-12 04:00:12 +03:00
Rustem Kamalov
59d076616c
Add docker image build job
2026-06-12 03:29:57 +03:00
Rustem Kamalov
0c7edcd7b6
fix integration tests, dedupe engine smoke tests
2026-06-10 15:42:26 +03:00
Rustem Kamalov
45287112df
fix docs + google: gate organic results by matched selector, harden answer-box parse
2026-06-10 15:15:42 +03:00
Rustem Kamalov
b31e84316c
Add contributor onboarding kit
2026-06-10 14:54:45 +03:00
Rustem Kamalov
ffd3d4250c
Add OpenSERP SDK usage examples
2026-06-09 04:14:35 +03:00
Rustem Kamalov
11d02731b6
Fix extraction limits and strategy
2026-06-09 02:28:54 +03:00
Rustem Kamalov
5be448e220
Add client SDKs and integrations
2026-06-05 19:48:27 +03:00
Rustem Kamalov
aa397eb2c1
Add new URL extraction endpoint and search feature
2026-06-04 02:22:38 +03:00
Rustem Kamalov
2b4a80fcb2
fix default search limits
2026-06-02 21:44:58 +03:00
Rustem Kamalov
1573642de8
extract region/UULE resolution into dependency-free core/region subpackage
2026-06-02 05:02:51 +03:00
Rustem Kamalov
d0006e30ff
dedup feature rendering, align default limit, simplify CSS guard
2026-06-02 04:21:56 +03:00
Rustem Kamalov
327b65869b
Fix yandex search limits
2026-06-02 04:01:14 +03:00
Rustem Kamalov
7ed0780192
Add google uule query param support for regional search
2026-06-02 03:54:55 +03:00
Rustem Kamalov
2b96a8330f
SERP features: AI summary, answer boxes, related searches/questions, etc - for supporting engines + other fixesa and improvements
2026-05-28 01:36:52 +03:00
Rustem Kamalov
cec16a08ed
Normalize query inputs and cache keys
2026-05-24 02:29:04 +03:00
Rustem Kamalov
25f4ff942b
feat(query): add region parameter for per-engine geo targeting (google, yandex, bing, duck)
2026-05-21 20:45:10 +03:00
Rustem Kamalov
82abe57aae
Improve SERP result classification and ad positioning
2026-05-19 00:32:11 +03:00
Rustem Kamalov
b3ca4e0803
feat(api): v2 response shape - absolute rank, ad/organic split, slimmer fields
2026-05-13 01:30:54 +03:00
Rustem Kamalov
00630a5d31
refactor: centralize engine prelude, page cleanup, and rate limiter
2026-05-11 11:35:09 +03:00
Rustem Kamalov
88337cd7ce
refactor: Speed up Baidu. Fix raw client + reuse HTML parsers for raw search
2026-05-08 00:00:33 +03:00
Rustem Kamalov
471e5ad7fe
refactor: speed up SERP parsing and centralize selectors
2026-05-06 04:12:15 +03:00
Rustem Kamalov
ed23af7acc
feat: add POST /{engine}/parse HTML parser endpoints. One source of truth for browser and raw mode selectors
2026-05-05 05:07:59 +03:00
Rustem Kamalov
3eaa30118e
fix(ecosia): wire proxy policy, validation, raw mode, and OpenAPI docs
2026-05-05 02:47:47 +03:00
Rustem Kamalov
718fad0542
Merge pull request #26 from miku/miku/ecosia
...
Add search engine ecosia
2026-05-04 02:51:08 +03:00
Martin Czygan
3964653e45
Add search engine ecosia
...
Ecosia's search results are [in parts] provided by its own search index,
Staan, developed by the co-founded European Search Perspective (EUSP) --
https://en.wikipedia.org/wiki/Ecosia
2026-05-03 13:56:14 +02:00
Rustem Kamalov
5d48690eb3
feat(mega): add mode fast/any/balanced with CB latency-based fast path
2026-05-02 16:30:56 +03:00
Rustem Kamalov
0864de9887
Rework browser profiles. Add x-use-profile header. Selectors fixes
2026-04-30 23:32:39 +03:00
Rustem Kamalov
063fb74f23
Add browser profile ID in logs and response headers
2026-04-30 20:46:08 +03:00
Rustem Kamalov
4d760fab55
fix: browser request blocking cleanup panic
2026-04-29 05:03:30 +03:00
Rustem Kamalov
f9890ae10f
feat: add network bytes telemetry, mega timeout, and per-engine error details
2026-04-29 02:45:16 +03:00
Rustem Kamalov
e8fb0c24fa
feat(proxy): support X-Proxy-URL via per-process browser pool
2026-04-28 03:46:08 +03:00
Rustem Kamalov
7a0fb21daf
feat: propagate query locale to accept-language header and browser profile
2026-04-27 04:28:53 +03:00
Rustem Kamalov
2b10aca4ac
feat: enhance query handling with locale support and date filtering for Bing, DuckDuckGo, Google, and Yandex
2026-04-27 04:07:28 +03:00
Rustem Kamalov
94ef0b20b4
docs: update readme to match new api format and features, update architecture doc
2026-04-26 03:50:48 +03:00
Rustem Kamalov
d47bf08523
fix api response format issues
2026-04-26 02:58:13 +03:00
Rustem Kamalov
1bdb103b23
docs: rewrite OpenAPI spec to v1 schema
...
Envelopes replace bare arrays on all search endpoints. New schemas for
QueryEcho, ResponseMeta, Pagination, Result (with Position, DomainInfo,
Classification), Cluster, and image variants. ErrorResponse gains reason
field. limit parameter gets min/max bounds. FormatQuery parameter and
X-Request-ID header documented.
2026-04-26 02:58:13 +03:00
Rustem Kamalov
ee42231a09
feat: output format support via ?format= param
...
All search endpoints accept ?format=json|markdown|text|ndjson (default:
json). Accept header negotiation also works (text/markdown, text/plain,
application/x-ndjson). Non-JSON formats skip the response cache to
avoid polluting the JSON cache key. Markdown output is optimised for
n8n Slack/email nodes; text output reduces token count ~25-30% vs JSON
for LLM grounding pipelines; NDJSON emits one result per line.
2026-04-26 02:58:06 +03:00
Rustem Kamalov
16d3a844ad
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
...
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-26 02:57:59 +03:00
Rustem Kamalov
6992ad585d
feat: typed 400 validation errors with stable reason codes
...
Invalid limit/start/boolean params now return 400 bad_request with a
machine-readable reason field (INVALID_LIMIT, INVALID_START,
INVALID_PARAM, EMPTY_QUERY) instead of 500. Limit is validated in
range [1, 100].
2026-04-26 02:57:50 +03:00
