zlei6/openserp
1
0
Fork 0
mirror of https://github.com/karust/openserp.git synced 2026-08-05 16:53:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
157 Commits 4 Branches 16 Tags
main
Commit Graph

93 Commits

Author SHA1 Message Date
Rustem Kamalov
29c7b0fbe0 expose proxy-attempts + use-profile CORS headers; simplify DDG url builders 2026-07-22 15:17:16 +03:00
Rustem Kamalov
2a57bd4248 fix: /extract/batch per-URL errors, param parity, proxy headers. 
Update readme
 2026-07-19 04:18:50 +03:00
Rustem Kamalov
00edd86c1c fix: openapi yaml parse error + /extract/batch timeout exemption 2026-07-18 21:31:46 +03:00
AIWintermuteAI
e430584324 refactor: use errInvalidParam helper and add per-URL took_ms to batch extract 2026-07-18 09:56:20 +02:00
AIWintermuteAI
aec0d9c42a cleanup 2026-07-18 09:49:38 +02:00
AIWintermuteAI
c3934bb75b feat: add POST /extract/batch endpoint for WebUI integration 2026-07-18 00:18:05 +02:00
Rustem Kamalov
e7827cc03d refactor(browser): native CDP fingerprinting, drop patch.js + localized profiles. Update docker chrome 2026-07-13 06:07:37 +03:00
Rustem Kamalov
cf21bf5150 fix google: detect /sorry/ block pages; scope proxy sessions to credentials. Update default config 2026-07-09 19:30:21 +03:00
Rustem Kamalov
9f68086d47 refactor(engines): unify captcha/block/no-results detection across raw and browser paths; Fix google proxy block diagnostics 2026-07-02 00:47:21 +03:00
Rustem Kamalov
3d5eddfeed fix(engines): better classify captcha/blocked/no-results pages instead of returning empty results, also enable it to <engine>/parse endpoints. Enable allow_request_proxy_url by default 2026-07-01 01:38:03 +03:00
Rustem Kamalov
e7af939599 fix: strip features when disabled and normalize feature output. Improve google+bing SERP features extraction 2026-06-29 10:36:46 +03:00
Rustem Kamalov
15bab0295f refactor: extract shared RankState, unify rod/goquery parse paths 2026-06-27 18:04:25 +03:00
Rustem Kamalov
4a81ab774c core/http: adopt tls-client for raw+extract fingerprinting. Update docs 2026-06-27 06:10:11 +03:00
Rustem Kamalov
9cc69da758 feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs 
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
 2026-06-16 03:51:37 +03:00
Rustem Kamalov
04db5347a8 fix(core): harden extract and stabilize request protection 
- breaker: stop counting client cancellations/deadlines (incl. bare
  rate-limiter wait errors) and circuit-open as engine failures
- rate limiting: cache limiters in SearchEngineOptions and rawEngine so
  pacing applies on raw/library paths; pool wrapper delegates
- /extract SSRF guard: public-IP-only policy with dial-time IP pinning,
  redirect re-validation, rendered-mode preflight, http/https allow-list,
  ErrTargetNotAllowed -> HTTP 400, extract.allow_private_networks
  escape hatch (default off)
- browser: no rod Must* on the request path; CLI parses block_resources
  without panicking
 2026-06-12 18:09:40 +03:00
Rustem Kamalov
665f09266a fix: derive a per-request deadline from the retry budget. 
Add `RequestTimeout` config that bounds wall-clock time of any request that does not manage its own deadline budget. It is derived from the engine timeout and retry budget via RequestTimeoutForRetries.
 2026-06-12 04:40:31 +03:00
Rustem Kamalov
d20a7c53cd fix: centralize engine panic recovery in resilient layer 
A rod panic during SearchImage propagated uncaught into fasthttp and killed the whole process: per-engine recover blocks only covered Search (5 of 6 engines had no recovery on SearchImage), and there was no Fiber
recover middleware.

Regression test: a panicking SearchImage returns 502 engine_internal and the server keeps serving.
 2026-06-12 04:24:06 +03:00
Rustem Kamalov
0c7edcd7b6 fix integration tests, dedupe engine smoke tests 2026-06-10 15:42:26 +03:00
Rustem Kamalov
11d02731b6 Fix extraction limits and strategy 2026-06-09 02:28:54 +03:00
Rustem Kamalov
aa397eb2c1 Add new URL extraction endpoint and search feature 2026-06-04 02:22:38 +03:00
Rustem Kamalov
2b4a80fcb2 fix default search limits 2026-06-02 21:44:58 +03:00
Rustem Kamalov
1573642de8 extract region/UULE resolution into dependency-free core/region subpackage 2026-06-02 05:02:51 +03:00
Rustem Kamalov
d0006e30ff dedup feature rendering, align default limit, simplify CSS guard 2026-06-02 04:21:56 +03:00
Rustem Kamalov
327b65869b Fix yandex search limits 2026-06-02 04:01:14 +03:00
Rustem Kamalov
7ed0780192 Add google uule query param support for regional search 2026-06-02 03:54:55 +03:00
Rustem Kamalov
2b96a8330f SERP features: AI summary, answer boxes, related searches/questions, etc - for supporting engines + other fixesa and improvements 2026-05-28 01:36:52 +03:00
Rustem Kamalov
cec16a08ed Normalize query inputs and cache keys 2026-05-24 02:29:04 +03:00
Rustem Kamalov
25f4ff942b feat(query): add region parameter for per-engine geo targeting (google, yandex, bing, duck) 2026-05-21 20:45:10 +03:00
Rustem Kamalov
82abe57aae Improve SERP result classification and ad positioning 2026-05-19 00:32:11 +03:00
Rustem Kamalov
b3ca4e0803 feat(api): v2 response shape - absolute rank, ad/organic split, slimmer fields 2026-05-13 01:30:54 +03:00
Rustem Kamalov
00630a5d31 refactor: centralize engine prelude, page cleanup, and rate limiter 2026-05-11 11:35:09 +03:00
Rustem Kamalov
88337cd7ce refactor: Speed up Baidu. Fix raw client + reuse HTML parsers for raw search 2026-05-08 00:00:33 +03:00
Rustem Kamalov
471e5ad7fe refactor: speed up SERP parsing and centralize selectors 2026-05-06 04:12:15 +03:00
Rustem Kamalov
ed23af7acc feat: add POST /{engine}/parse HTML parser endpoints. One source of truth for browser and raw mode selectors 2026-05-05 05:07:59 +03:00
Rustem Kamalov
5d48690eb3 feat(mega): add mode fast/any/balanced with CB latency-based fast path 2026-05-02 16:30:56 +03:00
Rustem Kamalov
0864de9887 Rework browser profiles. Add x-use-profile header. Selectors fixes 2026-04-30 23:32:39 +03:00
Rustem Kamalov
063fb74f23 Add browser profile ID in logs and response headers 2026-04-30 20:46:08 +03:00
Rustem Kamalov
4d760fab55 fix: browser request blocking cleanup panic 2026-04-29 05:03:30 +03:00
Rustem Kamalov
f9890ae10f feat: add network bytes telemetry, mega timeout, and per-engine error details 2026-04-29 02:45:16 +03:00
Rustem Kamalov
e8fb0c24fa feat(proxy): support X-Proxy-URL via per-process browser pool 2026-04-28 03:46:08 +03:00
Rustem Kamalov
7a0fb21daf feat: propagate query locale to accept-language header and browser profile 2026-04-27 04:28:53 +03:00
Rustem Kamalov
2b10aca4ac feat: enhance query handling with locale support and date filtering for Bing, DuckDuckGo, Google, and Yandex 2026-04-27 04:07:28 +03:00
Rustem Kamalov
d47bf08523 fix api response format issues 2026-04-26 02:58:13 +03:00
Rustem Kamalov
ee42231a09 feat: output format support via ?format= param 
All search endpoints accept ?format=json|markdown|text|ndjson (default:
json). Accept header negotiation also works (text/markdown, text/plain,
application/x-ndjson). Non-JSON formats skip the response cache to
avoid polluting the JSON cache key. Markdown output is optimised for
n8n Slack/email nodes; text output reduces token count ~25-30% vs JSON
for LLM grounding pipelines; NDJSON emits one result per line.
 2026-04-26 02:58:06 +03:00
Rustem Kamalov
16d3a844ad feat: v1 response envelope, normalized Result struct, clusters, and enrichment 
- All search endpoints now return a JSON envelope with query echo,
  meta (request_id, took_ms, engines_responded, engines_failed, version),
  results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
  id, type, display_url, domain, favicon, is_ad, position, engine_meta,
  domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
  across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
  marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
 2026-04-26 02:57:59 +03:00
Rustem Kamalov
6992ad585d feat: typed 400 validation errors with stable reason codes 
Invalid limit/start/boolean params now return 400 bad_request with a
machine-readable reason field (INVALID_LIMIT, INVALID_START,
INVALID_PARAM, EMPTY_QUERY) instead of 500. Limit is validated in
range [1, 100].
 2026-04-26 02:57:50 +03:00
Rustem Kamalov
0ccb13c36e fix(proxy): isolate proxy health from captcha/parser errors, replace re-enable thrash with quarantine 2026-04-24 04:33:41 +03:00
Rustem Kamalov
e4a84d8914 feat(browser): make configurable resource/tracker blocking 2026-04-24 04:02:23 +03:00
Rustem Kamalov
24f9bdab30 perf(browser): smart waits, fix captcha detection selectors + selectors single source of truth 2026-04-23 22:04:28 +03:00
Rustem Kamalov
d052c30bd5 feat(browser): replace stealth with custom profile functionality 2026-04-23 03:06:49 +03:00