zlei6/openserp
1
0
Fork 0
mirror of https://github.com/karust/openserp.git synced 2026-08-05 16:53:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
119 Commits 4 Branches 16 Tags
dev
Commit Graph

67 Commits

Author SHA1 Message Date
Rustem Kamalov
cec16a08ed Normalize query inputs and cache keys 2026-05-24 02:29:04 +03:00
Rustem Kamalov
25f4ff942b feat(query): add region parameter for per-engine geo targeting (google, yandex, bing, duck) 2026-05-21 20:45:10 +03:00
Rustem Kamalov
82abe57aae Improve SERP result classification and ad positioning 2026-05-19 00:32:11 +03:00
Rustem Kamalov
b3ca4e0803 feat(api): v2 response shape - absolute rank, ad/organic split, slimmer fields 2026-05-13 01:30:54 +03:00
Rustem Kamalov
00630a5d31 refactor: centralize engine prelude, page cleanup, and rate limiter 2026-05-11 11:35:09 +03:00
Rustem Kamalov
88337cd7ce refactor: Speed up Baidu. Fix raw client + reuse HTML parsers for raw search 2026-05-08 00:00:33 +03:00
Rustem Kamalov
471e5ad7fe refactor: speed up SERP parsing and centralize selectors 2026-05-06 04:12:15 +03:00
Rustem Kamalov
ed23af7acc feat: add POST /{engine}/parse HTML parser endpoints. One source of truth for browser and raw mode selectors 2026-05-05 05:07:59 +03:00
Rustem Kamalov
5d48690eb3 feat(mega): add mode fast/any/balanced with CB latency-based fast path 2026-05-02 16:30:56 +03:00
Rustem Kamalov
0864de9887 Rework browser profiles. Add x-use-profile header. Selectors fixes 2026-04-30 23:32:39 +03:00
Rustem Kamalov
063fb74f23 Add browser profile ID in logs and response headers 2026-04-30 20:46:08 +03:00
Rustem Kamalov
4d760fab55 fix: browser request blocking cleanup panic 2026-04-29 05:03:30 +03:00
Rustem Kamalov
f9890ae10f feat: add network bytes telemetry, mega timeout, and per-engine error details 2026-04-29 02:45:16 +03:00
Rustem Kamalov
e8fb0c24fa feat(proxy): support X-Proxy-URL via per-process browser pool 2026-04-28 03:46:08 +03:00
Rustem Kamalov
7a0fb21daf feat: propagate query locale to accept-language header and browser profile 2026-04-27 04:28:53 +03:00
Rustem Kamalov
2b10aca4ac feat: enhance query handling with locale support and date filtering for Bing, DuckDuckGo, Google, and Yandex 2026-04-27 04:07:28 +03:00
Rustem Kamalov
d47bf08523 fix api response format issues 2026-04-26 02:58:13 +03:00
Rustem Kamalov
ee42231a09 feat: output format support via ?format= param 
All search endpoints accept ?format=json|markdown|text|ndjson (default:
json). Accept header negotiation also works (text/markdown, text/plain,
application/x-ndjson). Non-JSON formats skip the response cache to
avoid polluting the JSON cache key. Markdown output is optimised for
n8n Slack/email nodes; text output reduces token count ~25-30% vs JSON
for LLM grounding pipelines; NDJSON emits one result per line.
 2026-04-26 02:58:06 +03:00
Rustem Kamalov
16d3a844ad feat: v1 response envelope, normalized Result struct, clusters, and enrichment 
- All search endpoints now return a JSON envelope with query echo,
  meta (request_id, took_ms, engines_responded, engines_failed, version),
  results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
  id, type, display_url, domain, favicon, is_ad, position, engine_meta,
  domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
  across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
  marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
 2026-04-26 02:57:59 +03:00
Rustem Kamalov
6992ad585d feat: typed 400 validation errors with stable reason codes 
Invalid limit/start/boolean params now return 400 bad_request with a
machine-readable reason field (INVALID_LIMIT, INVALID_START,
INVALID_PARAM, EMPTY_QUERY) instead of 500. Limit is validated in
range [1, 100].
 2026-04-26 02:57:50 +03:00
Rustem Kamalov
0ccb13c36e fix(proxy): isolate proxy health from captcha/parser errors, replace re-enable thrash with quarantine 2026-04-24 04:33:41 +03:00
Rustem Kamalov
e4a84d8914 feat(browser): make configurable resource/tracker blocking 2026-04-24 04:02:23 +03:00
Rustem Kamalov
24f9bdab30 perf(browser): smart waits, fix captcha detection selectors + selectors single source of truth 2026-04-23 22:04:28 +03:00
Rustem Kamalov
d052c30bd5 feat(browser): replace stealth with custom profile functionality 2026-04-23 03:06:49 +03:00
Rustem Kamalov
e02b374699 Add fingerprints debug tests and endpoint 2026-04-23 00:53:30 +03:00
Rustem Kamalov
cfd8c418dc feat(browser): reuse persistent rod connection with isolated browser contexts per request 2026-04-22 22:31:20 +03:00
Rustem Kamalov
5d15335501 Fix CI 2026-04-22 05:32:59 +03:00
Rustem Kamalov
47b5ad3b09 feat: graceful shutdown with drain signal and browser cleanup 2026-04-22 04:25:56 +03:00
Rustem Kamalov
0d4e2da960 Improve logging: add structured logs, enrich logs with useful info. 2026-04-22 03:39:30 +03:00
Rustem Kamalov
d12a301cf8 Enhance error handling and logging across search engines 2026-04-21 03:48:52 +03:00
Rustem Kamalov
3a67241cbe feat: refactor docker setup, use chromedp/headless 2026-04-21 02:25:29 +03:00
Rustem Kamalov
08edec5779 fix(captcha): exit non-zero on misconfig, tighten solver gate tests 2026-04-20 20:12:19 +03:00
Rustem Kamalov
1842c4bb07 feat: implement DrainAndCloseResponse for safe HTTP response handling and update search handlers 2026-04-20 19:39:27 +03:00
Rustem Kamalov
e89b1a3a50 feat: propagate request context across search, retry, limiter, and browser navigation 2026-04-20 05:53:13 +03:00
Rustem Kamalov
0b7fbd3a45 docs: add godoc for public api and readme response examples 2026-04-20 01:20:52 +03:00
Rustem Kamalov
a9986fd8fe docs: add architecture, OpenAPI spec, Swagger UI, and contributing guide 2026-04-15 03:25:06 +03:00
Rustem Kamalov
e5e0c96303 Add URL builder edge-case tests and server endpoint coverage 2026-04-15 02:50:52 +03:00
Rustem Kamalov
3af8612ce5 Fingerprint checks during browser integration tests 2026-04-15 02:36:55 +03:00
Rustem Kamalov
9e22f68c6f Add sanitized real html pages (google,yandex,baidu) for tests. Raw search HTML parser tests. 2026-04-14 00:41:35 +03:00
Rustem Kamalov
3c07c7789a Runtime proxy override 2026-04-01 17:38:13 +03:00
Rustem Kamalov
3daa48e6a3 Proxy policy wiring and fail-closed execution 
- replace proxy wiring with global and per-engine tag policies
- split stats endpoints and lock fail-closed proxy behavior with tests
 2026-04-01 00:54:21 +03:00
Rustem Kamalov
8bec5578c0 Implement proxy support with configuration, pooling, and integration tests 
- Added ProxyConfig and ProxyPool structures to manage proxy settings and rotation.
- Implemented normalization functions for proxy URLs and configurations.
- Created a new ProxyPool that supports failure tracking and round-robin selection.
- Integrated proxy handling into the ResilientSearcher for search queries.
- Added integration tests for various proxy scenarios including SOCKS5 and HTTP proxies.
- Updated server and resilience stats to include proxy information.
- Refactored search functions to utilize the new proxy client.
- Removed direct proxy handling from search_raw.go and yandex/search_raw.go, using the new core proxy client instead.
- Added unit tests for proxy configuration and pool behavior.
 2026-03-30 22:46:03 +03:00
Rustem Kamalov
0146891076 Add bounded TTL cache for dedicated and mega endpoints 2026-03-30 00:31:49 +03:00
Rustem Kamalov
f219c78d84 Add retry/circuit breaker, configurable fallback and CORS. 
Based in part on work from PR #21 by @Sai-Prashanth123, adapted and integrated with project-specific fixes.
 2026-03-25 23:31:41 +03:00
Rustem Kamalov
0fcfc06baa Add healthcheck endpoint 2026-03-25 04:09:37 +03:00
Rustem Kamalov
a4d459761e Add custom browser path support 2026-03-17 02:18:51 +03:00
Rustem Kamalov
7cdd358cad Add filter param to Google. Add start param for supporting engines 2026-03-17 01:54:25 +03:00
Rustem Kamalov
0a347edc77 New megasearch and megaimage functionality. Update readme 2025-10-01 04:34:41 +03:00
Rustem Kamalov
de0a6a80d4 DuckDuck fixes. Non-stealth antibot fixes 2025-10-01 04:30:58 +03:00
Rustem Kamalov
74fc15a6e2 Add DuckDuckGo. Add custom logger. Fix arg priority and other issues 2025-09-30 05:27:54 +03:00