zlei6/openserp
1
0
Fork 0
mirror of https://github.com/karust/openserp.git synced 2026-08-05 16:53:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
105 Commits 4 Branches 16 Tags
4d760fab55214d209e5dbca497f4cbac175f86ce
Commit Graph

105 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rustem Kamalov
4d760fab55 fix: browser request blocking cleanup panic 2026-04-29 05:03:30 +03:00
Rustem Kamalov
f9890ae10f feat: add network bytes telemetry, mega timeout, and per-engine error details 2026-04-29 02:45:16 +03:00
Rustem Kamalov
e8fb0c24fa feat(proxy): support X-Proxy-URL via per-process browser pool 2026-04-28 03:46:08 +03:00
Rustem Kamalov
7a0fb21daf feat: propagate query locale to accept-language header and browser profile v0.7.2 2026-04-27 04:28:53 +03:00
Rustem Kamalov
2b10aca4ac feat: enhance query handling with locale support and date filtering for Bing, DuckDuckGo, Google, and Yandex 2026-04-27 04:07:28 +03:00
Rustem Kamalov
94ef0b20b4 docs: update readme to match new api format and features, update architecture doc 2026-04-26 03:50:48 +03:00
Rustem Kamalov
d47bf08523 fix api response format issues 2026-04-26 02:58:13 +03:00
Rustem Kamalov
1bdb103b23 docs: rewrite OpenAPI spec to v1 schema 
Envelopes replace bare arrays on all search endpoints. New schemas for
QueryEcho, ResponseMeta, Pagination, Result (with Position, DomainInfo,
Classification), Cluster, and image variants. ErrorResponse gains reason
field. limit parameter gets min/max bounds. FormatQuery parameter and
X-Request-ID header documented.
 2026-04-26 02:58:13 +03:00
Rustem Kamalov
ee42231a09 feat: output format support via ?format= param 
All search endpoints accept ?format=json|markdown|text|ndjson (default:
json). Accept header negotiation also works (text/markdown, text/plain,
application/x-ndjson). Non-JSON formats skip the response cache to
avoid polluting the JSON cache key. Markdown output is optimised for
n8n Slack/email nodes; text output reduces token count ~25-30% vs JSON
for LLM grounding pipelines; NDJSON emits one result per line.
 2026-04-26 02:58:06 +03:00
Rustem Kamalov
16d3a844ad feat: v1 response envelope, normalized Result struct, clusters, and enrichment 
- All search endpoints now return a JSON envelope with query echo,
  meta (request_id, took_ms, engines_responded, engines_failed, version),
  results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
  id, type, display_url, domain, favicon, is_ad, position, engine_meta,
  domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
  across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
  marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
 2026-04-26 02:57:59 +03:00
Rustem Kamalov
6992ad585d feat: typed 400 validation errors with stable reason codes 
Invalid limit/start/boolean params now return 400 bad_request with a
machine-readable reason field (INVALID_LIMIT, INVALID_START,
INVALID_PARAM, EMPTY_QUERY) instead of 500. Limit is validated in
range [1, 100].
 2026-04-26 02:57:50 +03:00
Rustem Kamalov
0ccb13c36e fix(proxy): isolate proxy health from captcha/parser errors, replace re-enable thrash with quarantine 2026-04-24 04:33:41 +03:00
Rustem Kamalov
e4a84d8914 feat(browser): make configurable resource/tracker blocking 2026-04-24 04:02:23 +03:00
Rustem Kamalov
24f9bdab30 perf(browser): smart waits, fix captcha detection selectors + selectors single source of truth 2026-04-23 22:04:28 +03:00
Rustem Kamalov
d052c30bd5 feat(browser): replace stealth with custom profile functionality 2026-04-23 03:06:49 +03:00
Rustem Kamalov
e02b374699 Add fingerprints debug tests and endpoint 2026-04-23 00:53:30 +03:00
Rustem Kamalov
cfd8c418dc feat(browser): reuse persistent rod connection with isolated browser contexts per request 2026-04-22 22:31:20 +03:00
Rustem Kamalov
5d15335501 Fix CI 2026-04-22 05:32:59 +03:00
Rustem Kamalov
47b5ad3b09 feat: graceful shutdown with drain signal and browser cleanup 2026-04-22 04:25:56 +03:00
Rustem Kamalov
0d4e2da960 Improve logging: add structured logs, enrich logs with useful info. 2026-04-22 03:39:30 +03:00
Rustem Kamalov
d12a301cf8 Enhance error handling and logging across search engines 2026-04-21 03:48:52 +03:00
Rustem Kamalov
3a67241cbe feat: refactor docker setup, use chromedp/headless 2026-04-21 02:25:29 +03:00
Rustem Kamalov
08edec5779 fix(captcha): exit non-zero on misconfig, tighten solver gate tests 2026-04-20 20:12:19 +03:00
Rustem Kamalov
1842c4bb07 feat: implement DrainAndCloseResponse for safe HTTP response handling and update search handlers 2026-04-20 19:39:27 +03:00
Rustem Kamalov
e89b1a3a50 feat: propagate request context across search, retry, limiter, and browser navigation 2026-04-20 05:53:13 +03:00
Rustem Kamalov
0b7fbd3a45 docs: add godoc for public api and readme response examples 2026-04-20 01:20:52 +03:00
Rustem Kamalov
a9986fd8fe docs: add architecture, OpenAPI spec, Swagger UI, and contributing guide 2026-04-15 03:25:06 +03:00
Rustem Kamalov
e1142ac23b Improve tests and error handling for search engines 
Improve tests and error handling for search engines
 2026-04-15 02:53:11 +03:00
Rustem Kamalov
e5e0c96303 Add URL builder edge-case tests and server endpoint coverage 2026-04-15 02:50:52 +03:00
Rustem Kamalov
3af8612ce5 Fingerprint checks during browser integration tests 2026-04-15 02:36:55 +03:00
Rustem Kamalov
ecacfab4d6 Add integration tests for multiple search engines and enhance error handling 
- Implement integration tests for Baidu, Bing, DuckDuckGo, Google, and Yandex.
- Refactor test utility functions to avoid core import cycles.
 2026-04-14 02:05:58 +03:00
Rustem Kamalov
de59d07d47 Fix linter errors. Enhance error handling in search functions across multiple engines 2026-04-14 01:12:17 +03:00
Rustem Kamalov
bc59de46fb Add CI and integration workflows 2026-04-14 01:04:32 +03:00
Rustem Kamalov
9e22f68c6f Add sanitized real html pages (google,yandex,baidu) for tests. Raw search HTML parser tests. 2026-04-14 00:41:35 +03:00
Rustem Kamalov
518ccdbdbd Revise README for clarity and project details 
Updated project description and clarified affiliation.
 2026-04-07 22:44:20 +03:00
Rustem Kamalov
f4c6bf128d Clarify OpenSerp's open-source status and affiliations 
Added clarification about OpenSerp's open-source status and third-party affiliations.
 2026-04-07 18:34:12 +03:00
Rustem Kamalov
10decd5740 Fix config argument, update deault config. v0.6.0 2026-04-04 19:32:23 +03:00
Rustem Kamalov
78a9f1e264 Merge pull request #22 from karust/dev 
Add runtime proxy override, update readme
 2026-04-04 18:57:56 +03:00
Rustem Kamalov
9f84a648f4 Update config and readme 2026-04-04 18:54:10 +03:00
Rustem Kamalov
3c07c7789a Runtime proxy override 2026-04-01 17:38:13 +03:00
Rustem Kamalov
3daa48e6a3 Proxy policy wiring and fail-closed execution 
- replace proxy wiring with global and per-engine tag policies
- split stats endpoints and lock fail-closed proxy behavior with tests
 2026-04-01 00:54:21 +03:00
Rustem Kamalov
8bec5578c0 Implement proxy support with configuration, pooling, and integration tests 
- Added ProxyConfig and ProxyPool structures to manage proxy settings and rotation.
- Implemented normalization functions for proxy URLs and configurations.
- Created a new ProxyPool that supports failure tracking and round-robin selection.
- Integrated proxy handling into the ResilientSearcher for search queries.
- Added integration tests for various proxy scenarios including SOCKS5 and HTTP proxies.
- Updated server and resilience stats to include proxy information.
- Refactored search functions to utilize the new proxy client.
- Removed direct proxy handling from search_raw.go and yandex/search_raw.go, using the new core proxy client instead.
- Added unit tests for proxy configuration and pool behavior.
 2026-03-30 22:46:03 +03:00
Rustem Kamalov
0146891076 Add bounded TTL cache for dedicated and mega endpoints 2026-03-30 00:31:49 +03:00
Rustem Kamalov
f219c78d84 Add retry/circuit breaker, configurable fallback and CORS. 
Based in part on work from PR #21 by @Sai-Prashanth123, adapted and integrated with project-specific fixes.
 2026-03-25 23:31:41 +03:00
Rustem Kamalov
0fcfc06baa Add healthcheck endpoint 2026-03-25 04:09:37 +03:00
Rustem Kamalov
a4d459761e Add custom browser path support 2026-03-17 02:18:51 +03:00
Rustem Kamalov
7cdd358cad Add filter param to Google. Add start param for supporting engines 2026-03-17 01:54:25 +03:00
Rustem Kamalov
6c8e205af2 Fix duckduck date and limit filtering. Deduplicate results v0.5.4 2025-10-08 03:33:48 +03:00
Rustem Kamalov
b4ef23a913 Add megasearch API response example v0.5.3 2025-10-01 04:56:40 +03:00
Rustem Kamalov
0a347edc77 New megasearch and megaimage functionality. Update readme 2025-10-01 04:34:41 +03:00