zlei6/openserp
1
0
Fork 0
mirror of https://github.com/karust/openserp.git synced 2026-08-17 21:58:18 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs

Browse Source 
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
This commit is contained in:
Rustem Kamalov
2026-06-16 03:51:37 +03:00
parent ca3143ccd1
commit 9cc69da758
33 changed files with 1613 additions and 383 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
ecosia/testdata/search_captcha.html vendored Normal file
View File

@@ -0,0 +1 @@
<html xmlns="http://test.test" dir="ltr"><head><title>Just a moment...</title></head><body><main><h1>Confirm you’re not a robot</h1><p>Our system has detected unusual traffic from your network. Please solve the challenge below to show you’re not a robot.</p><div><div class="main-wrapper" role="main"><div class="main-content"><h2 id="FbRG5"></h2><p id="jpqrL2"></p><div id="BbLB6" style=""><div><div><input type="hidden" name="cf-turnstile-response" id="id_1"/></div></div></div><div id="TQpKs1" style=""><div>Verification successful. Waiting for www.ecosia.org to respond</div></div><div id="ROlTq4" class="spacer loading-verifying" style=""><div class="lds-ring"><div></div><div></div><div></div><div></div></div></div></div></div></div></main></body></html>