diff --git a/README.md b/README.md index aece1ad..73a1184 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,6 @@ # OpenSERP -[![Go Report Card](https://goreportcard.com/badge/github.com/karust/openserp)](https://goreportcard.com/report/github.com/karust/openserp) [![Go Reference](https://pkg.go.dev/badge/github/karust/openserp?style=for-the-badge)](https://pkg.go.dev/github.com/karust/openserp) [![release](https://img.shields.io/github/v/release/karust/openserp)](https://github.com/karust/openserp/releases) [![Docker Pulls](https://img.shields.io/docker/v/karust/openserp)](https://hub.docker.com/r/karust/openserp) @@ -12,7 +11,7 @@ Use it as a search tool for **LLMs, agents, and RAG pipelines**, or as a scraper backend for **SEO rank tracking across Google, Yandex, Baidu, and more**. It is especially useful when your workflow needs RU/CN web coverage instead of another Google-only API. -Run it locally, self-host it, or use [cloud version](https://openserp.org/cloud) when you want the same public API shape without operating the server. +Run it locally, self-host it, or use the [cloud version](https://openserp.org/cloud) when you want the same public API shape without operating the server. Cloud is also the simplest way to support OpenSERP: it is where the technology gets tested at scale, and that work flows straight back into this open-source project. ## Features @@ -166,9 +165,11 @@ curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust& ## Deployment Options - **Self-hosted (this repo)** - free, MIT-licensed, with full control over runtime, proxies, cache, and scaling. -- **[OpenSERP Cloud](https://openserp.org/cloud)** - optional managed version from the project maintainers, with the same API shape. +- **[OpenSERP Cloud](https://openserp.org/cloud)** - the same public API, run and maintained for you by the people who build OpenSERP. -The hosted API helps fund continued development of the open-source project. Same endpoints, same response schema, and client code can migrate either direction. +Same endpoints, same response schema, and client code migrates either direction, so you are never locked in. + +If OpenSERP is useful to you and you would rather not run the infrastructure yourself, using Cloud (or topping up an account) is a direct way to support the project. It is where the technology gets exercised against real traffic at scale, which is how the engines get hardened, new features get tested, and the open-source core keeps improving. Every request you send through Cloud helps fund and shape what lands back in this repository. ## API Docs @@ -183,12 +184,12 @@ To browse the spec without running the server, see [docs/openapi.yaml](./docs/op Official client packages. Each works against your self-hosted server (set `baseUrl`) or the [hosted API](https://openserp.org/cloud) (set `apiKey`): -| Type | Package | Install | -| --------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------- | -| JavaScript / TypeScript SDK | [`@openserp/sdk`](https://www.npmjs.com/package/@openserp/sdk) | `npm install @openserp/sdk` | -| Python SDK | [`openserp`](https://pypi.org/project/openserp/) | `pip install openserp` | -| MCP server (AI agents) | [`@openserp/mcp`](https://www.npmjs.com/package/@openserp/mcp) | `npx @openserp/mcp` | -| n8n community node | [`@openserp/n8n-nodes-openserp`](https://www.npmjs.com/package/@openserp/n8n-nodes-openserp) | Install via n8n community nodes | +| Type | Package | Source | Install | +| --------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------- | +| JavaScript / TypeScript SDK | [`@openserp/sdk`](https://www.npmjs.com/package/@openserp/sdk) | [openserpapi/sdk-js](https://github.com/openserpapi/sdk-js) | `npm install @openserp/sdk` | +| Python SDK | [`openserp`](https://pypi.org/project/openserp/) | [openserpapi/sdk-python](https://github.com/openserpapi/sdk-python) | `pip install openserp` | +| MCP server (AI agents) | [`@openserp/mcp`](https://www.npmjs.com/package/@openserp/mcp) | [openserpapi/mcp](https://github.com/openserpapi/mcp) | `npx @openserp/mcp` | +| n8n community node | [`@openserp/n8n-nodes-openserp`](https://www.npmjs.com/package/@openserp/n8n-nodes-openserp) | [openserpapi/n8n](https://github.com/openserpapi/n8n) | Install via n8n community nodes | See [**examples**](./examples) for small JavaScript and Python use cases covering search, AI grounding, SEO, content extraction, and image search. @@ -515,6 +516,6 @@ Contributions are welcome. See [docs/CONTRIBUTING.md](./docs/CONTRIBUTING.md). - [GitHub Issues](https://github.com/karust/openserp/issues) - bugs, feature ideas, and reproducible issues. - [feedback@openserp.org](mailto:feedback@openserp.org) - private notes, longer feedback, or anything that does not fit GitHub Issues. -- [Telegram Channel](https://t.me/+RJEKspw3mUlhZDMy) - OpenSERP news, release notes, and project updates. Direct messages are open for quick feedback and hosted API questions. +- [Telegram Channel](https://t.me/+RJEKspw3mUlhZDMy) - OpenSERP news, release notes, and project updates. > OpenSERP is free and open-source. Only links listed in this repository and on [openserp.org](https://openserp.org) are associated with the project. diff --git a/cmd/root.go b/cmd/root.go index 3d82b63..6cf7754 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -17,7 +17,7 @@ import ( ) const ( - version = "0.8.8" + version = "0.8.9" defaultConfigFilename = "config" envPrefix = "OPENSERP" ) diff --git a/cmd/serve.go b/cmd/serve.go index d39f385..24cfd0a 100644 --- a/cmd/serve.go +++ b/cmd/serve.go @@ -2,6 +2,8 @@ package cmd import ( "context" + "crypto/sha256" + "encoding/hex" "errors" "fmt" "io" @@ -289,12 +291,12 @@ func newBrowserPool(base core.BrowserOpts, defaultLaunchProxyURL string, laneSto stopSweeper: make(chan struct{}), sweeperDone: make(chan struct{}), } - // A configured global proxy (legacy) becomes a pre-bound entry on the - // shared "direct" key so requests without a per-request proxy still use it. if launchURL := strings.TrimSpace(defaultLaunchProxyURL); launchURL != "" { - pool.browsers[directBrowserKey] = &pooledBrowser{ - launchProxyURL: launchURL, - lastUsedAt: time.Now(), + if key := browserPoolKey(launchURL); key != directBrowserKey { + pool.browsers[key] = &pooledBrowser{ + launchProxyURL: launchURL, + lastUsedAt: time.Now(), + } } } if idleTTL > 0 { @@ -306,7 +308,8 @@ func newBrowserPool(base core.BrowserOpts, defaultLaunchProxyURL string, laneSto } // browserPoolKey derives the pool key from a request's proxy URL. Authenticated -// HTTP/HTTPS proxies get their own Chrome keyed by scheme+host+port+username. +// HTTP/HTTPS proxies get their own Chrome keyed by scheme+host+port+username +// plus a short auth hash, so provider session tokens do not share one Chrome. // Empty/unauthenticated/SOCKS request URLs fall through to the shared // "direct" Chrome. func browserPoolKey(requestProxyURL string) string { @@ -330,8 +333,10 @@ func browserPoolKey(requestProxyURL string) string { if parsed.User == nil { return directBrowserKey } - username := parsed.User.Username() - return fmt.Sprintf("%s|%s|%s", parsed.Scheme, parsed.Host, username) + // Hash the full userinfo so a rotating password gets its own Chrome without + // leaking credentials into the key. Scheme+host+username stay readable. + sum := sha256.Sum256([]byte(parsed.User.String())) + return fmt.Sprintf("%s|%s|%s|%s", parsed.Scheme, parsed.Host, parsed.User.Username(), hex.EncodeToString(sum[:])[:16]) } // browserLaunchURL returns the URL to pass to launcher.Proxy for a given diff --git a/cmd/serve_test.go b/cmd/serve_test.go index 65c3e42..8f6db7e 100644 --- a/cmd/serve_test.go +++ b/cmd/serve_test.go @@ -30,28 +30,44 @@ func TestCommandDefaultsToQuiet(t *testing.T) { func TestBrowserPoolKey(t *testing.T) { cases := []struct { - name string - raw string - want string + name string + raw string + wantDirect bool + wantPrefix string }{ - {"empty -> direct", "", directBrowserKey}, - {"unauth http -> direct", "http://proxy.example:8080", directBrowserKey}, - {"unauth socks -> direct", "socks5://proxy.example:1080", directBrowserKey}, - {"auth socks -> direct (rejected upstream)", "socks5://user:pass@proxy.example:1080", directBrowserKey}, - {"auth http", "http://user:pass@proxy.example:8080", "http|proxy.example:8080|user"}, - {"auth https different scheme", "https://user:pass@proxy.example:8443", "https|proxy.example:8443|user"}, - {"different password same key", "http://user:other-pass@proxy.example:8080", "http|proxy.example:8080|user"}, - {"different user different key", "http://user2:pass@proxy.example:8080", "http|proxy.example:8080|user2"}, - {"different host different key", "http://user:pass@proxy2.example:8080", "http|proxy2.example:8080|user"}, - {"different port different key", "http://user:pass@proxy.example:9090", "http|proxy.example:9090|user"}, + {"empty -> direct", "", true, ""}, + {"unauth http -> direct", "http://proxy.example:8080", true, ""}, + {"unauth socks -> direct", "socks5://proxy.example:1080", true, ""}, + {"auth socks -> direct (rejected upstream)", "socks5://user:pass@proxy.example:1080", true, ""}, + {"auth http", "http://user:pass@proxy.example:8080", false, "http|proxy.example:8080|user|"}, + {"auth https different scheme", "https://user:pass@proxy.example:8443", false, "https|proxy.example:8443|user|"}, + {"different user different key", "http://user2:pass@proxy.example:8080", false, "http|proxy.example:8080|user2|"}, + {"different host different key", "http://user:pass@proxy2.example:8080", false, "http|proxy2.example:8080|user|"}, + {"different port different key", "http://user:pass@proxy.example:9090", false, "http|proxy.example:9090|user|"}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - if got := browserPoolKey(tc.raw); got != tc.want { - t.Fatalf("browserPoolKey(%q) = %q, want %q", tc.raw, got, tc.want) + got := browserPoolKey(tc.raw) + if tc.wantDirect { + if got != directBrowserKey { + t.Fatalf("browserPoolKey(%q) = %q, want %q", tc.raw, got, directBrowserKey) + } + return + } + if !strings.HasPrefix(got, tc.wantPrefix) { + t.Fatalf("browserPoolKey(%q) = %q, want prefix %q", tc.raw, got, tc.wantPrefix) + } + if strings.Contains(got, "pass") { + t.Fatalf("browser pool key leaked password material: %q", got) } }) } + + first := browserPoolKey("http://user:pass@proxy.example:8080") + second := browserPoolKey("http://user:other-pass@proxy.example:8080") + if first == second { + t.Fatalf("expected different passwords to produce different browser pool keys: %q", first) + } } func TestBrowserLaunchURL(t *testing.T) { @@ -76,6 +92,28 @@ func TestBrowserLaunchURL(t *testing.T) { } } +func TestBrowserPoolGlobalProxyDoesNotOccupyDirectSlot(t *testing.T) { + globalProxy := "http://user:pass@proxy.example:8080" + pool := newBrowserPool(core.BrowserOpts{}, globalProxy, nil, 2, 0) + defer func() { + if err := pool.close(); err != nil { + t.Fatalf("close pool: %v", err) + } + }() + + if _, ok := pool.browsers[directBrowserKey]; ok { + t.Fatalf("global proxy must not occupy %q browser slot", directBrowserKey) + } + key := browserPoolKey(globalProxy) + entry, ok := pool.browsers[key] + if !ok { + t.Fatalf("expected global proxy slot %q to be pre-bound", key) + } + if entry.launchProxyURL != globalProxy { + t.Fatalf("expected launch proxy %q, got %q", globalProxy, entry.launchProxyURL) + } +} + func TestBrowserPoolEvictLRU(t *testing.T) { // Pre-populate with bare entries (browser=nil) so we exercise eviction // without launching real Chrome. closePooledBrowser handles nil safely. diff --git a/config.yaml b/config.yaml index 1f6ad8d..19e4838 100644 --- a/config.yaml +++ b/config.yaml @@ -14,6 +14,8 @@ app: head: false # Headful mode leakless: false # Force browser process cleanup after request leave_head: false # Keep tabs open after request + block_resources: "image,font,css,media" # Block heavy subresources in browser mode + block_trackers: true # Block known tracker domains max_processes: 6 # Concurrent Chrome processes idle_ttl: 5m # close a Chrome that has not served traffic for this long diff --git a/core/proxy_lane.go b/core/proxy_lane.go index f726378..6e7f7f1 100644 --- a/core/proxy_lane.go +++ b/core/proxy_lane.go @@ -236,11 +236,11 @@ func proxyLaneIDFromProxyURL(raw string) string { if err != nil { return "" } - username := "" + userInfo := "" if parsed.User != nil { - username = parsed.User.Username() + userInfo = parsed.User.String() } - sum := sha256.Sum256([]byte(parsed.Host + "|" + username)) + sum := sha256.Sum256([]byte(parsed.Scheme + "|" + parsed.Host + "|" + userInfo)) return hex.EncodeToString(sum[:])[:16] } diff --git a/core/proxy_lane_test.go b/core/proxy_lane_test.go index 69e3b60..b544f62 100644 --- a/core/proxy_lane_test.go +++ b/core/proxy_lane_test.go @@ -1,6 +1,7 @@ package core import ( + "strings" "testing" "time" @@ -77,14 +78,17 @@ func TestLaneStoreEvictsLRU(t *testing.T) { } } -func TestProxyLaneKeyForOmitsPassword(t *testing.T) { +func TestProxyLaneKeyForUsesCredentialIdentity(t *testing.T) { a := ProxyLaneKeyForTenant("Google", "", Query{}, "http://user:pass-a@proxy.example:8080") b := ProxyLaneKeyForTenant("google", "", Query{}, "http://user:pass-b@proxy.example:8080") if a.Empty() || b.Empty() { t.Fatalf("expected derived lane keys, got %#v %#v", a, b) } - if a != b { - t.Fatalf("expected password changes not to affect lane key: %#v %#v", a, b) + if a == b { + t.Fatalf("expected password changes to produce separate lane keys: %#v", a) + } + if strings.Contains(a.ID(), "pass-a") || strings.Contains(b.ID(), "pass-b") { + t.Fatalf("lane key leaked password material: %#v %#v", a, b) } } diff --git a/google/captcha_selector_test.go b/google/captcha_selector_test.go index 4cdd0a6..5e33f5a 100644 --- a/google/captcha_selector_test.go +++ b/google/captcha_selector_test.go @@ -25,6 +25,9 @@ func TestGooglePageTypeSelectors(t *testing.T) { {"search_captcha_new.html", Selectors.CaptchaPage, true}, {"search_captcha_new.html", Selectors.ResultStats, false}, + {"search_sorry.html", Selectors.CaptchaPage, true}, + {"search_sorry.html", Selectors.ResultStats, false}, + {"search_soft_block.html", Selectors.SoftBlock, true}, {"search_soft_block.html", Selectors.Captcha, false}, {"search_soft_block.html", Selectors.CaptchaPage, false}, @@ -60,7 +63,7 @@ func assertSelector(t *testing.T, fixture, selector string, wantHit bool) { got := doc.Find(selector).Length() > 0 if got != wantHit { if wantHit { - t.Fatalf("selector %q not found in %s — update selectors.go", selector, fixture) + t.Fatalf("selector %q not found in %s - update selectors.go", selector, fixture) } else { t.Fatalf("selector %q unexpectedly present in %s", selector, fixture) } diff --git a/google/search.go b/google/search.go index 854cad7..25573b2 100644 --- a/google/search.go +++ b/google/search.go @@ -116,7 +116,15 @@ func (gogl *Google) solveCaptcha(page *rod.Page, sitekey, datas, proxyURL string // needs the live page's captcha element attributes. func (gogl *Google) classifyPage(page *rod.Page, queryProxyURL string) error { err := core.ClassifyFromPage(page, classifyGoogleDocument) - if !errors.Is(err, core.ErrCaptcha) || gogl.solveCaptchaOnPage(page, queryProxyURL) { + if page != nil { + if info, infoErr := page.Info(); infoErr == nil && isGoogleSorryURL(info.URL) { + err = core.ErrCaptcha + } + } + if err == nil { + return nil + } + if errors.Is(err, core.ErrCaptcha) && gogl.solveCaptchaOnPage(page, queryProxyURL) { return nil } return err diff --git a/google/search_raw.go b/google/search_raw.go index 29e7a72..69b1433 100644 --- a/google/search_raw.go +++ b/google/search_raw.go @@ -119,6 +119,10 @@ func classifyGoogleRawHTML(body []byte) error { return classifyGoogleDocument(doc) } +func isGoogleSorryURL(rawURL string) bool { + return strings.Contains(strings.ToLower(rawURL), "/sorry/") +} + func classifyGoogleDocument(doc *goquery.Document) error { if isGoogleCaptchaDocument(doc) { return core.ErrCaptcha @@ -185,6 +189,9 @@ func Search(ctx context.Context, query core.Query) (results []core.SearchResult, core.WithRequest(ctx).WithField("status_code", res.StatusCode).Debug( fmt.Sprintf("Google Raw response: code=%d", res.StatusCode), ) + if res.Request != nil && res.Request.URL != nil && isGoogleSorryURL(res.Request.URL.String()) { + return nil, core.ErrCaptcha + } body, err := core.ReadRawSearchBody(res) if err != nil { diff --git a/google/search_raw_test.go b/google/search_raw_test.go index ef28616..70b4f17 100644 --- a/google/search_raw_test.go +++ b/google/search_raw_test.go @@ -106,6 +106,7 @@ func TestGoogleClassifyRawHTML(t *testing.T) { {name: "no results", fixture: "search_no_results.html", want: core.ErrEmptyResult}, {name: "captcha page", fixture: "search_captcha.html", want: core.ErrCaptcha}, {name: "new captcha page", fixture: "search_captcha_new.html", want: core.ErrCaptcha}, + {name: "sorry challenge page", fixture: "search_sorry.html", want: core.ErrCaptcha}, {name: "soft block page", fixture: "search_soft_block.html", want: core.ErrBlocked}, } diff --git a/google/selectors.go b/google/selectors.go index 7bb61b9..9bb8a98 100644 --- a/google/selectors.go +++ b/google/selectors.go @@ -29,11 +29,13 @@ var Selectors = struct { ImageTitle []string }{ Captcha: "[data-sitekey]", - CaptchaPage: "form#captcha-form, [data-sitekey], .g-recaptcha, script[src*='recaptcha']", + CaptchaPage: "form#captcha-form, form[action*='/sorry/'], body[onload*='captcha'], [data-sitekey], .g-recaptcha, script[src*='recaptcha']", // CaptchaMarkers is the page-text fallback for captcha variants whose // markup doesn't match CaptchaPage. CaptchaMarkers: []string{ - "our systems have detected unusual traffic", + "detected unusual traffic", + "unusual traffic from your computer network", + "before you continue", "not a robot", "solve the captcha", }, diff --git a/google/testdata/search_sorry.html b/google/testdata/search_sorry.html new file mode 100644 index 0000000..3132f81 --- /dev/null +++ b/google/testdata/search_sorry.html @@ -0,0 +1,15 @@ +https://www.google.com/sorry/index + +
+

+
+Our systems have detected unusual traffic from your computer network. Please try your request again later. Why did this happen?

+ +
+IP address: 212.192.4.209
Time: 2026-07-01T20:58:45Z
URL: https://www.google.com/sorry/index
+
+
+
+ \ No newline at end of file