fix google: detect /sorry/ block pages; scope proxy sessions to credentials. Update default config

README.md

@@ -2,7 +2,6 @@
 
 # OpenSERP
 
-[![Go Report Card](https://goreportcard.com/badge/github.com/karust/openserp)](https://goreportcard.com/report/github.com/karust/openserp)
 [![Go Reference](https://pkg.go.dev/badge/github/karust/openserp?style=for-the-badge)](https://pkg.go.dev/github.com/karust/openserp)
 [![release](https://img.shields.io/github/v/release/karust/openserp)](https://github.com/karust/openserp/releases)
 [![Docker Pulls](https://img.shields.io/docker/v/karust/openserp)](https://hub.docker.com/r/karust/openserp)
@@ -12,7 +11,7 @@
 
 Use it as a search tool for **LLMs, agents, and RAG pipelines**, or as a scraper backend for **SEO rank tracking across Google, Yandex, Baidu, and more**. It is especially useful when your workflow needs RU/CN web coverage instead of another Google-only API.
 
-Run it locally, self-host it, or use [cloud version](https://openserp.org/cloud) when you want the same public API shape without operating the server.
+Run it locally, self-host it, or use the [cloud version](https://openserp.org/cloud) when you want the same public API shape without operating the server. Cloud is also the simplest way to support OpenSERP: it is where the technology gets tested at scale, and that work flows straight back into this open-source project.
 
 ## Features
 
@@ -166,9 +165,11 @@ curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust&
 ## Deployment Options
 
 - **Self-hosted (this repo)** - free, MIT-licensed, with full control over runtime, proxies, cache, and scaling.
-- **[OpenSERP Cloud](https://openserp.org/cloud)** - optional managed version from the project maintainers, with the same API shape.
+- **[OpenSERP Cloud](https://openserp.org/cloud)** - the same public API, run and maintained for you by the people who build OpenSERP.
 
-The hosted API helps fund continued development of the open-source project. Same endpoints, same response schema, and client code can migrate either direction.
+Same endpoints, same response schema, and client code migrates either direction, so you are never locked in.
+
+If OpenSERP is useful to you and you would rather not run the infrastructure yourself, using Cloud (or topping up an account) is a direct way to support the project. It is where the technology gets exercised against real traffic at scale, which is how the engines get hardened, new features get tested, and the open-source core keeps improving. Every request you send through Cloud helps fund and shape what lands back in this repository.
 
 ## API Docs
 
@@ -183,12 +184,12 @@ To browse the spec without running the server, see [docs/openapi.yaml](./docs/op
 
 Official client packages. Each works against your self-hosted server (set `baseUrl`) or the [hosted API](https://openserp.org/cloud) (set `apiKey`):
 
-| Type                        | Package                                                                                      | Install                         |
-| --------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------- |
-| JavaScript / TypeScript SDK | [`@openserp/sdk`](https://www.npmjs.com/package/@openserp/sdk)                               | `npm install @openserp/sdk`     |
-| Python SDK                  | [`openserp`](https://pypi.org/project/openserp/)                                             | `pip install openserp`          |
-| MCP server (AI agents)      | [`@openserp/mcp`](https://www.npmjs.com/package/@openserp/mcp)                               | `npx @openserp/mcp`             |
-| n8n community node          | [`@openserp/n8n-nodes-openserp`](https://www.npmjs.com/package/@openserp/n8n-nodes-openserp) | Install via n8n community nodes |
+| Type                        | Package                                                                                      | Source                                                              | Install                         |
+| --------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------- |
+| JavaScript / TypeScript SDK | [`@openserp/sdk`](https://www.npmjs.com/package/@openserp/sdk)                               | [openserpapi/sdk-js](https://github.com/openserpapi/sdk-js)         | `npm install @openserp/sdk`     |
+| Python SDK                  | [`openserp`](https://pypi.org/project/openserp/)                                             | [openserpapi/sdk-python](https://github.com/openserpapi/sdk-python) | `pip install openserp`          |
+| MCP server (AI agents)      | [`@openserp/mcp`](https://www.npmjs.com/package/@openserp/mcp)                               | [openserpapi/mcp](https://github.com/openserpapi/mcp)               | `npx @openserp/mcp`             |
+| n8n community node          | [`@openserp/n8n-nodes-openserp`](https://www.npmjs.com/package/@openserp/n8n-nodes-openserp) | [openserpapi/n8n](https://github.com/openserpapi/n8n)               | Install via n8n community nodes |
 
 See [**examples**](./examples) for small JavaScript and Python use cases covering search, AI grounding, SEO, content extraction, and image search.
 
@@ -515,6 +516,6 @@ Contributions are welcome. See [docs/CONTRIBUTING.md](./docs/CONTRIBUTING.md).
 
 - [GitHub Issues](https://github.com/karust/openserp/issues) - bugs, feature ideas, and reproducible issues.
 - [feedback@openserp.org](mailto:feedback@openserp.org) - private notes, longer feedback, or anything that does not fit GitHub Issues.
-- [Telegram Channel](https://t.me/+RJEKspw3mUlhZDMy) - OpenSERP news, release notes, and project updates. Direct messages are open for quick feedback and hosted API questions.
+- [Telegram Channel](https://t.me/+RJEKspw3mUlhZDMy) - OpenSERP news, release notes, and project updates.
 
 > OpenSERP is free and open-source. Only links listed in this repository and on [openserp.org](https://openserp.org) are associated with the project.

cmd/root.go

@@ -17,7 +17,7 @@ import (
 )
 
 const (
-	version               = "0.8.8"
+	version               = "0.8.9"
 	defaultConfigFilename = "config"
 	envPrefix             = "OPENSERP"
 )

cmd/serve.go

@@ -2,6 +2,8 @@ package cmd
 
 import (
 	"context"
+	"crypto/sha256"
+	"encoding/hex"
 	"errors"
 	"fmt"
 	"io"
@@ -289,12 +291,12 @@ func newBrowserPool(base core.BrowserOpts, defaultLaunchProxyURL string, laneSto
 		stopSweeper:  make(chan struct{}),
 		sweeperDone:  make(chan struct{}),
 	}
-	// A configured global proxy (legacy) becomes a pre-bound entry on the
-	// shared "direct" key so requests without a per-request proxy still use it.
 	if launchURL := strings.TrimSpace(defaultLaunchProxyURL); launchURL != "" {
-		pool.browsers[directBrowserKey] = &pooledBrowser{
-			launchProxyURL: launchURL,
-			lastUsedAt:     time.Now(),
+		if key := browserPoolKey(launchURL); key != directBrowserKey {
+			pool.browsers[key] = &pooledBrowser{
+				launchProxyURL: launchURL,
+				lastUsedAt:     time.Now(),
+			}
 		}
 	}
 	if idleTTL > 0 {
@@ -306,7 +308,8 @@ func newBrowserPool(base core.BrowserOpts, defaultLaunchProxyURL string, laneSto
 }
 
 // browserPoolKey derives the pool key from a request's proxy URL. Authenticated
-// HTTP/HTTPS proxies get their own Chrome keyed by scheme+host+port+username.
+// HTTP/HTTPS proxies get their own Chrome keyed by scheme+host+port+username
+// plus a short auth hash, so provider session tokens do not share one Chrome.
 // Empty/unauthenticated/SOCKS request URLs fall through to the shared
 // "direct" Chrome.
 func browserPoolKey(requestProxyURL string) string {
@@ -330,8 +333,10 @@ func browserPoolKey(requestProxyURL string) string {
 	if parsed.User == nil {
 		return directBrowserKey
 	}
-	username := parsed.User.Username()
-	return fmt.Sprintf("%s|%s|%s", parsed.Scheme, parsed.Host, username)
+	// Hash the full userinfo so a rotating password gets its own Chrome without
+	// leaking credentials into the key. Scheme+host+username stay readable.
+	sum := sha256.Sum256([]byte(parsed.User.String()))
+	return fmt.Sprintf("%s|%s|%s|%s", parsed.Scheme, parsed.Host, parsed.User.Username(), hex.EncodeToString(sum[:])[:16])
 }
 
 // browserLaunchURL returns the URL to pass to launcher.Proxy for a given

cmd/serve_test.go

@@ -30,28 +30,44 @@ func TestCommandDefaultsToQuiet(t *testing.T) {
 
 func TestBrowserPoolKey(t *testing.T) {
 	cases := []struct {
-		name string
-		raw  string
-		want string
+		name       string
+		raw        string
+		wantDirect bool
+		wantPrefix string
 	}{
-		{"empty -> direct", "", directBrowserKey},
-		{"unauth http -> direct", "http://proxy.example:8080", directBrowserKey},
-		{"unauth socks -> direct", "socks5://proxy.example:1080", directBrowserKey},
-		{"auth socks -> direct (rejected upstream)", "socks5://user:pass@proxy.example:1080", directBrowserKey},
-		{"auth http", "http://user:pass@proxy.example:8080", "http|proxy.example:8080|user"},
-		{"auth https different scheme", "https://user:pass@proxy.example:8443", "https|proxy.example:8443|user"},
-		{"different password same key", "http://user:other-pass@proxy.example:8080", "http|proxy.example:8080|user"},
-		{"different user different key", "http://user2:pass@proxy.example:8080", "http|proxy.example:8080|user2"},
-		{"different host different key", "http://user:pass@proxy2.example:8080", "http|proxy2.example:8080|user"},
-		{"different port different key", "http://user:pass@proxy.example:9090", "http|proxy.example:9090|user"},
+		{"empty -> direct", "", true, ""},
+		{"unauth http -> direct", "http://proxy.example:8080", true, ""},
+		{"unauth socks -> direct", "socks5://proxy.example:1080", true, ""},
+		{"auth socks -> direct (rejected upstream)", "socks5://user:pass@proxy.example:1080", true, ""},
+		{"auth http", "http://user:pass@proxy.example:8080", false, "http|proxy.example:8080|user|"},
+		{"auth https different scheme", "https://user:pass@proxy.example:8443", false, "https|proxy.example:8443|user|"},
+		{"different user different key", "http://user2:pass@proxy.example:8080", false, "http|proxy.example:8080|user2|"},
+		{"different host different key", "http://user:pass@proxy2.example:8080", false, "http|proxy2.example:8080|user|"},
+		{"different port different key", "http://user:pass@proxy.example:9090", false, "http|proxy.example:9090|user|"},
 	}
 	for _, tc := range cases {
 		t.Run(tc.name, func(t *testing.T) {
-			if got := browserPoolKey(tc.raw); got != tc.want {
-				t.Fatalf("browserPoolKey(%q) = %q, want %q", tc.raw, got, tc.want)
+			got := browserPoolKey(tc.raw)
+			if tc.wantDirect {
+				if got != directBrowserKey {
+					t.Fatalf("browserPoolKey(%q) = %q, want %q", tc.raw, got, directBrowserKey)
+				}
+				return
+			}
+			if !strings.HasPrefix(got, tc.wantPrefix) {
+				t.Fatalf("browserPoolKey(%q) = %q, want prefix %q", tc.raw, got, tc.wantPrefix)
+			}
+			if strings.Contains(got, "pass") {
+				t.Fatalf("browser pool key leaked password material: %q", got)
 			}
 		})
 	}
+
+	first := browserPoolKey("http://user:pass@proxy.example:8080")
+	second := browserPoolKey("http://user:other-pass@proxy.example:8080")
+	if first == second {
+		t.Fatalf("expected different passwords to produce different browser pool keys: %q", first)
+	}
 }
 
 func TestBrowserLaunchURL(t *testing.T) {
@@ -76,6 +92,28 @@ func TestBrowserLaunchURL(t *testing.T) {
 	}
 }
 
+func TestBrowserPoolGlobalProxyDoesNotOccupyDirectSlot(t *testing.T) {
+	globalProxy := "http://user:pass@proxy.example:8080"
+	pool := newBrowserPool(core.BrowserOpts{}, globalProxy, nil, 2, 0)
+	defer func() {
+		if err := pool.close(); err != nil {
+			t.Fatalf("close pool: %v", err)
+		}
+	}()
+
+	if _, ok := pool.browsers[directBrowserKey]; ok {
+		t.Fatalf("global proxy must not occupy %q browser slot", directBrowserKey)
+	}
+	key := browserPoolKey(globalProxy)
+	entry, ok := pool.browsers[key]
+	if !ok {
+		t.Fatalf("expected global proxy slot %q to be pre-bound", key)
+	}
+	if entry.launchProxyURL != globalProxy {
+		t.Fatalf("expected launch proxy %q, got %q", globalProxy, entry.launchProxyURL)
+	}
+}
+
 func TestBrowserPoolEvictLRU(t *testing.T) {
 	// Pre-populate with bare entries (browser=nil) so we exercise eviction
 	// without launching real Chrome. closePooledBrowser handles nil safely.

config.yaml

@@ -14,6 +14,8 @@ app:
   head: false # Headful mode
   leakless: false # Force browser process cleanup after request
   leave_head: false # Keep tabs open after request
+  block_resources: "image,font,css,media" # Block heavy subresources in browser mode
+  block_trackers: true # Block known tracker domains
   max_processes: 6 # Concurrent Chrome processes
   idle_ttl: 5m # close a Chrome that has not served traffic for this long
 

core/proxy_lane.go

@@ -236,11 +236,11 @@ func proxyLaneIDFromProxyURL(raw string) string {
 	if err != nil {
 		return ""
 	}
-	username := ""
+	userInfo := ""
 	if parsed.User != nil {
-		username = parsed.User.Username()
+		userInfo = parsed.User.String()
 	}
-	sum := sha256.Sum256([]byte(parsed.Host + "|" + username))
+	sum := sha256.Sum256([]byte(parsed.Scheme + "|" + parsed.Host + "|" + userInfo))
 	return hex.EncodeToString(sum[:])[:16]
 }
 

core/proxy_lane_test.go

@@ -1,6 +1,7 @@
 package core
 
 import (
+	"strings"
 	"testing"
 	"time"
 
@@ -77,14 +78,17 @@ func TestLaneStoreEvictsLRU(t *testing.T) {
 	}
 }
 
-func TestProxyLaneKeyForOmitsPassword(t *testing.T) {
+func TestProxyLaneKeyForUsesCredentialIdentity(t *testing.T) {
 	a := ProxyLaneKeyForTenant("Google", "", Query{}, "http://user:pass-a@proxy.example:8080")
 	b := ProxyLaneKeyForTenant("google", "", Query{}, "http://user:pass-b@proxy.example:8080")
 	if a.Empty() || b.Empty() {
 		t.Fatalf("expected derived lane keys, got %#v %#v", a, b)
 	}
-	if a != b {
-		t.Fatalf("expected password changes not to affect lane key: %#v %#v", a, b)
+	if a == b {
+		t.Fatalf("expected password changes to produce separate lane keys: %#v", a)
+	}
+	if strings.Contains(a.ID(), "pass-a") || strings.Contains(b.ID(), "pass-b") {
+		t.Fatalf("lane key leaked password material: %#v %#v", a, b)
 	}
 }
 

google/captcha_selector_test.go

@@ -25,6 +25,9 @@ func TestGooglePageTypeSelectors(t *testing.T) {
 		{"search_captcha_new.html", Selectors.CaptchaPage, true},
 		{"search_captcha_new.html", Selectors.ResultStats, false},
 
+		{"search_sorry.html", Selectors.CaptchaPage, true},
+		{"search_sorry.html", Selectors.ResultStats, false},
+
 		{"search_soft_block.html", Selectors.SoftBlock, true},
 		{"search_soft_block.html", Selectors.Captcha, false},
 		{"search_soft_block.html", Selectors.CaptchaPage, false},
@@ -60,7 +63,7 @@ func assertSelector(t *testing.T, fixture, selector string, wantHit bool) {
 	got := doc.Find(selector).Length() > 0
 	if got != wantHit {
 		if wantHit {
-			t.Fatalf("selector %q not found in %s — update selectors.go", selector, fixture)
+			t.Fatalf("selector %q not found in %s - update selectors.go", selector, fixture)
 		} else {
 			t.Fatalf("selector %q unexpectedly present in %s", selector, fixture)
 		}

google/search.go

@@ -116,7 +116,15 @@ func (gogl *Google) solveCaptcha(page *rod.Page, sitekey, datas, proxyURL string
 // needs the live page's captcha element attributes.
 func (gogl *Google) classifyPage(page *rod.Page, queryProxyURL string) error {
 	err := core.ClassifyFromPage(page, classifyGoogleDocument)
-	if !errors.Is(err, core.ErrCaptcha) || gogl.solveCaptchaOnPage(page, queryProxyURL) {
+	if page != nil {
+		if info, infoErr := page.Info(); infoErr == nil && isGoogleSorryURL(info.URL) {
+			err = core.ErrCaptcha
+		}
+	}
+	if err == nil {
+		return nil
+	}
+	if errors.Is(err, core.ErrCaptcha) && gogl.solveCaptchaOnPage(page, queryProxyURL) {
 		return nil
 	}
 	return err

google/search_raw.go

@@ -119,6 +119,10 @@ func classifyGoogleRawHTML(body []byte) error {
 	return classifyGoogleDocument(doc)
 }
 
+func isGoogleSorryURL(rawURL string) bool {
+	return strings.Contains(strings.ToLower(rawURL), "/sorry/")
+}
+
 func classifyGoogleDocument(doc *goquery.Document) error {
 	if isGoogleCaptchaDocument(doc) {
 		return core.ErrCaptcha
@@ -185,6 +189,9 @@ func Search(ctx context.Context, query core.Query) (results []core.SearchResult,
 	core.WithRequest(ctx).WithField("status_code", res.StatusCode).Debug(
 		fmt.Sprintf("Google Raw response: code=%d", res.StatusCode),
 	)
+	if res.Request != nil && res.Request.URL != nil && isGoogleSorryURL(res.Request.URL.String()) {
+		return nil, core.ErrCaptcha
+	}
 
 	body, err := core.ReadRawSearchBody(res)
 	if err != nil {

google/search_raw_test.go

@@ -106,6 +106,7 @@ func TestGoogleClassifyRawHTML(t *testing.T) {
 		{name: "no results", fixture: "search_no_results.html", want: core.ErrEmptyResult},
 		{name: "captcha page", fixture: "search_captcha.html", want: core.ErrCaptcha},
 		{name: "new captcha page", fixture: "search_captcha_new.html", want: core.ErrCaptcha},
+		{name: "sorry challenge page", fixture: "search_sorry.html", want: core.ErrCaptcha},
 		{name: "soft block page", fixture: "search_soft_block.html", want: core.ErrBlocked},
 	}
 

google/selectors.go

@@ -29,11 +29,13 @@ var Selectors = struct {
 	ImageTitle        []string
 }{
 	Captcha:     "[data-sitekey]",
-	CaptchaPage: "form#captcha-form, [data-sitekey], .g-recaptcha, script[src*='recaptcha']",
+	CaptchaPage: "form#captcha-form, form[action*='/sorry/'], body[onload*='captcha'], [data-sitekey], .g-recaptcha, script[src*='recaptcha']",
 	// CaptchaMarkers is the page-text fallback for captcha variants whose
 	// markup doesn't match CaptchaPage.
 	CaptchaMarkers: []string{
-		"our systems have detected unusual traffic",
+		"detected unusual traffic",
+		"unusual traffic from your computer network",
+		"before you continue",
 		"not a robot",
 		"solve the captcha",
 	},

google/testdata/search_sorry.html

@@ -0,0 +1,15 @@
+<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>https://www.google.com/sorry/index</title></head>
+<body style="font-family: arial, sans-serif; background-color: #fff; color: #000; padding:20px; font-size:18px; overscroll-behavior:contain;" onload="e=document.getElementById('captcha');if(e){e.focus();} if(solveSimpleChallenge) {solveSimpleChallenge(0,0);}">
+<div style="max-width:400px;">
+<hr noshade="" size="1" style="color:#ccc; background-color:#ccc;"><br>
+<div style="font-size:13px;">
+Our systems have detected unusual traffic from your computer network.  Please try your request again later.  <a href="#" onclick="document.getElementById('infoDiv0').style.display='block';">Why did this happen?</a><br><br>
+<div id="infoDiv0" style="display:none; background-color:#eee; padding:10px; margin:0 0 15px 0; line-height:1.4em;">
+This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the <a href="//www.google.com/policies/terms/">Terms of Service</a>. The block will expire shortly after those requests stop.<br><br>This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.  <a href="//support.google.com/websearch/answer/86640">Learn more</a><br><br>Sometimes you may see this page if you are using advanced terms that robots are known to use, or sending requests very quickly.
+</div>
+<br><div style="font-size:13px; line-break: anywhere;">
+IP address: 212.192.4.209<br>Time: 2026-07-01T20:58:45Z<br>URL: https://www.google.com/sorry/index<br>
+</div>
+</div>
+</div>
+</body></html>