diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..da00f47 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,76 @@ +name: Bug report +description: Report a reproducible OpenSERP bug. +title: "bug: " +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + Thanks for taking the time to report a bug. Reproducible reports are the fastest to fix. + - type: textarea + id: summary + attributes: + label: What happened? + description: Describe the unexpected behavior and what you expected instead. + placeholder: "Google search returns empty results for ..." + validations: + required: true + - type: textarea + id: reproduce + attributes: + label: How do we reproduce it? + description: Include the exact command, request, config, and minimal input. + placeholder: | + 1. Run `openserp serve ...` + 2. Request `curl "..."` + 3. See ... + validations: + required: true + - type: dropdown + id: mode + attributes: + label: Retrieval mode + options: + - browser + - raw + - both + - not sure + validations: + required: true + - type: dropdown + id: engine + attributes: + label: Engine + options: + - google + - yandex + - baidu + - bing + - duckduckgo + - ecosia + - megasearch + - not engine-specific + validations: + required: true + - type: input + id: version + attributes: + label: OpenSERP version + description: Paste `openserp --version`, the Docker tag, or the commit SHA. + placeholder: "0.8.0 / v0.7.15 / commit SHA" + validations: + required: true + - type: textarea + id: environment + attributes: + label: Environment + description: OS, Go version, Docker/browser version, proxy/captcha setup if relevant. + placeholder: "Windows 11, Go 1.24, Docker, Chromium ..." + validations: + required: true + - type: textarea + id: logs + attributes: + label: Logs or response body + description: Paste relevant logs, stack traces, or API responses. Remove secrets and proxy credentials. + render: shell diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..eabeaba --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: true +contact_links: + - name: Contributing guide + url: https://github.com/karust/openserp/blob/main/docs/CONTRIBUTING.md + about: Start here before opening a pull request. + - name: Add a new engine guide + url: https://github.com/karust/openserp/blob/main/docs/ADDING_AN_ENGINE.md + about: Implementation checklist for new search engines. diff --git a/.github/ISSUE_TEMPLATE/engine_request.yml b/.github/ISSUE_TEMPLATE/engine_request.yml new file mode 100644 index 0000000..0b4684c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/engine_request.yml @@ -0,0 +1,47 @@ +name: Engine request +description: Request support for a new search engine or engine vertical. +title: "engine: " +labels: ["enhancement"] +body: + - type: input + id: engine + attributes: + label: Engine name + placeholder: "Brave Search" + validations: + required: true + - type: input + id: search_url + attributes: + label: Public search URL + placeholder: "https://search.brave.com/search?q=openserp" + validations: + required: true + - type: dropdown + id: vertical + attributes: + label: Result type + options: + - web + - images + - web and images + - other + validations: + required: true + - type: textarea + id: selectors + attributes: + label: Known stable selectors or sample HTML + description: If you inspected the page, mention stable data attributes or attach a sanitized fixture. + - type: textarea + id: notes + attributes: + label: Notes + description: Region/language quirks, captcha behavior, pagination, or API constraints. + - type: checkboxes + id: contribution + attributes: + label: Contribution + options: + - label: I can help test this engine. + - label: I am willing to implement this engine. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..93dd749 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,47 @@ +name: Feature request +description: Suggest an OpenSERP improvement that is not engine-specific. +title: "feat: " +labels: ["enhancement"] +body: + - type: textarea + id: problem + attributes: + label: Problem + description: What user problem does this solve? + placeholder: "It is hard to ..." + validations: + required: true + - type: textarea + id: proposal + attributes: + label: Proposed solution + description: Describe the behavior, API, CLI, or documentation change you want. + validations: + required: true + - type: textarea + id: alternatives + attributes: + label: Alternatives considered + description: What workarounds or smaller versions have you considered? + - type: dropdown + id: area + attributes: + label: Area + options: + - CLI + - API + - docs + - parser + - proxy + - captcha + - resilience + - tests + - other + validations: + required: true + - type: checkboxes + id: contribution + attributes: + label: Contribution + options: + - label: I am willing to open a PR for this. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..3b2e704 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,20 @@ +## Summary + +- + +## Why + +- + +## Testing + +- [ ] `make test` +- [ ] `make lint` +- [ ] `make test-integration` (only for browser, proxy, captcha, or live-engine changes) + +## Checklist + +- [ ] I linked the related issue or explained why there is none. +- [ ] I updated docs or examples for changed user-facing behavior. +- [ ] I kept unit tests deterministic and free of browser/network dependencies. +- [ ] I removed secrets, proxy credentials, and private logs from examples. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 52f6aa2..d8c6ce9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -42,3 +42,6 @@ jobs: - name: Run golangci-lint uses: golangci/golangci-lint-action@v6 + with: + version: v1.64.8 + args: --config .golangci.yml diff --git a/.golangci.yml b/.golangci.yml new file mode 100644 index 0000000..35af932 --- /dev/null +++ b/.golangci.yml @@ -0,0 +1,7 @@ +run: + tests: true + timeout: 5m + +issues: + max-issues-per-linter: 0 + max-same-issues: 0 diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..8e9de6d --- /dev/null +++ b/Makefile @@ -0,0 +1,25 @@ +GO ?= go +BINARY ?= openserp +PKGS ?= ./... +GOFILES := $(shell git ls-files '*.go') + +.PHONY: build test test-integration lint run fmt + +build: + $(GO) build -o $(BINARY) . + +test: + $(GO) test -race -count=1 $(PKGS) + +test-integration: + OPENSERP_INTEGRATION_TESTS=1 $(GO) test -race -count=1 -timeout=120s -tags=integration $(PKGS) + +lint: + $(GO) vet $(PKGS) + golangci-lint run --config .golangci.yml + +run: + $(GO) run . serve + +fmt: + gofmt -w $(GOFILES) diff --git a/docs/ADDING_AN_ENGINE.md b/docs/ADDING_AN_ENGINE.md new file mode 100644 index 0000000..d87c6c5 --- /dev/null +++ b/docs/ADDING_AN_ENGINE.md @@ -0,0 +1,75 @@ +# Adding a Search Engine + +This is the short checklist for adding a new engine. Keep the first PR small: +web search, deterministic parser tests, and registration. Add images or advanced +parameters in follow-up PRs. + +## 1. Create the engine package + +Use an existing engine package as the template. A complete engine normally has: + +- `url.go` - pure URL builders that reject an empty query. +- `selectors.go` - stable selectors shared by browser mode, raw mode, and parser tests. +- `parse_html.go` - `ParseHTML(io.Reader)` using goquery. +- `search.go` - browser-mode implementation using `core.Browser`. +- `search_raw.go` - optional raw HTTP implementation. +- `features.go` - optional SERP feature extraction. +- `*_test.go` and `testdata/` - URL and parser fixtures. + +Prefer stable data attributes over generated CSS classes. When a selector is +fragile, add two or three explicit fallbacks in `selectors.go`. + +## 2. Implement the search contract + +Browser engines implement `core.SearchEngine`: + +- `Search(context.Context, core.Query) ([]core.SearchResult, error)` +- `SearchImage(context.Context, core.Query) ([]core.SearchResult, error)` +- `IsInitialized() bool` +- `Name() string` +- `GetRateLimiter() *rate.Limiter` + +Raw engines should expose a `ParseHTML(io.Reader)` path so tests and +`POST /{engine}/parse` can use the same parser. + +## 3. Register the engine + +Update: + +- `cmd/serve.go` for server wiring. +- CLI search dispatch when the engine is CLI-visible. +- `config.yaml` with rate limits and optional proxy tag. +- `README.md` and `docs/openapi.yaml` when public endpoints or parameters change. + +## 4. Add tests + +Required for the first PR: + +- Table-driven URL builder tests. +- Parser tests using small sanitized HTML fixtures. +- Integration tests only when needed, gated with `testutil.RequireIntegration(t)`. + +Default tests must pass without browser or network access: + +```bash +make test +``` + +Run live/browser checks only for engine behavior: + +```bash +make test-integration +``` + +## 5. Open the PR + +Before opening the PR: + +```bash +make fmt +make lint +make test +``` + +Keep the PR focused. A good first engine PR should not refactor shared browser, +server, or response-envelope behavior unless the engine cannot work without it. diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 9fa9dde..2052362 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -1,5 +1,16 @@ # Contributing to OpenSERP +## Fastest Path to a Merged PR + +1. Pick a focused issue, ideally one labeled `good first issue`. +2. Comment on the issue with the approach you plan to take. +3. Keep the PR narrow: one bug, one parser fallback, one docs page, or one engine step. +4. Run `make fmt`, `make lint`, and `make test` before opening the PR. +5. Explain what changed, why it changed, and how you tested it. + +Good first issues are curated in [`GOOD_FIRST_ISSUES.md`](GOOD_FIRST_ISSUES.md). +New engine work should start with [`ADDING_AN_ENGINE.md`](ADDING_AN_ENGINE.md). + ## Development Setup ### Prerequisites @@ -7,14 +18,15 @@ - Go 1.24+ - Chromium/Chrome (only required for browser-mode work and integration tests) - Optional: Docker +- Optional: `make` and `golangci-lint` for the same local workflow used by CI ### Clone, build, run ```bash git clone https://github.com/karust/openserp.git cd openserp -go build -o openserp . -./openserp serve +make build +make run ``` ### Test commands @@ -22,13 +34,13 @@ go build -o openserp . Unit tests (default, no browser/network assumptions): ```bash -go test -race ./... +make test ``` Integration tests (explicitly enabled): ```bash -OPENSERP_INTEGRATION_TESTS=1 go test -race -timeout=120s ./... +make test-integration ``` Notes: @@ -80,15 +92,16 @@ Update [`config.yaml`](../config.yaml) with your engine section: - Parser tests (prefer deterministic fixtures in `testdata/`) - Integration tests guarded by `testutil.RequireIntegration(t)` +See [`ADDING_AN_ENGINE.md`](ADDING_AN_ENGINE.md) for the full checklist. + ## Code Style and Quality Checks Run these before opening a PR: ```bash -gofmt -w . -go vet ./... -golangci-lint run -go test -race ./... +make fmt +make lint +make test ``` Guidelines: diff --git a/docs/GOOD_FIRST_ISSUES.md b/docs/GOOD_FIRST_ISSUES.md new file mode 100644 index 0000000..aa13256 --- /dev/null +++ b/docs/GOOD_FIRST_ISSUES.md @@ -0,0 +1,67 @@ +# Good First Issue Backlog + +Small, contributor-friendly tasks curated from the OSS roadmap. The first section +tracks issues that are **already filed** on GitHub with the `good first issue` +label. The second section is a backlog of vetted ideas that are **not yet filed** — +open one (or ask a maintainer to) before starting work so effort isn't duplicated. + +New contributor? Read [`CONTRIBUTING.md`](CONTRIBUTING.md) first; new engines start +with [`ADDING_AN_ENGINE.md`](ADDING_AN_ENGINE.md). + +## Open starter issues + +### [#27 Add CLI validation tests for supported engines](https://github.com/karust/openserp/issues/27) + +Area: CLI, tests + +Add a table test verifying every supported engine name is accepted by the CLI and +an unknown engine returns a deterministic, user-friendly error listing valid +engines. Covers `google`, `yandex`, `baidu`, `bing`, `duckduckgo`, `ecosia`. +Start in [`cmd/search.go`](../cmd/search.go). + +### [#28 Add Yandex parser fallback micro-fixtures](https://github.com/karust/openserp/issues/28) + +Area: parser tests + +Add compact Yandex HTML fixtures covering link, title, and snippet fallback +selectors without depending on full saved result pages. No browser or network. +Start in `yandex/selectors.go` and `yandex/parse_html.go`. + +### [#29 Add Baidu parser fallback micro-fixtures](https://github.com/karust/openserp/issues/29) + +Area: parser tests + +Add compact Baidu HTML fixtures for title, URL, and description fallback paths. +Missing optional fields must not panic; result order stays stable. No browser or +network. Start in `baidu/selectors.go` and `baidu/parse_html.go`. + +### [#30 Document raw-mode support per engine](https://github.com/karust/openserp/issues/30) + +Area: docs + +Add a small table showing which engines support browser mode, raw mode, and +`/{engine}/parse`. The table must match current code; link it from the README +search-endpoint section. Start in `cmd/serve.go` and `README.md`. + +### [#31 Add release smoke-check script stub](https://github.com/karust/openserp/issues/31) + +Area: release tooling + +Add a script under `.release/` that builds the binary, starts the server, checks +`/health`, and exits cleanly. It must fail fast with a useful error when the +server does not become healthy. Docker and `go install` checks are follow-ups. +Document it in [`.release/build.md`](../.release/build.md). + +## Backlog + +These are good candidates from the roadmap but have **no GitHub issue yet**. File +one before starting. + +### Draft a Brave Search engine skeleton + +Area: new engine + +Create a non-registered `brave/` package skeleton with URL builder table tests and +a minimal parser fixture (title, URL, snippet). Leave live browser search for a +follow-up. Do not expose the engine in README or API docs until browser search +works. See [`ADDING_AN_ENGINE.md`](ADDING_AN_ENGINE.md).