1620 Commits

Author SHA1 Message Date
Karim shoair 48da61d1ee Updating contribution rules
Rephrased sections for improved readability and consistency.
2026-09-12 17:59:54 +03:00
Karim shoair 3115acab99 docs: update translation buttons 2026-09-11 22:20:26 +03:00
Karim shoair a54a23be7a docs: updating a sponsor image 2026-09-11 21:02:58 +03:00
Karim shoair 28c3296714 docs: adding a new sponsor 2026-09-04 15:33:23 +03:00
Karim shoair 1f40d9b11b docs: moving up a sponsor 2026-09-02 15:09:56 +03:00
Karim shoair 196c81ad67 docs: update a sponsor url 2026-09-01 15:38:52 +03:00
Karim shoair 458e2a2ac9 docs: removing a sponsor 2026-08-25 21:38:48 +03:00
Karim shoair 3b11b85eac docs: update a sponsor image 2026-08-24 21:41:06 +03:00
Karim shoair 333fa22b7a v0.4.15 (#423) v0.4.15 2026-08-23 22:43:08 +03:00
Karim shoair d04613e52e docs: update the changelog 2026-08-23 22:42:37 +03:00
Karim shoair 3e03f78bc0 chore: pump deps up and update stars history image 2026-08-23 22:14:09 +03:00
Yiğit ERDOĞAN ac18622ea2 fix(static): send the request once when retries is below 1 (#420) 2026-08-23 22:09:37 +03:00
Yiğit ERDOĞAN 63cdc99cfd fix(spiders): keep the request meta on cached responses (#419) 2026-08-23 21:58:21 +03:00
Karim shoair 76395b7273 docs: update the intro 2026-08-23 20:31:17 +03:00
Karim shoair f164713efa feat(fetchers): keep browser tabs open and reuse them across requests
Browser sessions no longer close a tab after its request is completed. The tab returns to the pool as ready, and the next request reuses it, reapplying its own timeouts, extra headers, and resource/domain routes (after `unroute_all`) so nothing leaks between requests. Tabs that hit an error or got closed by the browser are closed and evicted, and the internal response listener is detached after every request so reused tabs don't stack handlers.

Adds `close_pages()` to all browser sessions to close every open tab; the next request opens a fresh one. `PagePool` gains `get_ready_page`/`remove_page`/`clear` and `PageInfo.mark_ready`, new pages start busy, and the unused `cleanup_error_pages` is removed.

Proxy-rotation contexts keep closing per request. Docs and the agent skill describe the new lifecycle.
2026-08-23 20:28:30 +03:00
Karim shoair 6c5b691db7 docs: add new ai sub-section with the new changes + add changelog file 2026-08-23 18:00:15 +03:00
Karim shoair 6ca35d0448 feat(spiders): Adding easy interface for building RAG systems 2026-08-23 16:26:15 +03:00
Karim shoair 6815ca4f4f docs: add a page for the agent skill and restructure the website sections to clearer 2026-08-22 19:32:11 +03:00
Karim shoair 6e6373c997 feat(mcp): add two tools to open requests sessions and use them 2026-08-22 19:17:32 +03:00
Karim shoair 0cb3e971c9 Merge branch 'main' into dev 2026-08-21 20:28:49 +03:00
Karim shoair 4ce0fb1c1a docs: adding a new sponsor 2026-08-21 20:24:35 +03:00
Karim shoair 0154400933 docs: updating a sponsor 2026-08-21 19:38:45 +03:00
Karim shoair 1181fc6a1f feat(mcp)!: rename get to make_request and add more methods. 2026-08-21 19:26:47 +03:00
Karim shoair 93ae0ac9c4 fix(mcp): Set the session proxy to be browser-level argument
Thanks for @Yigtwxx for pointing that out in #418
2026-08-21 19:25:29 +03:00
Karim shoair 73ad18be58 refactor(mcp)!: More reliable structure + new tools
A better fix than #418 is to eliminate this bug class and keep the AI agent knowing the default parameter values.

I have also shortened the docstrings where possible to save tokens.
2026-08-21 03:16:17 +03:00
Yiğit ERDOĞAN d4e121ce7c fix(ai)!: keep session settings on MCP session fetches (#418)
The three MCP tools that accept a `session_id` forwarded every per-fetch
parameter to the session's `fetch()`, including their own defaults when
the caller supplied nothing. `validate_fetch` treats every key it
receives as an override, so the branch that reads the value from the
session config never ran, and the tool defaults replaced the settings the
session was opened with.

- Forward only the parameters the caller actually supplied
- Default those parameters to `None` on `fetch`, `bulk_fetch`,
  `stealthy_fetch`, `bulk_stealthy_fetch` and `screenshot`
- Add regression tests for the forwarded kwargs, and for the session
  settings surviving the validation the tools go through
2026-08-19 23:19:22 +03:00
Mrityunjay SIngh 0003f6444b fix(parser): keep filtering on blank class_ and escape CSS string values (#417) 2026-08-19 22:13:14 +03:00
Karim shoair 39ecbf0f9a docs: update stars history 2026-08-19 22:04:12 +03:00
Karim shoair c0326263b5 fix(stealth): solve Cloudflare challenges regardless of locale + retry cap + fix crashing mid-solve
The solver matched English challenge strings, so any non-default locale left the challenge unsolved.
The real cause was Playwright's context locale option patching the main thread only, while Web Workers kept the browser language, a mismatch Cloudflare rejects. Locale is now applied via browser launch flags, so the page, workers, and the Accept-Language header stay consistent, and the solver checks the language-free challenge markers with a retry cap.

Also catch patchright errors in the page-content retry so stealth pages stop crashing mid-solve.

Co-Authored-By: Parash Subedi <parash.subedi@grepsr.com>
2026-08-19 21:59:34 +03:00
Karim shoair 9376a3d8c4 Merge branch 'main' into dev 2026-08-19 20:23:42 +03:00
Karim shoair cb00f1bba8 docs: adding a new sponsor 2026-08-19 20:06:03 +03:00
Karim shoair e90875f31f docs: fix missing parts from the updates
Missing from #414
2026-08-19 02:03:32 +03:00
Mrityunjay SIngh 029d098c06 fix: require authentication by default for the HTTP transport (#414)
Co-authored-by: MJ <kaalu@MJs-MacBook-Air.local>
2026-08-19 01:56:23 +03:00
Karim shoair 0e1e085aa7 build: correcting dep version 2026-08-19 01:54:05 +03:00
Karim shoair 1cd938e040 build: pump version and deps 2026-08-19 01:13:30 +03:00
Karim shoair 316b6d238c fix(parser): find/find_all with class_ silently miss multi-class elements (#410) 2026-08-19 00:18:12 +03:00
yetval 69a2985ada fix(parser): match multi-class elements in find/find_all class_ filter 2026-08-12 00:02:23 -04:00
Karim shoair 5d213a2d47 v0.4.14 (#409) v0.4.14 2026-08-11 01:27:00 +03:00
Karim shoair 90c30446ff Merge branch 'main' into dev 2026-08-11 01:09:42 +03:00
Karim shoair 2b9d0e2933 style: image size adjustment 2026-08-11 01:07:32 +03:00
Karim shoair 2c371e0f2e docs: add stars history manually because of Github API changes
Ref.: https://www.star-history.com/blog/github-stargazer-api-restriction
2026-08-11 01:03:00 +03:00
Karim shoair 4e0ac98d8c build: pump version and deps 2026-08-11 00:58:50 +03:00
Karim shoair 3f32fa9050 docs: re-adding a sponsor
Revert: https://github.com/D4Vinci/Scrapling/commit/253d316a89c4207b6959d08484fcbd4243bb879f#diff-d0a2065f1a22f388fa5e96f04ebba0efa783c45613b500d78b3b2f1a2341d9cb
2026-08-11 00:37:19 +03:00
Karim shoair 9af7e2796a docs: adding a new sponsor 2026-08-11 00:30:14 +03:00
Karim shoair 64bc7007b0 v0.4.13 (#406) v0.4.13 2026-08-09 22:06:36 +03:00
Karim shoair 10c6c8951e perf(mcp): Add more specific instruction to the server-level instructions 2026-08-09 22:05:58 +03:00
Karim shoair b2106e6054 docs: add an AI policy for contributions 2026-08-09 21:20:49 +03:00
Karim shoair b498fe7ee0 docs: update benchmarks results with the latest numbers 2026-08-09 21:14:53 +03:00
Karim shoair fad6893d66 docs: add the new templates to the docs 2026-08-09 20:56:29 +03:00
Karim shoair 6e41f335f9 fix(xml template): Be specific about the node type hint 2026-08-09 20:49:28 +03:00