Replaces a heuristic with a declaration on the retrieval hot path.
_search_relevant_source_urls() had to decide, per result, whether a
retriever had already fetched the page or had only returned a link. With
nothing recording that, it guessed from length:
if url and raw_content and len(raw_content) > 100:
Any snippet over 100 characters was therefore treated as article text:
the URL was never scraped and the report carried no verifiable citation
for that source. That is #1846 and #1892. The fix at the time was to cap
snippet lengths inside the Searx and DuckDuckGo retrievers so they stayed
under the threshold -- per-retriever tuning to satisfy a heuristic, which
has to be repeated for every retriever that ever returns a long snippet.
Adds BaseRetriever with a `requires_scraping` attribute, defaulting to
True, and declares it on the five retrievers where the answer is known:
TavilySearch True search() never sets include_raw_content
SearxSearch True "content" is an ordinary result snippet
Duckduckgo True "body" is an ordinary result snippet
PubMedCentralSearch False returns full article text inline
CustomRetriever False contract is list[{url, raw_content}]
Deliberately conservative in three ways:
- Declaring is optional. A retriever with no `requires_scraping` keeps
the exact legacy length behaviour, so third-party and user-defined
retrievers are untouched. This is the difference between an additive
change and a breaking one.
- The other 18 shipped retrievers are left undeclared for now. Their
behaviour is unchanged and they can be annotated as each is confirmed.
- The existing snippet caps in searx.py and duckduckgo.py stay. They are
redundant under the declaration but harmless, and removing them is a
separate change that should be justified on its own.
Subclassing BaseRetriever is not required -- the call site only looks for
the attribute. The class exists to document the contract and give new
retrievers somewhere to read it.
Verified: 406 tests pass on 3.12 and 3.14. The two behavioural tests fail
against the previous classification and pass after it, so they are
load-bearing rather than decorative. Live research runs on both the
default Tavily path and DuckDuckGo produce results in line with the
pre-change baselines (context volume 40.4k vs 40.9k-43.6k across runs).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assaf Elovic
a167526848