Chunk-level ask used to inject the top-6 chunks into the generation
prompt unconditionally. On the 60-question eval, all 10 negative
questions (answer not in the corpus) were still answered, and the
top-1 BM25 scores of negatives overlap 7/10 with positives, so no
score threshold can gate this reliably.
The ask path now routes through a lightweight LLM grader (single call,
~200-400 tokens) before generation:
- all relevant -> generate as before
- partially relevant -> filter to the graded-relevant chunks, generate
- none relevant -> rewrite the query once with LLM, re-search and
re-grade; still none -> refuse explicitly instead of hallucinating
Every LLM step degrades safely: no provider, grading failure or
unparseable output falls back to the previous plain-generation
behaviour, so nothing breaks when the flag is on but no provider is
configured.
Only active under COUNTBOT_RAG_CHUNKS=1. No new dependencies.
Tests: 12 new (three-way routing, refusal after one rewrite, retry
succeeds, grading failure/unparseable/no-provider fallbacks, grader
output parsing contract). Full suite: 138 passed.
Refs: #107
With chunk-level retrieval the top-k list can be monopolised by
several high-scoring chunks of the same document: on the 52-doc eval
corpus, collecting 10 distinct documents required scanning 25.3 chunks
on average, starving multi-doc questions of sources.
search()/search_chunks() now oversample (top_k * 5), keep at most
max_per_doc chunks per document first, then backfill by score if the
result list is short - so small knowledge bases keep full top_k while
large ones gain source diversity.
Measured on the 60-question eval (production top-6 injection):
- cross-doc source coverage: 0.500 -> 0.556
- paraphrased-question hit: 0.600 -> 0.700
- overall positive hit: 0.800 -> 0.829
- single-doc / needle / cross hit: no regression
max_per_doc<=0 restores the previous behaviour exactly.
Tests: 4 new (front-cap respected, backfill keeps result count, cap
disabled falls back to legacy ranking, top_k and ordering preserved).
Full suite: 126 passed.
Refs: #107
BM25Index already supports jieba and falls back to per-character
tokenization when it is missing. Because the dependency was commented
out, a default 'pip install -r requirements.txt' silently ran the
degraded path, so the wiki search quality users actually get is well
below what the code (and our published eval numbers in #107) assume.
Measured impact on the 60-question eval (with vs without jieba):
- direct-question recall@5: 0.80 -> 0.53
- paraphrased-question recall@5: 0.20 -> 0.10 (no top-1 hit at all)
- exact-fact recall@10: 1.00 -> 0.80
One-line change, no code changes: the graceful fallback stays for
anyone who deliberately removes it.
Refs: #107
Three scripts + 60 hand-annotated questions over the public
countbot.cn/docs corpus (52 pages, fetched by fetch_corpus.py):
- run_g0.py: doc-level BM25 baseline (pre-change behaviour)
- run_g1.py: chunk-level BM25 (this PR) on the same questions
- fetch_corpus.py: downloads the public corpus into rag-bench/corpus/
Pure programmatic scoring (recall@k / MRR@10 / NDCG@10 / top1 /
injected-token volume), no LLM judge, no private data. The headline
numbers quoted in the PR description come from running these on the
same corpus/questions before and after the chunk change.
Refs: #107
Splits each wiki doc by Markdown headings (long sections re-split by
paragraph with overlap; code fences kept intact) and indexes every chunk
as a small doc on the existing BM25Index - no algorithm rewrite, no new
third-party dependencies.
Retrieval unit changes from whole doc to chunk (same 60-question eval):
- context injected: 12,334 -> 910 est tokens avg (-92.6%)
- direct-question top1: 0.067 -> 0.533; production hit rate 56% -> 82%
- every result carries [slug#section] provenance (was 0%)
Default off (COUNTBOT_RAG_CHUNKS=1 to enable); unset behaves exactly as
before, verified by switch-OFF regression tests. AgentLoop,
ToolRegistry and Cron are untouched. Chunk index is a derived artifact
(workspace/wiki/chunk_index.json), deletable and rebuildable.
Tests: tests/rag/ 34 new; full suite passes.
Refs: #107
countbot-ai