mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
Nicolò Boschi
828a6e55d3
* refactor(retain): format pgvector literals without orjson orjson was a single-purpose dependency: one function (`embedding_to_pgvector`), on the retain write path only, rendering an embedding as the '[0.1,...]' literal asyncpg binds to `vector`. It played no part in HTTP serialization — there is no ORJSONResponse in the API, and recall builds its query literal with plain str(). Measurement decided it. On `perf-test --suite retain --scale small` (200 items, 384d, mock LLM + pg0) the orjson calls totalled 2.9ms of a ~1.9s retain, and paired throughput runs could not tell the two implementations apart from noise (orjson 75-124 items/s, replacement 81-115). What replaces it is one C-level %-format call per vector against a per-dimension cached "[%.9g,...]" template: ~1.8x faster than the repr() generator that was already the non-finite fallback, ~5x slower than orjson. No pure-Python formatter closes that gap — orjson formats floats with SIMD Ryu in Rust. Beating it means not emitting text at all, which the benchmark now measures as `binary_pgvector` (~2 orders of magnitude faster, 2.8x fewer wire bytes) but which needs an asyncpg codec for the `vector` type and would still leave Oracle on the text path. Values are narrowed to float32 before formatting, exactly once. Applying %.9g straight to a float64 rounds twice — to nine digits, then to float32 — and the two roundings pick different neighbouring float32s for ~0.8% of values. Narrowing first makes the single remaining rounding the one PostgreSQL would have done, so stored bytes are unchanged. The new regression test samples 4,096 elements, where that rate misses by ~31. The literal is now fixed-width rather than shortest-form (0.100000001 where orjson wrote 0.1) — same float32, ~12% more SQL text. The orjson version floor stays in pyproject as a transitive-only pin (langsmith still pulls it), matching how pyjwt is handled. * build: drop orjson from the runtime dependency closure The previous commit removed the last import; this removes the package from what production resolves. orjson's only other path into the environment is langchain-text-splitters -> langchain-core -> langsmith -> orjson, and langchain has been test-only since #3756 — retain's chunker is its own streaming RecursiveCharacterTextSplitter, and langchain survives purely as the reference implementation test_chunking_matches_legacy.py diffs against (with a guard test asserting the reference is still really langchain). So the `orjson>=3.11.6` floor follows langsmith into the test extra and the dev group, exactly as the langchain-core and langsmith floors already did when langchain left the runtime. Deleting the floor outright would have dropped the unbounded-recursion DoS bound for the version langsmith pulls. uv export --no-dev -> orjson absent uv export -> orjson 3.11.7 (via langsmith, test only) uv.lock is unchanged. * test: vendor the chunking oracle and drop langchain, langsmith and orjson orjson's last path into the environment was the test extra: langchain-text-splitters -> langchain-core -> langsmith -> orjson. langchain was there for one reason — the differential chunking tests need the splitter retain used before #3756 as their oracle, and that oracle has to be genuinely the old implementation or the tests pass vacuously. So the oracle moves in-tree. tests/chunking_reference.py transcribes langchain 1.1.2's RecursiveCharacterTextSplitter (MIT), specialised to the one configuration retain ever used: chunk_overlap=0, keep_separator default "start", length_function=len, non-regex separators. The branches that configuration cannot reach are left out rather than transcribed untested. The transcription was verified against the real splitter before the dependency was removed: 20,079 comparisons — every case and size the chunking tests use, plus 20,000 seeded fuzz documents built from the fragment shapes that break splitters (empty strings, whitespace runs, CRLF, CJK, oversized tokens) — with zero mismatches. tests/test_chunking_reference_matches_langchain.py keeps that check runnable: uv run --with langchain-text-splitters pytest \ tests/test_chunking_reference_matches_langchain.py It skips by default, since langchain is deliberately absent. test_the_legacy_reference_really_is_the_old_implementation loses its live langchain call, so its anchor becomes a table of five expected splits — one per separator tier, including the per-character fallback — produced BY langchain and pasted in. A reference that quietly became the new implementation has to reproduce all five to slip through. uv export --no-dev -> no orjson, no langchain uv export -> no orjson, no langchain uv.lock loses 392 lines.
90 lines
3.3 KiB
Python
90 lines
3.3 KiB
Python
"""Re-verify the vendored chunking oracle against the real langchain splitter.
|
|
|
|
``tests/chunking_reference.py`` is a transcription of langchain's
|
|
``RecursiveCharacterTextSplitter``, kept so the differential chunking tests kept their
|
|
oracle when ``langchain-text-splitters`` left the dependency tree (it pulled in
|
|
``langchain-core`` -> ``langsmith`` -> ``orjson``).
|
|
|
|
A transcription is only as good as its last check against the original, so that check
|
|
lives here rather than in a commit message. **This test skips by default** — langchain is
|
|
deliberately not installed. To run it:
|
|
|
|
uv run --with langchain-text-splitters pytest tests/test_chunking_reference_matches_langchain.py
|
|
|
|
Do that after touching ``chunking_reference.py``, or when bumping the langchain version
|
|
the transcription claims to mirror. It matched byte-for-byte over 20,079 comparisons
|
|
against ``langchain-text-splitters`` 1.1.2 when the dependency was removed.
|
|
"""
|
|
|
|
import random
|
|
|
|
import pytest
|
|
|
|
from hindsight_api.engine.retain.fact_extraction import _RECURSIVE_TEXT_SEPARATORS
|
|
from tests.chunking_reference import recursive_split
|
|
from tests.test_chunking_streams import _PLAIN_TEXT_CASES
|
|
|
|
# Skips the whole module when langchain is absent, which is the normal state.
|
|
langchain_text_splitters = pytest.importorskip(
|
|
"langchain_text_splitters",
|
|
reason="langchain is not a dependency; install it explicitly to re-verify the oracle",
|
|
)
|
|
|
|
|
|
def _langchain_split(text: str, max_chars: int) -> list[str]:
|
|
splitter = langchain_text_splitters.RecursiveCharacterTextSplitter(
|
|
chunk_size=max_chars,
|
|
chunk_overlap=0,
|
|
length_function=len,
|
|
is_separator_regex=False,
|
|
separators=_RECURSIVE_TEXT_SEPARATORS,
|
|
)
|
|
return splitter.split_text(text)
|
|
|
|
|
|
def test_reference_matches_langchain_on_the_separator_tier_corpus():
|
|
"""Every case the streaming tests use, at every size they use."""
|
|
for name, text in _PLAIN_TEXT_CASES.items():
|
|
for max_chars in (10, 17, 40, 100, 1000):
|
|
assert recursive_split(text, max_chars, _RECURSIVE_TEXT_SEPARATORS) == _langchain_split(text, max_chars), (
|
|
f"{name} at max_chars={max_chars}"
|
|
)
|
|
|
|
|
|
def test_reference_matches_langchain_on_fuzzed_documents():
|
|
"""Randomly assembled documents, over the fragment shapes the legacy corpus uses.
|
|
|
|
Seeded so a failure is reproducible; the fragments deliberately include the cases that
|
|
break splitters — empty strings, whitespace runs, CRLF, CJK, and tokens larger than any
|
|
chunk size tried.
|
|
"""
|
|
fragments = [
|
|
"Alpha sentence one. ",
|
|
"Beta two! ",
|
|
"Gamma three? ",
|
|
"delta; ",
|
|
"eps, ",
|
|
"\n\n",
|
|
"\n",
|
|
" ",
|
|
"",
|
|
"文章です。",
|
|
"x" * 50,
|
|
"y" * 200,
|
|
'{"k": "v"}',
|
|
"...",
|
|
"\r\n",
|
|
"\t",
|
|
" ",
|
|
"Ålpha ",
|
|
"a" * 300,
|
|
]
|
|
rng = random.Random(20260902)
|
|
|
|
for case in range(2000):
|
|
document = "".join(rng.choice(fragments) for _ in range(rng.randint(1, 60)))
|
|
max_chars = rng.choice([1, 2, 5, 10, 37, 100, 256, 1500])
|
|
assert recursive_split(document, max_chars, _RECURSIVE_TEXT_SEPARATORS) == _langchain_split(
|
|
document, max_chars
|
|
), f"case {case} at max_chars={max_chars}: {document!r}"
|