Nicolò Boschi
1d5f852809
* test(system): blackbox system-test suite driven only through the public API The api-slim suite is ~500 files, each covering one mechanism, which leaves composition bugs — consolidation wiping facts, a delta refresh missing a backdated window, a transfer dropping evidence — with nothing watching them. This adds the layer that does: tests that drive a real `hindsight-api` process over HTTP through the published Python client, with no engine access and no SQL. Determinism comes from a stub server implementing the OpenAI chat-completions/embeddings APIs and a Cohere-compatible rerank endpoint. The server under test is pointed at it with ordinary environment variables (`HINDSIGHT_API_LLM_BASE_URL`, `..._EMBEDDINGS_OPENAI_BASE_URL`, `..._RERANKER_SILICONFLOW_BASE_URL`), so no production code changes are needed and the real provider transport — the OpenAI client, JSON repair, retries, structured output — is exercised for real rather than replaced by a fake. Design decisions worth knowing: - Rules match on a named pipeline step, not the request body. Nothing on the wire identifies the caller (the json_schema name is the constant "response", and the soft json_object path sends no schema at all), so a step is a short anchor phrase from its prompt — owned by `steps.py` so a prompt edit is one line, not thirty red tests. - An unmatched call fails the test and prints the rule to paste in. `MockLLM` synthesizes plausible facts from its input when it doesn't recognize a call, which is why tests using it pass without proving anything. - Requests are validated strictly. A fake you control drifts permissive, and the provider bugs that have actually hurt here (Bedrock rejecting response_format, Azure 400ing on prompt_cache_key) are all "the provider refused our request". - Background work stays on. Consolidation runs in the worker after retain returns; tests wait for it via the operations API rather than disabling it, because that asynchronous half is where the composition bugs live. - The server runs from a scratch dir holding an empty `.env`, since a discovered `.env` deliberately overrides the ambient environment (#2961) and would otherwise silently replace the whole test configuration. CI (`test-system`) needs no provider secrets, so unlike every `test-api` job it also runs on fork PRs — and with embeddings and reranking both stubbed, nothing loads sentence-transformers, so it skips torch and the HuggingFace cache too. * test(system): pin the whole recall payload, not just a keyword With the LLM, embedder and reranker all stubbed, a recall is a pure function of its input — so assert it as one. Ranking order, the rendered fact text, the document/chunk identity composite, the temporal fields, the empty envelope sections, and each of the four scores are now pinned. The three retrieval components are reproducible to the bit across runs; only `final` drifts (~1e-9), because it folds in recency measured against wall-clock now, so it gets a tolerance and a comment saying why. The previous assertion ("Berlin appears somewhere") passed just as happily with fusion inverted, the reranker contributing nothing, or the temporal fields silently stopping being parsed. * ci: skip the two unconditional jobs for a system-tests-only change Iterating on hindsight-system-tests/ costs ~6 minutes of CI against a 36-second test. Of the 96 jobs, 91 already skip for a change confined to that package; the remainder were build-docs (2.2 min) and verify-generated-files (3.6 min), both deliberately unconditional. Both now hang off a new `outside-system-tests` filter rather than a positive one, so they still run for every other change in the repo and step aside only for a PR touching nothing but the system-test package — which no generated file is produced from and the docs site never reads. The filter is a lone negation on purpose: paths-filter builds one matcher per pattern and ORs them, so the natural-looking ['**', '!dir/**'] pair matches every file and leaves the filter permanently true. Verified against picomatch directly before committing.
Hindsight system tests
Blackbox tests over a real hindsight-api process, driven only through the
published Python client. No engine imports, no SQL, no internals.
Why this package exists
hindsight-api-slim/tests/ holds ~500 files, each covering one mechanism. That
catches mechanism bugs. It does not catch composition bugs — consolidation
wiping facts, a delta refresh missing a backdated window, a transfer dropping
evidence — because no single-mechanism test spans the steps where those live.
These tests do.
How determinism works
A stub server implements the OpenAI chat-completions and embeddings APIs plus a Cohere-compatible rerank endpoint. The server under test is pointed at it with ordinary environment variables, so no production code changes are needed, and the real provider transport (JSON repair, retries, structured output) is exercised for real.
- Rules match on content, not request bodies — the requested JSON-schema name and prompt substrings — so editing a prompt does not break every test.
- An unmatched call fails the test with the rule to paste in. Nothing is ever answered by a plausible-looking default.
- Requests are validated strictly. If Hindsight sends something real OpenAI would 400, so does the stub.
- Embeddings are lexical and deterministic, so the suite needs no torch and no model download.
Running
# once: the server needs pg0, and nothing else beyond its base dependencies
(cd ../hindsight-api-slim && uv sync --frozen --extra embedded-db)
cd hindsight-system-tests
uv run pytest tests -v
Note the missing extra: with embeddings and reranking pointed at the stub,
nothing loads sentence-transformers, so the suite runs with no torch and no
model download. CI (test-system) needs no provider secrets either, which
means it runs on fork PRs — unlike every test-api job.
The fixtures start their own embedded Postgres (pg0://hindsight-systest:15499),
separate from the dev database and from the api-slim suite's, and run the server
from a scratch directory holding an empty .env so your own .env cannot leak
into the test configuration.
If your shell exports PYTEST_ADDOPTS=-n ..., clear it for this suite —
pytest-xdist is not installed here, and the session-scoped server makes it
pointless anyway.
Background work is off by default. Observation extraction and auto consolidation run in the worker after a retain returns, so their LLM calls would land at a moment no test controls — after the assertions, sometimes after the next test has started. Both are per-bank settings, so a story about either switches it on for its own bank and waits for the operation to finish.
Conventions
- One story per file, named
test_NN_<story>.py. The number is reading order for a human, not execution order — every test must pass when run alone. - Assert through the client's responses only.