Nicolò Boschi
630c3a63e7
* test(dev): add a deterministic retrieval eval for reflect's forced prelude
We had no baseline to judge a retrieval change against, so a change to how
reflect's opening hierarchy (mental models -> observations -> recall) picks
its queries could not be told apart from a regression.
The corpus is authored, never extracted: every stored row's text is
byte-identical to the YAML, which is what makes gold labelling possible at
all. Facts and observations are retained with the mock provider scripted
through set_response_callback; mental models are created with explicit
content; staleness is produced by ORDERING (stale models created before the
facts, fresh ones after) because it is derived, not stored.
Questions are grouped by which layer should answer them -- mm_only,
mm_stale, observations_only, raw_facts_only, multi_layer, near_miss, absent
-- so the eval exercises the descent decision and not just one search. The
facts carry deliberate near misses; a corpus of unrelated facts scores 1.0
for any query and measures nothing.
Scored by rank WITHIN each layer. Set membership saturates once the corpus
is smaller than one recall page, and a flat ranking scores every raw fact
behind every mental model however good the query was. So recall@3 and MRR
per layer, plus the short-circuit fire rate reported separately, since the
mental-model query is what decides it.
First deliverable is the floor/ceiling experiment, which needs no LLM in
either arm: the question verbatim (the real fallback when planning fails)
against a hand-written ideal query. First run:
recall@3 floor=0.929 ceiling=1.000 gap=+0.071
MRR floor=0.762 ceiling=1.000 gap=+0.238
Wording moves retrieval, mostly through rank rather than presence, with the
mm_stale question the largest mover (0.50 -> 1.00) -- the case where a wrong
mental model has to be superseded by raw facts.
Two behaviours it already surfaced, neither introduced here:
- A created-but-never-refreshed mental model is always stale: staleness
resolves from the refresh stamps and an unstamped model is reported stale
unconditionally, so it can never short-circuit the descent until something
refreshes it once.
- One stale model suppresses the short-circuit for all of them, since the
rule requires every returned model to be fresh. An unrelated stale model
in the top-5 keeps the descent going even when a fresh model answers the
question outright.
* test(dev): grade reflect's actual answer, not just what it retrieved
The retrieval eval scores the evidence set, never the prose reflect returns.
That is a necessary condition -- reflect is grounded, so an unretrieved fact
cannot be answered -- and nowhere near a sufficient one.
The blind spot is the category the whole exercise is about. For the mm_stale
question, retrieval can return the stale "Stripe" mental model AND the Adyen
facts, score recall@k = 1.0, and the answer can still say Stripe because the
model trusted the summary over the raw facts. The retrieval tier calls that a
pass.
So answer_eval.py runs reflect_async end to end on a real model and grades
the text with an independent judge, scoring two things separately because
they fail differently:
- correct: meets answer_criteria. Missing it can just mean incomplete.
- trap: asserts must_not_claim, the specific wrong answer the question baits.
That is a grounding failure, and it is the number that matters -- a
confidently wrong answer is worse than a hedged one.
Every question runs N times and the output is a rate, not a verdict. Not for
CI. The judge mirrors tests/llm_judge.py (independent model, majority
confirmation on a "not met") but is reimplemented here because that module
lives under tests/ and is not importable from this package; the eval warns
when the judge and reflect resolve to the same model, since the local .env
makes that the default and a model grading its own output agrees with itself.
First run -- gemini-2.5-flash-lite reflecting, gemini-2.5-flash judging, 2
runs, budget=low, on this branch:
overall correct 93.8%, trap rate 0% on every baited question
multi_layer 50%, everything else 100%
The multi_layer miss is real run-to-run variance rather than a judge
artifact: one run named the platform-team handover, the other dropped it.
This is one arm only. Comparing main against the branch on the same corpus,
model and N is what actually settles the cold-query question, and is not done
here.
* test(dev): hard corpus, failure attribution, and runner env-precedence fixes
WIP checkpoint before A/B testing the thought_signature failure.
* test(dev): stop the outage corpus generating two "April 2026 outage" rows
The numeric_precision cluster cycled months with `i % 12` and years with
`i // 12`, which produced a second row claiming to be THE April 2026 outage
with different values (850 connections / 87 minutes vs the gold's 200 / 47).
The question then had two contradictory answers, and reflect reporting
"conflicting information" -- exactly what its Conflicts and Ambiguity rules
prescribe -- was scored as a failure. The corpus was wrong, not the answer.
I reported it as a reflect defect before checking; it was mine.
Near-misses must differ in what they ASSERT, never in what they claim to BE.
Each outage now owns a distinct (month, year) slot with April 2026 reserved
for the gold row, and `_assert_subjects_are_unique` fails the build when two
rows in a cluster name the same subject -- verified by re-introducing the
collision, which the guard catches.
* fix(reflect): don't manufacture a value for a period the memories don't cover
Asked for an engineering headcount in a year the bank held no data for,
reflect extrapolated backwards from the following year's monthly figures and
answered with a specific number -- calling it "reliably inferred" and
"reliably deduced". Three runs out of three, on gemini-3.7-flash. That is not
a hedge: it is a fabricated data point wearing the language of certainty, and
it is worse than "not recorded" because a reader cannot tell the difference.
The prompts asked for it. Every path that writes an answer said some version
of "if the exact answer isn't stated, use what IS stated to give the best
possible answer", with "only say you don't have information if the retrieved
data is truly unrelated" closing the escape hatch. A neighbouring year IS
related, so declining was effectively disallowed.
The missing distinction: inference may CHARACTERISE what the data covers; it
may not MANUFACTURE a value for something the data does not cover.
_GROUNDING_BOUNDARY states that, and is shared by all three answer paths (the
tool-loop system prompt, the forced-synthesis system prompt, and the
final-synthesis instructions) so they cannot drift apart. It explicitly
preserves qualitative inference, because a rule read as "never infer" would
break the synthesis that makes reflect worth having.
Tests split per the convention: the wiring is deterministic and asserted
directly, including that the rule keeps its teeth and its carve-out. The
behavioural pair is marked hs_llm_core and its docstring says plainly what it
does NOT do -- it does not reproduce the incident (verified: it passes against
the pre-fix prompt on two models), it guards the contract, and its more
valuable half is the check that the rule has not become a refusal reflex.
Reproduction lives in hindsight-dev/benchmarks/prelude (hq-absent-2024).
488 reflect/prompt tests pass; the golden prompt fixture is updated.
* test(dev): accept reflect's accurate UK qualification on the scoped_truth question
The criteria said 2FA is "mandatory for EU (and UK) accounts", which reads as
unconditional. The memory actually says UK accounts are mandatory FROM 2026,
and reflect answered "UK Accounts: Mandatory starting in 2026" -- more precise
than the criteria, and marked wrong for it (1 run in 6).
Scoring a correct answer as a failure is the worse error for a benchmark: it
manufactures a defect to chase. The criteria now accepts any accurate
treatment of the UK and keeps the real assertion, which is that the answer
must be regionally qualified rather than a flat yes or no.
* fix(mental-models): stop delta ops treating the batch-only synthesis as authoritative
The delta refresh writes a synthesis from the new batch alone, then asks a
second call to merge it into the stored page. That call read the synthesis as
evidence: its "a total of 4" counted only the batch and replaced a page's 3
customers with 4; its "no release was deployed" described only the batch and
overwrote a production release recorded one wave earlier.
Label the synthesis UNTRUSTED (only the supporting facts justify an operation)
and add the combine-not-swap, absence-is-not-contradiction and refutation
threshold rules. Replayed against both captured failures: 0/5 -> 5/5.
* test: add hindsight-system-evals, published as a quality metric by the perf monitor
Blackbox quality evals over a real hindsight-api and a real model, through the
published Python client only — the system-tests shape without the stub, since
stubbing the model would score the stub. First suite: knowledge-page
convergence, the eval that found the delta-ops regressions. Each page is
graded twice: correct, and whether it stores the specific baited falsehood.
Runs in perf-test.yml (daily, not on PRs — needs secrets, and one red run is as
likely noise as regression) and publishes correct rate and trap count to the
continuous performance monitor. Seeding uses chunks retain with consolidation
off, so the only model calls are the ones under test: minimum acceptance ~90s,
full ~5 min.
Also: the hindsight-dev knowledge-page tools used to find and replay those
failures (kp_eval, kp_diagnose, replay_gemini, iterate_delta_prompt).
* refactor: move the reflect evals into hindsight-system-evals and trim the new prompt text
Everything from hindsight-dev/benchmarks/prelude now lives in the blackbox
package, driven only through the public client:
- test_02_reflect_answers: the one-shot reflect eval (minimum acceptance is the
2024-headcount incident), with retrieval-vs-reasoning blame from the tool trace;
- debug/diagnose_page: dry-runs a page's second refresh and dumps every traced
prompt; debug/replay_delta_ops: replays one captured delta-ops request per
prompt variant and interrogates the model.
The retrieval floor/ceiling tier is dropped: it measured the planned-prelude
change (#4066), which was closed, and needed engine internals to author layers.
Prompt size, measured against main:
- grounding boundary rewritten compactly and no longer repeated in the final
instructions (the final and reduce calls already carry it in the system
prompt, so it was sent twice): tool loop +131 tokens, final +112;
- delta ops: the combine-not-swap rule removed. Ablation by replaying the two
captured failures: every other piece is load-bearing (dropping the absence or
refutation rules, or the long synthesis paragraph, falls to 2-3/5), this one
is not (5/5 on both without it). +491 tokens.
Full run on the result: 14/14 correct, 0 traps (7 pages, 7 reflect answers).
* test(system-evals): drop an unused property and type the page eval tests
Hindsight system evals
Blackbox quality evals over a real hindsight-api process and a real
model, driven only through the published Python client. No engine imports, no
SQL, no internals — the same rule hindsight-system-tests follows.
Why this is a separate package
hindsight-system-tests points every LLM call at a stub. That is what makes it
deterministic, secret-free, and runnable on fork PRs.
An eval cannot do that. Stub the model and you score the stub. So this package inherits the blackbox shape of the system tests and none of their determinism mechanism, and everything downstream follows from that:
| system-tests | system-evals | |
|---|---|---|
| model | stub | real provider, required |
| secrets | none, runs on forks | needed → not run on PRs |
| assertion | exact equality | judged, and only meaningful as a rate |
| a failure means | a broken mechanism | a quality regression, or sampling noise |
That last row is the one to keep in mind. The same request can produce a destructive edit once and a correct one the next time, so a single red run is a signal to re-run, not proof of a regression.
What it evaluates
Both suites share one corpus and grade twice per question: correct (meets its criteria — can fail on an incomplete answer) and trap (asserts the specific baited falsehood — the one that matters). The trap is asserted first.
test_01 — knowledge-page convergence. A page is created with a source
query and then accumulates: data arrives in waves and each refresh edits what
is already stored. That is where a wrong answer stops being a wrong answer and
becomes a wrong memory. Two regressions found this way, neither visible to a
one-shot reflect:
- a page saying "Release 0.9.3 was deployed to production on 18 March 2026" came back one wave later saying "No release was deployed" — the fact was in an earlier wave, outside the delta window;
- a page counting 3 customers, handed 4 more, reported 4.
Both had one cause: the delta step treated the reflect synthesis as authoritative, when that synthesis is written from the new batch alone.
test_02 — reflect answers. The whole corpus in one bank, one reflect call,
the answer judged. The regression behind it: asked for the 2024 headcount in a
bank covering only 2025-26, reflect extrapolated a number and called it
"reliably deduced". A failure reports whether every gold fact reached the model
(from the tool trace), because a retrieval miss and a reasoning miss need
opposite fixes.
The corpus
hindsight_system_evals/corpus.py generates facts and their gold labels
together, so a label cannot drift from the text it points at. Two properties it
enforces, both learned by getting them wrong:
- Every subject is internally consistent. An earlier version had one release "deployed to production" on three different dates. That is not a hard question, it is a contradiction, and a model superseding it was following its rules.
- Waves never split a subject. All facts about one release travel together. Split across waves, the later batch reads as a correction of the earlier one.
_assert_subjects_are_unique fails the build if two rows in a cluster ever claim
to describe the same thing again.
Where it runs
Not on PRs. It needs provider secrets, and a single red run is as likely to
be sampling noise as a regression. It runs in the system-evals job of
.github/workflows/perf-test.yml — the daily schedule, alongside LoComo and
obs-dedup — and publishes to the
continuous performance monitor
as a quality metric tracked over time: correct rate and trap count, overall and
per suite (by_kind). Traps must stay at 0.
A failing run is still published: for a quality metric the red run is the data
point. scripts/benchmarks/publish-system-evals-results.sh does the push.
Two modes
# minimum acceptance: the cases that have actually regressed
uv run pytest evals
# full: every category — supersession, entity confusion, scoped truth,
# dense absence, numeric precision — plus the page-collapse check
uv run pytest evals --full
# either, writing the JSON the dashboard publishes
uv run pytest evals --full --output system-evals-results.json
The workflow runs full by default; system_evals_mode: minimum on a manual
dispatch runs the small set.
Why seeding costs no model calls
Measured on the first blackbox run, per page: 789s of LLM time on fact
extraction and 199s on consolidation, against 21s for the reflect and delta
calls actually under test. So each bank is configured — through the public config
endpoint, still blackbox — with retain_extraction_mode=chunks (store each item
as written) and consolidation/observations off, and each page refresh is
triggered explicitly. chunks also removes a confound: extraction may paraphrase
a fact, while the gold labels point at the exact authored text.
Running
# the model under test (api key, or vertexai with a service account)
export HINDSIGHT_EVAL_LLM_PROVIDER=gemini
export HINDSIGHT_EVAL_LLM_MODEL=gemini-3.7-flash
export HINDSIGHT_EVAL_LLM_API_KEY=...
# The judge is configured separately ON PURPOSE — a model grading its own output
# agrees with itself. The suite warns when these resolve to the same model.
export HINDSIGHT_EVAL_JUDGE_MODEL=gemini-2.5-flash
export HINDSIGHT_EVAL_JUDGE_API_KEY=... # or HINDSIGHT_EVAL_JUDGE_PROVIDER=vertexai
cd hindsight-system-evals && uv run pytest evals
VertexAI works for both: set HINDSIGHT_EVAL_LLM_PROVIDER=vertexai and
HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY / _PROJECT_ID — the judge
falls back to the same service account when no judge key is set. That is how the
perf workflow runs it. HINDSIGHT_EVAL_REFLECT_BUDGET (default low) sets the
reflect budget for test_02.
If your shell exports PYTEST_ADDOPTS with -n (the repo .env does), unset
it: xdist is not installed here, and parallel evals against one server would
compete for it anyway.
The server runs on its own pg0 instance (hindsight-system-evals), so a run does
not compete for connections with a developer's server or with the system tests.
Debugging a failure
Each assertion prints the bank id, the page id (or the reflect queries and whether the gold evidence arrived), and what the page or answer actually reads. The bank is left in place, so it can be opened in the control plane.
For a knowledge page that goes wrong, two tools, both through the public API:
# 1. Where does it go wrong? Builds wave 1, then dry-runs the second refresh N
# times: raw synthesis vs the document after the delta operations. --dump
# writes each traced prompt verbatim.
uv run python -m hindsight_system_evals.debug.diagnose_page \
--question hq-count-billing --repeats 3 --dump captures/
# 2. Is it the prompt? Replays one captured delta-ops request N times per
# system-prompt variant and scores the resulting document; --interrogate
# asks the model which lines misled it.
uv run python -m hindsight_system_evals.debug.replay_delta_ops \
--input captures/hq-count-billing-mental_model_delta_ops-<n>.json \
--must-keep "Denning|Barrow|Fairbank" --variants variants.json
A variant that wins the replay is a lead; it still has to pass the eval.