skills/vectorize-io__hindsight
1
0
Fork 0
mirror of https://github.com/vectorize-io/hindsight.git synced 2026-09-14 19:31:49 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(reflect): grounding defects in reflect and knowledge-page delta refresh, plus system-evals (#4304)

Browse Source 
* test(dev): add a deterministic retrieval eval for reflect's forced prelude

We had no baseline to judge a retrieval change against, so a change to how
reflect's opening hierarchy (mental models -> observations -> recall) picks
its queries could not be told apart from a regression.

The corpus is authored, never extracted: every stored row's text is
byte-identical to the YAML, which is what makes gold labelling possible at
all. Facts and observations are retained with the mock provider scripted
through set_response_callback; mental models are created with explicit
content; staleness is produced by ORDERING (stale models created before the
facts, fresh ones after) because it is derived, not stored.

Questions are grouped by which layer should answer them -- mm_only,
mm_stale, observations_only, raw_facts_only, multi_layer, near_miss, absent
-- so the eval exercises the descent decision and not just one search. The
facts carry deliberate near misses; a corpus of unrelated facts scores 1.0
for any query and measures nothing.

Scored by rank WITHIN each layer. Set membership saturates once the corpus
is smaller than one recall page, and a flat ranking scores every raw fact
behind every mental model however good the query was. So recall@3 and MRR
per layer, plus the short-circuit fire rate reported separately, since the
mental-model query is what decides it.

First deliverable is the floor/ceiling experiment, which needs no LLM in
either arm: the question verbatim (the real fallback when planning fails)
against a hand-written ideal query. First run:

    recall@3  floor=0.929  ceiling=1.000  gap=+0.071
    MRR       floor=0.762  ceiling=1.000  gap=+0.238

Wording moves retrieval, mostly through rank rather than presence, with the
mm_stale question the largest mover (0.50 -> 1.00) -- the case where a wrong
mental model has to be superseded by raw facts.

Two behaviours it already surfaced, neither introduced here:

- A created-but-never-refreshed mental model is always stale: staleness
  resolves from the refresh stamps and an unstamped model is reported stale
  unconditionally, so it can never short-circuit the descent until something
  refreshes it once.
- One stale model suppresses the short-circuit for all of them, since the
  rule requires every returned model to be fresh. An unrelated stale model
  in the top-5 keeps the descent going even when a fresh model answers the
  question outright.

* test(dev): grade reflect's actual answer, not just what it retrieved

The retrieval eval scores the evidence set, never the prose reflect returns.
That is a necessary condition -- reflect is grounded, so an unretrieved fact
cannot be answered -- and nowhere near a sufficient one.

The blind spot is the category the whole exercise is about. For the mm_stale
question, retrieval can return the stale "Stripe" mental model AND the Adyen
facts, score recall@k = 1.0, and the answer can still say Stripe because the
model trusted the summary over the raw facts. The retrieval tier calls that a
pass.

So answer_eval.py runs reflect_async end to end on a real model and grades
the text with an independent judge, scoring two things separately because
they fail differently:

- correct: meets answer_criteria. Missing it can just mean incomplete.
- trap: asserts must_not_claim, the specific wrong answer the question baits.
  That is a grounding failure, and it is the number that matters -- a
  confidently wrong answer is worse than a hedged one.

Every question runs N times and the output is a rate, not a verdict. Not for
CI. The judge mirrors tests/llm_judge.py (independent model, majority
confirmation on a "not met") but is reimplemented here because that module
lives under tests/ and is not importable from this package; the eval warns
when the judge and reflect resolve to the same model, since the local .env
makes that the default and a model grading its own output agrees with itself.

First run -- gemini-2.5-flash-lite reflecting, gemini-2.5-flash judging, 2
runs, budget=low, on this branch:

    overall correct 93.8%, trap rate 0% on every baited question
    multi_layer 50%, everything else 100%

The multi_layer miss is real run-to-run variance rather than a judge
artifact: one run named the platform-team handover, the other dropped it.

This is one arm only. Comparing main against the branch on the same corpus,
model and N is what actually settles the cold-query question, and is not done
here.

* test(dev): hard corpus, failure attribution, and runner env-precedence fixes

WIP checkpoint before A/B testing the thought_signature failure.

* test(dev): stop the outage corpus generating two "April 2026 outage" rows

The numeric_precision cluster cycled months with `i % 12` and years with
`i // 12`, which produced a second row claiming to be THE April 2026 outage
with different values (850 connections / 87 minutes vs the gold's 200 / 47).

The question then had two contradictory answers, and reflect reporting
"conflicting information" -- exactly what its Conflicts and Ambiguity rules
prescribe -- was scored as a failure. The corpus was wrong, not the answer.
I reported it as a reflect defect before checking; it was mine.

Near-misses must differ in what they ASSERT, never in what they claim to BE.
Each outage now owns a distinct (month, year) slot with April 2026 reserved
for the gold row, and `_assert_subjects_are_unique` fails the build when two
rows in a cluster name the same subject -- verified by re-introducing the
collision, which the guard catches.

* fix(reflect): don't manufacture a value for a period the memories don't cover

Asked for an engineering headcount in a year the bank held no data for,
reflect extrapolated backwards from the following year's monthly figures and
answered with a specific number -- calling it "reliably inferred" and
"reliably deduced". Three runs out of three, on gemini-3.7-flash. That is not
a hedge: it is a fabricated data point wearing the language of certainty, and
it is worse than "not recorded" because a reader cannot tell the difference.

The prompts asked for it. Every path that writes an answer said some version
of "if the exact answer isn't stated, use what IS stated to give the best
possible answer", with "only say you don't have information if the retrieved
data is truly unrelated" closing the escape hatch. A neighbouring year IS
related, so declining was effectively disallowed.

The missing distinction: inference may CHARACTERISE what the data covers; it
may not MANUFACTURE a value for something the data does not cover.
_GROUNDING_BOUNDARY states that, and is shared by all three answer paths (the
tool-loop system prompt, the forced-synthesis system prompt, and the
final-synthesis instructions) so they cannot drift apart. It explicitly
preserves qualitative inference, because a rule read as "never infer" would
break the synthesis that makes reflect worth having.

Tests split per the convention: the wiring is deterministic and asserted
directly, including that the rule keeps its teeth and its carve-out. The
behavioural pair is marked hs_llm_core and its docstring says plainly what it
does NOT do -- it does not reproduce the incident (verified: it passes against
the pre-fix prompt on two models), it guards the contract, and its more
valuable half is the check that the rule has not become a refusal reflex.
Reproduction lives in hindsight-dev/benchmarks/prelude (hq-absent-2024).

488 reflect/prompt tests pass; the golden prompt fixture is updated.

* test(dev): accept reflect's accurate UK qualification on the scoped_truth question

The criteria said 2FA is "mandatory for EU (and UK) accounts", which reads as
unconditional. The memory actually says UK accounts are mandatory FROM 2026,
and reflect answered "UK Accounts: Mandatory starting in 2026" -- more precise
than the criteria, and marked wrong for it (1 run in 6).

Scoring a correct answer as a failure is the worse error for a benchmark: it
manufactures a defect to chase. The criteria now accepts any accurate
treatment of the UK and keeps the real assertion, which is that the answer
must be regionally qualified rather than a flat yes or no.

* fix(mental-models): stop delta ops treating the batch-only synthesis as authoritative

The delta refresh writes a synthesis from the new batch alone, then asks a
second call to merge it into the stored page. That call read the synthesis as
evidence: its "a total of 4" counted only the batch and replaced a page's 3
customers with 4; its "no release was deployed" described only the batch and
overwrote a production release recorded one wave earlier.

Label the synthesis UNTRUSTED (only the supporting facts justify an operation)
and add the combine-not-swap, absence-is-not-contradiction and refutation
threshold rules. Replayed against both captured failures: 0/5 -> 5/5.

* test: add hindsight-system-evals, published as a quality metric by the perf monitor

Blackbox quality evals over a real hindsight-api and a real model, through the
published Python client only — the system-tests shape without the stub, since
stubbing the model would score the stub. First suite: knowledge-page
convergence, the eval that found the delta-ops regressions. Each page is
graded twice: correct, and whether it stores the specific baited falsehood.

Runs in perf-test.yml (daily, not on PRs — needs secrets, and one red run is as
likely noise as regression) and publishes correct rate and trap count to the
continuous performance monitor. Seeding uses chunks retain with consolidation
off, so the only model calls are the ones under test: minimum acceptance ~90s,
full ~5 min.

Also: the hindsight-dev knowledge-page tools used to find and replay those
failures (kp_eval, kp_diagnose, replay_gemini, iterate_delta_prompt).

* refactor: move the reflect evals into hindsight-system-evals and trim the new prompt text

Everything from hindsight-dev/benchmarks/prelude now lives in the blackbox
package, driven only through the public client:
- test_02_reflect_answers: the one-shot reflect eval (minimum acceptance is the
  2024-headcount incident), with retrieval-vs-reasoning blame from the tool trace;
- debug/diagnose_page: dry-runs a page's second refresh and dumps every traced
  prompt; debug/replay_delta_ops: replays one captured delta-ops request per
  prompt variant and interrogates the model.
The retrieval floor/ceiling tier is dropped: it measured the planned-prelude
change (#4066), which was closed, and needed engine internals to author layers.

Prompt size, measured against main:
- grounding boundary rewritten compactly and no longer repeated in the final
  instructions (the final and reduce calls already carry it in the system
  prompt, so it was sent twice): tool loop +131 tokens, final +112;
- delta ops: the combine-not-swap rule removed. Ablation by replaying the two
  captured failures: every other piece is load-bearing (dropping the absence or
  refutation rules, or the long synthesis paragraph, falls to 2-3/5), this one
  is not (5/5 on both without it). +491 tokens.

Full run on the result: 14/14 correct, 0 traps (7 pages, 7 reflect answers).

* test(system-evals): drop an unused property and type the page eval tests
This commit is contained in:
Nicolò Boschi
2026-09-11 11:41:34 +02:00
committed by GitHub
parent cabfbb9fe0
commit 630c3a63e7
24 changed files with 4237 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/perf-test.yml
+94
View File
@@ -47,6 +47,17 @@ on:
description: "Obs benchmark fraction (0-1] of each document to run."
type: string
default: "1.0"
system_evals_skip:
description: "Skip the system-evals quality job (knowledge-page convergence)"
type: boolean
default: false
system_evals_mode:
description: "system-evals mode: full (every category) or minimum (the two cases that have regressed)."
type: choice
options:
- full
- minimum
default: full
ref:
description: "Git ref to test (branch, tag, or SHA). Defaults to main."
type: string
@@ -295,3 +306,86 @@ jobs:
PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: ./scripts/benchmarks/publish-obs-results.sh hindsight-dev/obs-results.json
system-evals:
# Knowledge-page convergence, graded by an independent judge. Blackbox: a real
# hindsight-api process driven only through the published Python client, the
# same shape as hindsight-system-tests — except the model is REAL, because a
# quality eval against a stub would score the stub. That also means it needs
# provider secrets, which is why it lives here and not in the PR workflow.
if: inputs.system_evals_skip != true
runs-on: ubuntu-latest
timeout-minutes: 60
env:
HINDSIGHT_EVAL_LLM_PROVIDER: vertexai
HINDSIGHT_EVAL_LLM_MODEL: google/gemini-2.5-flash-lite
HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: /tmp/gcp-credentials.json
# A different model from the one under test: a model grading its own output
# agrees with itself.
HINDSIGHT_EVAL_JUDGE_PROVIDER: vertexai
HINDSIGHT_EVAL_JUDGE_MODEL: google/gemini-2.5-flash
steps:
- uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup GCP credentials
run: |
printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > /tmp/gcp-credentials.json
PROJECT_ID=$(jq -r '.project_id' /tmp/gcp-credentials.json)
echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> $GITHUB_ENV
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true
prune-cache: false
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version-file: ".python-version"
- name: Cache HuggingFace models
uses: actions/cache@v5
with:
path: ~/.cache/huggingface
key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-huggingface-
- name: Install server and eval dependencies
# The server under test keeps its default local embeddings and reranker (only
# the LLM is real-provider), so it needs the full extras, unlike test-system.
run: |
(cd hindsight-api-slim && uv sync --frozen --all-extras --index-strategy unsafe-best-match)
(cd hindsight-system-evals && uv sync)
- name: Run system evals
run: |
MODE="${{ inputs.system_evals_mode }}"
FLAG="--full"
if [ "$MODE" = "minimum" ]; then FLAG=""; fi
cd hindsight-system-evals
uv run pytest evals $FLAG --output system-evals-results.json
- name: Upload system-evals results
if: always()
uses: actions/upload-artifact@v7
with:
name: system-evals-results-${{ github.sha }}
path: hindsight-system-evals/system-evals-results.json
retention-days: 90
- name: Publish system-evals to dashboard
# Published even when an eval fails: for a quality metric the red run IS the
# data point, and dropping it would hide exactly the regressions it exists to
# show. Skipped only when nothing was written (the run died before grading).
if: >-
always() &&
hashFiles('hindsight-system-evals/system-evals-results.json') != '' &&
(github.event_name == 'schedule' || github.event_name == 'push' || github.event_name == 'workflow_dispatch')
env:
PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: ./scripts/benchmarks/publish-system-evals-results.sh hindsight-system-evals/system-evals-results.json
hindsight-api-slim/hindsight_api/engine/reflect/prompts.py
+60 -5
View File
@@ -240,6 +240,8 @@ def build_system_prompt_for_tools(
"- Be a thoughtful interpreter, not just a literal repeater",
"- When the exact answer isn't stated, use what IS stated to give a best-effort answer AND surface any uncertainty — never invent confidence the data doesn't support.",
"",
_GROUNDING_BOUNDARY,
"",
"## Temporal Reasoning",
"Every memory and observation carries temporal fields in the JSON tool result:",
"- `mentioned_at` — when the user retained the fact (always set).",
@@ -547,11 +549,36 @@ _SPLIT_SYNTHESIS_WARN_CHUNKS = 4
#: safe for any real model, so the floor caps fan-out without dropping data.
_MIN_SPLIT_CHUNK_TOKENS = 1024
#: The line between synthesis and invention, shared by every path that writes an
#: answer (the tool-loop system prompt, the forced-synthesis system prompt, and
#: the final-synthesis instructions) so they cannot drift apart.
#:
#: Reflect is told throughout to infer rather than repeat literally, which is
#: what makes it useful. But "if the exact answer isn't stated, use what IS
#: stated" has no floor: asked for a headcount in a year the bank does not cover,
#: a model extrapolated backwards from the following year's growth trend and
#: reported a specific number as "reliably deduced". That is not a hedge — it is
#: a fabricated data point wearing the language of certainty, and it is worse
#: than "not recorded" because a reader cannot tell the difference.
#:
#: The distinction that holds: inference may CHARACTERISE what the data covers;
#: it may not MANUFACTURE a value for something the data does not cover.
_GROUNDING_BOUNDARY = (
"## What Counts As Inference\n"
"Infer freely about what the retrieved data covers. Never produce a value (number, date, name, "
"status, amount) for a period, entity or person the data does not cover: extrapolating a trend, "
"interpolating between dated facts, or borrowing from a similar entity is invention. If no fact "
"states the value for the thing asked, say the data does not record it (a complete answer), then "
"give what IS recorded, labelled with the period or entity it belongs to. Never call a derived "
"value exact, reliable, deduced or confirmed; label any derivation an estimate. Qualitative "
"inference is unaffected."
)
_FINAL_INSTRUCTIONS = (
"Provide a thoughtful answer by synthesizing and reasoning from the retrieved data above. "
"You can make reasonable inferences from the memories, but don't completely fabricate information. "
"If the exact answer isn't stated, use what IS stated to give the best possible answer. "
"Only say 'I don't have information' if the retrieved data is truly unrelated to the question.\n\n"
"If the exact answer isn't stated, use what IS stated to give the best possible answer, "
"within the inference rules in the system prompt.\n\n"
"IMPORTANT: Output ONLY the final answer. Do NOT include meta-commentary like "
'"I\'ll search..." or "Let me analyze...". Do NOT explain your reasoning process. '
"Just provide the direct synthesized answer."
@@ -874,7 +901,7 @@ Your approach:
- Be helpful - if you have related information, use it to give the best possible answer
- ONLY use information from tool results - no external knowledge or guessing
Only say "I don't have information" if the retrieved data is truly unrelated to the question.
{grounding_boundary}
FORMATTING: Use proper markdown formatting in your answer:
- Headers (##, ###) for sections
@@ -942,7 +969,7 @@ def build_final_system_prompt(
role_section = escape_for_prompt(mission.strip()) if mission else _DEFAULT_FINAL_ROLE
parts = [build_directives_section(directives) if directives else ""]
parts.append(_FINAL_SYSTEM_PROMPT_BASE.format(role_section=role_section))
parts.append(_FINAL_SYSTEM_PROMPT_BASE.format(role_section=role_section, grounding_boundary=_GROUNDING_BOUNDARY))
parts.append(default_language_section(_FINAL_LANGUAGE_RULE, llm_output_language))
parts.append(build_directives_reminder(directives) if directives else "")
# Volatile "now" reference last, so the static/per-bank instructions above
@@ -966,9 +993,23 @@ You will be given:
(1..6) and an ordered list of ``blocks``. Each block has a stable ``id`` and
a ``text`` field holding one markdown fragment — a paragraph, a list, a
table, or a fenced code block.
3. NEW INFORMATION SYNTHESIS (markdown) — a synthesis showing how the new facts
3. NEW INFORMATION SYNTHESIS (markdown) — UNTRUSTED. Prose written by another
model that saw ONLY the supporting facts below. It is a reading aid, not
evidence, and it is frequently wrong about what exists: it says things like
"no X was found" or "a total of N" when X is merely absent from this batch
and N counts only this batch. NEVER edit the document on the strength of a
sentence in the synthesis — only the SUPPORTING FACTS justify an operation.
A synthesis showing how the new facts
relate to the document's topic. Use it to understand context and relevance,
but do NOT copy its formatting or wording wholesale.
It was written from the SUPPORTING FACTS BELOW AND NOTHING ELSE. It could not
see the current document or any earlier fact, so every count, total, list or
summary in it describes ONLY the new facts — never the topic as a whole.
"A total of 4 customers..." in the synthesis means four in this batch, not
four altogether. Such a figure NEVER contradicts a different figure in the
document: the document counted what it could see, the synthesis counted what
it could see, and the answer is usually the two combined. Likewise the
synthesis saying nothing about something is not evidence against it.
4. SUPPORTING FACTS — observations and facts created since the last refresh.
These are genuinely new — they were NOT available when the current document
was written.
@@ -1005,6 +1046,20 @@ RULES
- **Update** existing content with ``replace_block`` or ``replace_section_blocks``
when new facts provide corrections, updates, or more specific information
about topics already in the document.
- **Absence is not contradiction**: an entity, count or detail missing from
SUPPORTING FACTS is NOT thereby wrong, superseded or removed. The facts are one
batch, not the whole memory — the document was built from facts you cannot see.
"The batch does not mention X" and "X did not happen" are different statements,
and only the second would justify an edit. This applies to the SYNTHESIS too: if
it reports that something is absent, unrecorded or not found, that is a
statement about the batch, never about the topic.
- **Refutation threshold for removal or overwrite**: you may only remove or
overwrite existing text when a SUPPORTING FACT explicitly refutes or corrects
that exact detail, OR is a later statement about the same facet (a status,
count, owner or location that has since changed). Failing both tests, keep the
existing text: use ``append_block`` / ``insert_block``, or re-emit the block
with the new detail merged into a cohesive statement that still carries the old
one. Combining two disjoint sets is a merge, never a replacement.
- **Remove** content with ``remove_block`` or ``remove_section`` ONLY when
the new facts explicitly contradict or supersede it.
- Prefer the *smallest* operation that expresses the change: appending or
hindsight-api-slim/tests/test_reflect_grounding_behaviour.py
+106
View File
@@ -0,0 +1,106 @@
"""Reflect declines to invent a value for a period its memories do not cover.
The behavioural half of the grounding boundary (see
``test_reflect_grounding_boundary.py`` for the prompt wiring). Whether a model
actually follows the rule cannot be simulated with MockLLM and cannot be asserted
by string matching, so this drives the real synthesis prompt and judges the
answer.
**Scope, stated honestly.** These two cases do NOT reproduce the incident that
prompted the fix. Verified: run against the pre-fix prompt, on two different
models, both still pass. The failure needed the full agent path — a real bank,
the tool loop, hundreds of competing rows — and over a five-row fixture the model
declines to extrapolate with or without the rule.
So this guards the *contract*, not the *incident*:
- that a clearly uncovered period is not answered with a manufactured figure, and
- that the rule does not become a refusal reflex, which is the regression a rule
like this most plausibly causes and the more valuable of the two guards.
The incident itself is reproduced end to end by
``hindsight-system-evals`` (``evals/test_02_reflect_answers.py``, question
``hq-absent-2024``), which is what caught the bug and what confirmed the fix.
"""
import pytest
from hindsight_api import LLMConfig
from hindsight_api.engine.reflect.prompts import _FINAL_INSTRUCTIONS, build_final_system_prompt
from tests.llm_judge import assert_meets_criteria
pytestmark = pytest.mark.hs_llm_core
# Deliberately adjacent, never overlapping: every figure is 2025 or later, so a
# 2024 answer can only come from extrapolation. The steady monthly climb is the
# bait — it makes a backward projection look arithmetically respectable.
_RETRIEVED_DATA = """
### From recall:
```json
{
"memories": [
{"id": "m1", "text": "Engineering headcount reached 40 at the end of January 2025."},
{"id": "m2", "text": "Engineering headcount reached 47 at the end of February 2025."},
{"id": "m3", "text": "Engineering headcount reached 54 at the end of March 2025."},
{"id": "m4", "text": "Engineering headcount reached 61 at the end of April 2025."},
{"id": "m5", "text": "The engineering org was reorganised into four groups in June 2025."}
]
}
```
"""
async def _answer(question: str) -> str:
"""Run the real forced-synthesis prompt pair over the fixture above."""
llm_config = LLMConfig.from_env()
result = await llm_config.call(
messages=[
{"role": "system", "content": build_final_system_prompt(None, None, None)},
{
"role": "user",
"content": (
f"## Retrieved Data (synthesize and reason from this data)\n{_RETRIEVED_DATA}\n"
f"## Question\n{question}\n\n## Instructions\n{_FINAL_INSTRUCTIONS}"
),
},
],
scope="reflect",
temperature=0.0,
)
return str(result.content or "").strip()
class TestUncoveredPeriods:
@pytest.mark.asyncio
async def test_does_not_invent_a_figure_for_an_uncovered_year(self):
answer = await _answer("What was the engineering headcount at the end of 2024?")
await assert_meets_criteria(
response=answer,
criteria=(
"The response says the data does not record an engineering headcount for 2024. It does "
"NOT assert a specific 2024 headcount number, and does not describe any number it "
"mentions for 2024 as exact, reliable, deduced or confirmed. Quoting the 2025 figures, "
"clearly labelled as 2025, is acceptable; so is an explicitly-labelled estimate that is "
"not presented as the answer."
),
context=(
"The retrieved memories contain monthly engineering headcount figures for January "
"through April 2025 and nothing at all for 2024. The question asked for the 2024 "
"figure. Extrapolating backwards from the 2025 trend would produce a plausible number "
"that no memory states."
),
)
@pytest.mark.asyncio
async def test_still_answers_what_the_data_does_cover(self):
"""The boundary must not turn into a refusal reflex.
A rule against inventing values is worthless if it also suppresses answers
the data fully supports — that trade would be a worse regression than the
bug it fixes.
"""
answer = await _answer("What was the engineering headcount at the end of March 2025?")
await assert_meets_criteria(
response=answer,
criteria="The response states that the engineering headcount at the end of March 2025 was 54.",
context="The retrieved memories state the March 2025 headcount explicitly.",
)
hindsight-api-slim/tests/test_reflect_grounding_boundary.py
+65
View File
@@ -0,0 +1,65 @@
"""Reflect must not manufacture a value for something its data does not cover.
Reflect is told throughout to infer rather than repeat literally, which is what
makes it useful. But "if the exact answer isn't stated, use what IS stated" had
no floor. Asked for a headcount in a year the bank did not cover, the model
extrapolated backwards from the following year's growth and reported a specific
number as "reliably deduced" — a fabricated data point wearing the language of
certainty, which is worse than "not recorded" because a reader cannot tell the
difference.
Split per the testing convention: the prompt wiring is deterministic and asserted
directly here; whether a model actually FOLLOWS the rule is not, so it gets one
judge test against a real LLM.
"""
import pytest
from hindsight_api.engine.reflect.prompts import (
_FINAL_INSTRUCTIONS,
_GROUNDING_BOUNDARY,
build_final_prompt,
build_final_system_prompt,
build_system_prompt_for_tools,
)
class TestGroundingBoundaryWiring:
"""Deterministic: every path that writes an answer carries the rule."""
def test_tool_loop_system_prompt_carries_it(self):
prompt = build_system_prompt_for_tools({"name": "Bank", "mission": "testing"})
assert _GROUNDING_BOUNDARY in prompt
def test_forced_synthesis_system_prompt_carries_it(self):
assert _GROUNDING_BOUNDARY in build_final_system_prompt("testing", None, None)
def test_final_call_carries_it_exactly_once(self):
"""The final and reduce calls send the final system prompt, which carries
the rule. An earlier version also embedded it in the user-side
instructions, so one call carried the same text twice."""
user = build_final_prompt("q", [], {"name": "Bank"})
assert _GROUNDING_BOUNDARY not in _FINAL_INSTRUCTIONS
assert _GROUNDING_BOUNDARY not in user
assert build_final_system_prompt("testing", None, None).count(_GROUNDING_BOUNDARY) == 1
@pytest.mark.parametrize(
"phrasing",
["extrapolating a trend", "does not record it", "Never call a derived"],
)
def test_rule_states_the_actual_constraint(self, phrasing: str):
"""Guards the substance, not just that some text is present.
A rule that survives as a heading while its teeth are edited out would
otherwise keep every wiring test above green.
"""
assert phrasing in _GROUNDING_BOUNDARY
def test_qualitative_inference_is_explicitly_preserved(self):
"""The rule must not read as "never infer" — that would break synthesis.
Reflect's value is connecting memories. The boundary is about
manufacturing values, so it says so out loud; without this carve-out a
model reasonably reads the section as a blanket prohibition.
"""
assert "Qualitative inference is unaffected" in _GROUNDING_BOUNDARY
hindsight-api-slim/tests/test_reflect_prompt_builder.py
+3
View File
@@ -59,6 +59,9 @@ _LANGUAGE_AND_RULES = """\
- Be a thoughtful interpreter, not just a literal repeater
- When the exact answer isn't stated, use what IS stated to give a best-effort answer AND surface any uncertainty — never invent confidence the data doesn't support.
## What Counts As Inference
Infer freely about what the retrieved data covers. Never produce a value (number, date, name, status, amount) for a period, entity or person the data does not cover: extrapolating a trend, interpolating between dated facts, or borrowing from a similar entity is invention. If no fact states the value for the thing asked, say the data does not record it (a complete answer), then give what IS recorded, labelled with the period or entity it belongs to. Never call a derived value exact, reliable, deduced or confirmed; label any derivation an estimate. Qualitative inference is unaffected.
## Temporal Reasoning
Every memory and observation carries temporal fields in the JSON tool result:
- `mentioned_at` — when the user retained the fact (always set).
hindsight-system-evals/README.md
+161
View File
@@ -0,0 +1,161 @@
# Hindsight system evals
Blackbox **quality** evals over a real `hindsight-api` process and a **real
model**, driven only through the published Python client. No engine imports, no
SQL, no internals — the same rule `hindsight-system-tests` follows.
## Why this is a separate package
`hindsight-system-tests` points every LLM call at a stub. That is what makes it
deterministic, secret-free, and runnable on fork PRs.
An eval cannot do that. Stub the model and you score the stub. So this package
inherits the blackbox *shape* of the system tests and none of their determinism
mechanism, and everything downstream follows from that:
| | system-tests | system-evals |
|---|---|---|
| model | stub | real provider, required |
| secrets | none, runs on forks | needed → **not run on PRs** |
| assertion | exact equality | judged, and only meaningful as a rate |
| a failure means | a broken mechanism | a quality regression, or sampling noise |
That last row is the one to keep in mind. The same request can produce a
destructive edit once and a correct one the next time, so a single red run is a
signal to re-run, not proof of a regression.
## What it evaluates
Both suites share one corpus and grade twice per question: **correct** (meets
its criteria — can fail on an incomplete answer) and **trap** (asserts the
specific baited falsehood — the one that matters). The trap is asserted first.
**`test_01` — knowledge-page convergence.** A page is created with a source
query and then *accumulates*: data arrives in waves and each refresh edits what
is already stored. That is where a wrong answer stops being a wrong answer and
becomes a wrong *memory*. Two regressions found this way, neither visible to a
one-shot reflect:
- a page saying *"Release 0.9.3 was deployed to production on 18 March 2026"*
came back one wave later saying *"No release was deployed"* — the fact was in
an earlier wave, outside the delta window;
- a page counting 3 customers, handed 4 more, reported **4**.
Both had one cause: the delta step treated the reflect synthesis as
authoritative, when that synthesis is written from the new batch alone.
**`test_02` — reflect answers.** The whole corpus in one bank, one reflect call,
the answer judged. The regression behind it: asked for the 2024 headcount in a
bank covering only 2025-26, reflect extrapolated a number and called it
"reliably deduced". A failure reports whether every gold fact reached the model
(from the tool trace), because a retrieval miss and a reasoning miss need
opposite fixes.
## The corpus
`hindsight_system_evals/corpus.py` generates facts and their gold labels
together, so a label cannot drift from the text it points at. Two properties it
enforces, both learned by getting them wrong:
- **Every subject is internally consistent.** An earlier version had one release
"deployed to production" on three different dates. That is not a hard question,
it is a contradiction, and a model superseding it was following its rules.
- **Waves never split a subject.** All facts about one release travel together.
Split across waves, the later batch reads as a correction of the earlier one.
`_assert_subjects_are_unique` fails the build if two rows in a cluster ever claim
to describe the same thing again.
## Where it runs
**Not on PRs.** It needs provider secrets, and a single red run is as likely to
be sampling noise as a regression. It runs in the `system-evals` job of
`.github/workflows/perf-test.yml` — the daily schedule, alongside LoComo and
obs-dedup — and publishes to the
[continuous performance monitor](https://vectorize-io.github.io/hindsight-continuous-performance-monitor/system-evals.html)
as a quality metric tracked over time: correct rate and trap count, overall and
per suite (`by_kind`). Traps must stay at 0.
A failing run is still published: for a quality metric the red run is the data
point. `scripts/benchmarks/publish-system-evals-results.sh` does the push.
## Two modes
```bash
# minimum acceptance: the cases that have actually regressed
uv run pytest evals
# full: every category — supersession, entity confusion, scoped truth,
# dense absence, numeric precision — plus the page-collapse check
uv run pytest evals --full
# either, writing the JSON the dashboard publishes
uv run pytest evals --full --output system-evals-results.json
```
The workflow runs `full` by default; `system_evals_mode: minimum` on a manual
dispatch runs the small set.
## Why seeding costs no model calls
Measured on the first blackbox run, per page: 789s of LLM time on fact
extraction and 199s on consolidation, against 21s for the reflect and delta
calls actually under test. So each bank is configured — through the public config
endpoint, still blackbox — with `retain_extraction_mode=chunks` (store each item
as written) and consolidation/observations off, and each page refresh is
triggered explicitly. `chunks` also removes a confound: extraction may paraphrase
a fact, while the gold labels point at the exact authored text.
## Running
```bash
# the model under test (api key, or vertexai with a service account)
export HINDSIGHT_EVAL_LLM_PROVIDER=gemini
export HINDSIGHT_EVAL_LLM_MODEL=gemini-3.7-flash
export HINDSIGHT_EVAL_LLM_API_KEY=...
# The judge is configured separately ON PURPOSE — a model grading its own output
# agrees with itself. The suite warns when these resolve to the same model.
export HINDSIGHT_EVAL_JUDGE_MODEL=gemini-2.5-flash
export HINDSIGHT_EVAL_JUDGE_API_KEY=... # or HINDSIGHT_EVAL_JUDGE_PROVIDER=vertexai
cd hindsight-system-evals && uv run pytest evals
```
VertexAI works for both: set `HINDSIGHT_EVAL_LLM_PROVIDER=vertexai` and
`HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY` / `_PROJECT_ID` — the judge
falls back to the same service account when no judge key is set. That is how the
perf workflow runs it. `HINDSIGHT_EVAL_REFLECT_BUDGET` (default `low`) sets the
reflect budget for `test_02`.
If your shell exports `PYTEST_ADDOPTS` with `-n` (the repo `.env` does), unset
it: xdist is not installed here, and parallel evals against one server would
compete for it anyway.
The server runs on its own pg0 instance (`hindsight-system-evals`), so a run does
not compete for connections with a developer's server or with the system tests.
## Debugging a failure
Each assertion prints the bank id, the page id (or the reflect queries and
whether the gold evidence arrived), and what the page or answer actually reads.
The bank is left in place, so it can be opened in the control plane.
For a knowledge page that goes wrong, two tools, both through the public API:
```bash
# 1. Where does it go wrong? Builds wave 1, then dry-runs the second refresh N
# times: raw synthesis vs the document after the delta operations. --dump
# writes each traced prompt verbatim.
uv run python -m hindsight_system_evals.debug.diagnose_page \
--question hq-count-billing --repeats 3 --dump captures/
# 2. Is it the prompt? Replays one captured delta-ops request N times per
# system-prompt variant and scores the resulting document; --interrogate
# asks the model which lines misled it.
uv run python -m hindsight_system_evals.debug.replay_delta_ops \
--input captures/hq-count-billing-mental_model_delta_ops-<n>.json \
--must-keep "Denning|Barrow|Fairbank" --variants variants.json
```
A variant that wins the replay is a lead; it still has to pass the eval.
hindsight-system-evals/evals/__init__.py
View File
hindsight-system-evals/evals/conftest.py
+130
View File
@@ -0,0 +1,130 @@
"""Fixtures for the eval suite.
Same shape as the system tests: one server for the session, a client, and a
settle helper. The differences are the ones a real model forces —
credentials are required, and the judge is checked against the server's model so
a run cannot quietly grade itself.
"""
from __future__ import annotations
import uuid
import warnings
from collections.abc import AsyncIterator, Iterator
from pathlib import Path
import pytest
from hindsight_client import Hindsight
from hindsight_system_evals import (
judge_model,
provider_environment,
start_eval_server,
wait_until_settled,
)
from hindsight_system_evals.pages import SettleFn
from hindsight_system_evals.report import RECORDED, ModelConfig, ModelRef, RunReport, summarise
BANK_PREFIX = "syseval-"
@pytest.fixture(scope="session")
def eval_server(tmp_path_factory: pytest.TempPathFactory) -> Iterator[object]:
log_path: Path = tmp_path_factory.mktemp("hindsight-eval-server") / "server.log"
server = start_eval_server(log_path=log_path)
yield server
server.stop()
@pytest.fixture(scope="session", autouse=True)
def _judge_is_independent() -> None:
"""Warn when the judge and the model under test are the same.
Not an error, because a single-provider setup is a legitimate way to run
locally — but a model grading its own output agrees with itself, and a run
that does so silently is worth less than it appears.
"""
server_model = provider_environment()["HINDSIGHT_API_LLM_MODEL"]
if judge_model() == server_model:
warnings.warn(
f"The judge and the model under test are both {server_model!r}. "
"Set HINDSIGHT_EVAL_JUDGE_MODEL to something else — self-grading inflates the score.",
stacklevel=1,
)
@pytest.fixture
async def client(eval_server) -> AsyncIterator[Hindsight]:
client = Hindsight(base_url=eval_server.url)
yield client
await client.aclose()
@pytest.fixture
def bank_id() -> str:
# One bank per eval. Sharing would let every page's refresh reflect over
# every other question's corpus, which is not the scenario and makes a
# failure impossible to attribute.
return f"{BANK_PREFIX}{uuid.uuid4().hex[:10]}"
@pytest.fixture
def settled(client: Hindsight) -> SettleFn:
async def _settle(bank: str) -> None:
await wait_until_settled(client, bank)
return _settle
def pytest_configure(config: pytest.Config) -> None:
config.addinivalue_line("markers", "full: the complete set; deselected in minimum-acceptance runs")
def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
"""Minimum acceptance is the default; ``--full`` opts into everything.
CI runs the minimum set, so the gate stays inside a sensible wall-clock
budget and its threshold can be set against measured variance. The full set
is for humans changing a prompt, where breadth matters more than duration.
"""
if config.getoption("--full"):
return
skip_full = pytest.mark.skip(reason="full-only; pass --full to include")
for item in items:
if "full" in item.keywords:
item.add_marker(skip_full)
def pytest_addoption(parser: pytest.Parser) -> None:
parser.addoption("--full", action="store_true", default=False, help="run the complete eval set")
parser.addoption(
"--output",
default=None,
help="write every page outcome as JSON here — what the perf dashboard publishes",
)
def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None:
output = session.config.getoption("--output")
if not output:
return
import datetime
overall = summarise(RECORDED)
env = provider_environment()
report = RunReport(
timestamp=datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds"),
suite="system-evals",
mode="full" if session.config.getoption("--full") else "minimum-acceptance",
llm_config=ModelConfig(
hindsight=ModelRef(provider=env["HINDSIGHT_API_LLM_PROVIDER"], model=env["HINDSIGHT_API_LLM_MODEL"]),
judge=ModelRef(model=judge_model()),
),
total=overall.total,
correct=overall.correct,
correct_rate=(overall.correct / overall.total) if overall.total else None,
trap_count=overall.trap_count,
by_kind={kind: summarise([r for r in RECORDED if r.kind == kind]) for kind in sorted({r.kind for r in RECORDED})},
items=RECORDED,
)
Path(output).write_text(report.to_json(), encoding="utf-8")
hindsight-system-evals/evals/test_01_knowledge_page_convergence.py
+141
View File
@@ -0,0 +1,141 @@
"""A knowledge page must converge on the right answer as data arrives.
This is the eval that motivated the package. A page is created with a source
query and then accumulates: each ingest triggers a delta refresh that edits what
is already stored. Two failures found that way, both of which a one-shot reflect
eval scores as passes because reflect sees the whole bank at once:
* a page that said "Release 0.9.3 was deployed to production on 18 March 2026"
came back one wave later saying "No release was deployed to production on 18
March 2026" — the fact was in the earlier wave and outside the delta window;
* a page counting 3 customers, then handed 4 more, reported 4 and dropped the
first three.
Both traced to the same cause: the delta step treated the reflect synthesis as
authoritative, when that synthesis is written from the new batch alone. Its
totals count only the batch and its absences describe only the batch.
Two assertions per case, and the second matters more. ``correct`` can fail
because an answer is incomplete. ``hit_trap`` means the page asserts the specific
wrong thing — a stored falsehood every later reflect will read back as true.
"""
from __future__ import annotations
import logging
import pytest
from hindsight_client import Hindsight
from hindsight_system_evals import build_page, evaluate, questions, split_into_waves
from hindsight_system_evals.pages import PageOutcome, SettleFn, facts
from hindsight_system_evals.report import RECORDED, EvalRecord
log = logging.getLogger(__name__)
_QUESTIONS = {question.id: question for question in questions()}
#: The minimum-acceptance set: one question per failure mode that has actually
#: regressed, kept small so CI stays inside a sane budget. Everything else is
#: marked ``full``. These two are here because they are the two that broke.
MINIMUM_ACCEPTANCE = ("hq-release-prod", "hq-count-billing")
async def _run(
client: Hindsight, bank_id: str, settled: SettleFn, question_id: str, *, record: bool = True
) -> PageOutcome:
question = _QUESTIONS[question_id]
outcome = await build_page(client, bank_id, question, split_into_waves(facts()), settled)
assert outcome.final_content.strip(), (
f"{question_id}: the page is empty after {len(outcome.waves)} waves — sizes {[w.chars for w in outcome.waves]}"
)
context = f"This is a knowledge page built to answer: {question.question}"
verdict = await evaluate(outcome.final_content, question.answer_criteria, context=context)
outcome.correct, outcome.correct_reason = verdict.meets_criteria, verdict.reasoning
if question.must_not_claim:
trap = await evaluate(
outcome.final_content,
f"The page asserts, as currently true, that {question.must_not_claim}.",
context=context,
)
outcome.hit_trap, outcome.trap_reason = trap.meets_criteria, trap.reasoning
# A check that re-runs a question for a different property (the collapse
# check below) must not count as a second graded page: it would inflate the
# total and weight that one question twice in the published correct rate.
if record:
RECORDED.append(
EvalRecord(
kind="knowledge_page",
question_id=question_id,
category=question.category,
correct=outcome.correct,
hit_trap=outcome.hit_trap,
sizes=[w.chars for w in outcome.waves],
bank_id=outcome.bank_id,
page_id=outcome.page_id,
reason=outcome.trap_reason if outcome.hit_trap else outcome.correct_reason,
)
)
log.info(
"%s: correct=%s trap=%s sizes=%s",
question_id,
outcome.correct,
outcome.hit_trap,
[w.chars for w in outcome.waves],
)
return outcome
def _assert_sound(outcome: PageOutcome) -> None:
# The trap first: a page that asserts the wrong thing is worse than one that
# is merely incomplete, and reporting them in that order makes a failure
# readable without opening the log.
assert not outcome.hit_trap, (
f"{outcome.question_id}: the page asserts the wrong answer — {outcome.trap_reason}\n"
f"bank {outcome.bank_id}, page {outcome.page_id}\n"
f"page reads: {outcome.final_content[:400]}"
)
assert outcome.correct, (
f"{outcome.question_id}: {outcome.correct_reason}\n"
f"bank {outcome.bank_id}, page {outcome.page_id}\n"
f"page sizes by wave: {[w.chars for w in outcome.waves]}\n"
f"page reads: {outcome.final_content[:400]}"
)
@pytest.mark.parametrize("question_id", MINIMUM_ACCEPTANCE)
async def test_page_converges_minimum_acceptance(
client: Hindsight, bank_id: str, settled: SettleFn, question_id: str
) -> None:
"""The two cases that have actually regressed. This is the CI gate."""
outcome = await _run(client, bank_id, settled, question_id)
_assert_sound(outcome)
@pytest.mark.full
@pytest.mark.parametrize("question_id", [q for q in _QUESTIONS if q not in MINIMUM_ACCEPTANCE])
async def test_page_converges_full(client: Hindsight, bank_id: str, settled: SettleFn, question_id: str) -> None:
"""The rest of the categories: supersession, entity confusion, scoped truth,
dense absence, numeric precision. Run with ``--full``."""
outcome = await _run(client, bank_id, settled, question_id)
_assert_sound(outcome)
@pytest.mark.full
async def test_a_page_never_silently_collapses(client: Hindsight, bank_id: str, settled: SettleFn) -> None:
"""A wave that shrinks a page to almost nothing is a destructive edit.
Separate from correctness because a page can shrink legitimately — a
superseded claim should go — but a collapse to a fraction of its former size
is the signature of a replace that should have been a merge, and a final-text
score cannot see it.
"""
outcome = await _run(client, bank_id, settled, "hq-count-billing", record=False)
sizes = [w.chars for w in outcome.waves]
assert sizes[-1] >= sizes[0] * 0.5, (
f"the page collapsed across waves ({sizes}) — bank {outcome.bank_id}\npage reads: {outcome.final_content[:400]}"
)
hindsight-system-evals/evals/test_02_reflect_answers.py
+87
View File
@@ -0,0 +1,87 @@
"""Reflect must answer the corpus questions — and never assert the bait.
The whole corpus is in the bank at once, so this is the easy case for retrieval
and the hard one for grounding: every category's near-misses are present
together. The incident behind the minimum-acceptance question: asked for the
2024 engineering headcount in a bank that only covers 2025-2026, reflect
extrapolated backwards from the later growth and reported a specific number as
"reliably deduced". Nothing in the bank said it.
Same two assertions as the knowledge-page suite, in the same order: the trap
(the answer asserts the specific wrong thing) before correctness (the answer is
incomplete). A failure also says whether the gold evidence reached the model,
which decides where the fix goes.
"""
from __future__ import annotations
import logging
import pytest
from hindsight_client import Hindsight
from hindsight_system_evals import AnswerOutcome, ask, evaluate, facts, questions, seed_bank
from hindsight_system_evals.pages import SettleFn
from hindsight_system_evals.report import RECORDED, EvalRecord
log = logging.getLogger(__name__)
_QUESTIONS = {question.id: question for question in questions()}
#: The one that has actually regressed. Everything else runs with ``--full``.
MINIMUM_ACCEPTANCE = ("hq-absent-2024",)
async def _run(client: Hindsight, bank_id: str, settled: SettleFn, question_id: str) -> AnswerOutcome:
question = _QUESTIONS[question_id]
corpus = facts()
await seed_bank(client, bank_id, corpus, settled)
outcome = await ask(client, bank_id, question, corpus)
assert outcome.answer, f"{question_id}: reflect returned no text — bank {bank_id}"
context = f"Question asked: {question.question}"
verdict = await evaluate(outcome.answer, question.answer_criteria, context=context)
outcome.correct, outcome.correct_reason = verdict.meets_criteria, verdict.reasoning
if question.must_not_claim:
trap = await evaluate(
outcome.answer, f"The answer asserts, as currently true, that {question.must_not_claim}.", context=context
)
outcome.hit_trap, outcome.trap_reason = trap.meets_criteria, trap.reasoning
RECORDED.append(
EvalRecord(
kind="reflect",
question_id=question_id,
category=question.category,
correct=outcome.correct,
hit_trap=outcome.hit_trap,
bank_id=bank_id,
reason=outcome.trap_reason if outcome.hit_trap else outcome.correct_reason,
)
)
log.info("%s: correct=%s trap=%s blame=%s", question_id, outcome.correct, outcome.hit_trap, outcome.blame)
return outcome
def _assert_sound(outcome: AnswerOutcome) -> None:
evidence = f"blame: {outcome.blame}\nqueries: {outcome.queries}\nbank {outcome.bank_id}"
assert not outcome.hit_trap, (
f"{outcome.question_id}: the answer asserts the wrong thing — {outcome.trap_reason}\n"
f"{evidence}\nanswer: {outcome.answer[:400]}"
)
assert outcome.correct, (
f"{outcome.question_id}: {outcome.correct_reason}\n{evidence}\nanswer: {outcome.answer[:400]}"
)
@pytest.mark.parametrize("question_id", MINIMUM_ACCEPTANCE)
async def test_reflect_answers_minimum_acceptance(
client: Hindsight, bank_id: str, settled: SettleFn, question_id: str
) -> None:
_assert_sound(await _run(client, bank_id, settled, question_id))
@pytest.mark.full
@pytest.mark.parametrize("question_id", [q for q in _QUESTIONS if q not in MINIMUM_ACCEPTANCE])
async def test_reflect_answers_full(client: Hindsight, bank_id: str, settled: SettleFn, question_id: str) -> None:
_assert_sound(await _run(client, bank_id, settled, question_id))
hindsight-system-evals/hindsight_system_evals/__init__.py
+34
View File
@@ -0,0 +1,34 @@
"""Blackbox quality evals: a real server, a real model, an independent judge.
The sibling of ``hindsight-system-tests``, and deliberately a separate package.
Those tests stub every LLM call so they are deterministic and need no secrets —
which is exactly what an eval cannot do, because a stubbed model means scoring
the stub. So these need provider credentials, cannot run on fork PRs, and report
RATES rather than equalities.
Read ``README.md`` for the two modes and what each is for.
"""
from hindsight_system_evals.answers import AnswerOutcome, ask, seed_bank
from hindsight_system_evals.judge import Verdict, evaluate, judge_model
from hindsight_system_evals.pages import PageOutcome, build_page, facts, questions, split_into_waves
from hindsight_system_evals.server import EvalServer, provider_environment, start_eval_server
from hindsight_system_evals.waiting import wait_until_settled
__all__ = [
"AnswerOutcome",
"EvalServer",
"PageOutcome",
"Verdict",
"ask",
"build_page",
"evaluate",
"facts",
"judge_model",
"provider_environment",
"questions",
"seed_bank",
"split_into_waves",
"start_eval_server",
"wait_until_settled",
]
hindsight-system-evals/hindsight_system_evals/answers.py
+84
View File
@@ -0,0 +1,84 @@
"""Ask reflect a corpus question over a fully seeded bank, and keep what it did.
The knowledge-page suite measures what a page converges to across waves. This
one measures the one-shot answer: the whole corpus is in the bank, reflect gets
the question, and the judge reads what it wrote.
Retrieval is a necessary condition, not a sufficient one — reflect can retrieve
every gold fact and still answer from the wrong one — so the ANSWER is what gets
graded. The tool trace is kept only to say which half failed, because the two
have opposite fixes: evidence that never arrived is a query or ranking problem,
evidence that arrived and was misused is a prompt or model problem.
"""
from __future__ import annotations
import json
import os
from dataclasses import dataclass, field
from hindsight_client import Hindsight
from hindsight_system_evals.corpus import HardFact, HardQuestion
from hindsight_system_evals.pages import SettleFn, prepare_bank
#: ``low`` is what most callers send, and the budget the forced prelude and its
#: short-circuit are tuned for. Override to measure another one.
REFLECT_BUDGET = os.getenv("HINDSIGHT_EVAL_REFLECT_BUDGET", "low")
@dataclass
class AnswerOutcome:
"""One reflect answer, graded, with enough of its trace to attribute a failure."""
question_id: str
category: str
bank_id: str
answer: str = ""
queries: list[str] = field(default_factory=list)
gold_retrieved: int = 0
gold_total: int = 0
correct: bool = False
correct_reason: str = ""
hit_trap: bool = False
trap_reason: str = ""
@property
def blame(self) -> str:
if self.correct and not self.hit_trap:
return "ok"
if self.gold_total and self.gold_retrieved < self.gold_total:
return f"retrieval ({self.gold_retrieved}/{self.gold_total} gold facts reached the model)"
return "reasoning (every gold fact reached the model)"
async def seed_bank(client: Hindsight, bank_id: str, facts: list[HardFact], settle: SettleFn) -> None:
"""Store the whole corpus as written, with no model calls — see ``prepare_bank``."""
await prepare_bank(client, bank_id)
await client.aretain_batch(bank_id=bank_id, items=[{"content": fact.text} for fact in facts])
await settle(bank_id)
async def ask(client: Hindsight, bank_id: str, question: HardQuestion, facts: list[HardFact]) -> AnswerOutcome:
"""Reflect once and record which gold facts its tools actually returned."""
outcome = AnswerOutcome(question_id=question.id, category=question.category, bank_id=bank_id)
response = await client.areflect(
bank_id=bank_id, query=question.question, budget=REFLECT_BUDGET, include_tool_calls=True
)
outcome.answer = (response.text or "").strip()
# Matched by text, not id: ``chunks`` retain stores each fact verbatim, and
# the server's ids are its own. Walks the tool OUTPUTS rather than
# ``based_on``, which is what the model declared it used — downstream of the
# very thing being diagnosed.
gold_ids = set(question.gold)
gold_texts = {fact.text for fact in facts if fact.id in gold_ids}
outcome.gold_total = len(gold_texts)
seen: set[str] = set()
for call in (response.trace.tool_calls if response.trace else None) or []:
if query := (call.input or {}).get("query"):
outcome.queries.append(f"{call.tool}({query})")
payload = json.dumps(call.output or {}, ensure_ascii=False, default=str)
seen.update(text for text in gold_texts if text in payload)
outcome.gold_retrieved = len(seen)
return outcome
hindsight-system-evals/hindsight_system_evals/corpus.py
+491
View File
@@ -0,0 +1,491 @@
"""A corpus built to be hard, where a big one would merely be slow.
An earlier version padded the bank to ~1000 rows with filler from unrelated
domains (botany, cycling, carpentry). Separating that from the gold is something
bag-of-words would manage: contamination was zero and the scores were identical
to a 28-row bank. Row count is not difficulty, so the padding is gone.
What makes retrieval hard is **near-miss density inside the question's own
topic**. So every fact here belongs to a cluster that shares the gold's
vocabulary, and the near-misses differ from the answer by exactly one attribute —
a version, an environment, a date, a number, a polarity, an entity. Nothing can
be separated by topic; it has to be separated by the attribute the question asks
about.
And difficulty is not only retrieval. Several clusters are built so that
retrieval returns EVERY candidate and still tells you nothing — the answer is
only right if reflect reasons over what it retrieved:
* ``supersession`` — a value changes three times; every version is retrievable
and equally on-topic. Only the latest is correct.
* ``counting`` — the answer is how many facts match, so missing one is wrong
while recall@k still looks fine.
* ``entity_confusion`` — two services one character apart, with parallel facts.
* ``scoped_truth`` — true in one region, false in another; an unqualified answer
is wrong.
* ``absence_dense`` — the topic is densely present, the specific fact is not.
Saying "no information" is the correct answer, surrounded by plausible bait.
Questions and their gold ids are produced by the SAME generator that writes the
facts, so a label can never drift from the text it points at — the failure mode
that makes hand-maintained corpora quietly wrong.
"""
from __future__ import annotations
from dataclasses import dataclass, field
@dataclass
class HardFact:
id: str
text: str
cluster: str
#: What this fact is ABOUT. Facts sharing a subject must be ingested in the
#: same wave: split across waves, the later batch reads as a correction of the
#: earlier one and gets superseded -- which is how "Release 0.9.3 reached
#: production" was legitimately overwritten by a later batch that only
#: mentioned staging and canary.
subject: str = ""
@dataclass
class HardQuestion:
id: str
category: str
question: str
ideal_query: str
gold: list[str] = field(default_factory=list)
answer_criteria: str = ""
must_not_claim: str = ""
short_circuit: bool = False
@dataclass
class HardCorpus:
"""Facts and the questions labelled against them — one cluster, or all of them."""
facts: list[HardFact]
questions: list[HardQuestion]
_MONTHS = [
"January",
"February",
"March",
"April",
"May",
"June",
"July",
"August",
"September",
"October",
"November",
"December",
]
def _releases() -> HardCorpus:
"""Ten releases, each with ONE coherent lifecycle. Near-misses are other releases.
The first version generated the same release "deployed production" on three
different dates and "rolled back from production" on three others. That is not
a hard question, it is a contradiction: read as a narrative about one release
it says the release reached production repeatedly. Worse, when the corpus was
split into ingest waves, a later batch of staging/canary facts about that same
release read exactly like a correction of the earlier production claim -- and
the model superseded it, correctly applying the documented temporal rule. I
twice reported that as a delta bug. It was this.
So each (release, environment) pair occurs at most ONCE and every release runs
staging -> canary -> production in order. Discrimination comes from ten
similar releases, never from one release contradicting itself. The gold is the
only production deploy on 18 March 2026; 0.9.6's canary promotion on that same
date is the near-miss that makes the environment the thing you must get right.
"""
facts: list[HardFact] = []
gold_id = "rel-prod-3"
# (staging, canary, production) for each release. Production dates are unique,
# so "which release reached production on D" has exactly one answer.
schedule = [
("2026-02-03", "2026-02-10", "2026-02-17"),
("2026-02-06", "2026-02-13", "2026-02-24"),
("2026-03-02", "2026-03-09", "2026-03-16"),
("2026-03-04", "2026-03-11", "2026-03-18"), # 0.9.3 -> the gold
("2026-03-06", "2026-03-13", "2026-03-20"),
("2026-04-01", "2026-04-08", "2026-04-15"),
("2026-03-09", "2026-03-18", "2026-04-22"), # 0.9.6 canary ON the gold date
("2026-04-06", "2026-04-13", "2026-04-29"),
("2026-05-04", "2026-05-11", "2026-05-18"),
("2026-05-06", "2026-05-13", "2026-05-26"),
]
for minor, (staging, canary, production) in enumerate(schedule):
version = f"0.9.{minor}"
for suffix, text in (
("stg", f"Release {version} was deployed to staging on {staging}."),
("can", f"Release {version} was promoted to canary on {canary}."),
("prod", f"Release {version} was deployed to production on {production}."),
):
facts.append(HardFact(f"rel-{suffix}-{minor}", text, "releases", subject=version))
q = HardQuestion(
id="hq-release-prod",
category="precision_discrimination",
question="Which release was deployed to production on 18 March 2026?",
ideal_query="release deployed production 2026-03-18",
gold=[gold_id],
answer_criteria="Identifies release 0.9.3 as the one deployed to production on 2026-03-18.",
must_not_claim="a release other than 0.9.3 was deployed to production on 18 March 2026",
)
return HardCorpus(facts=facts, questions=[q])
def _supersession() -> HardCorpus:
"""The value changes four times. Every version is on-topic and retrievable.
Retrieval cannot be wrong here and cannot be right either: it returns the
whole history. Only ordering the history correctly produces the right answer,
which is precisely what a retrieval metric cannot see.
"""
owners = ["the platform team", "the retrieval team", "the search guild", "the core infra team"]
# Every handover must be in the PAST. The first version of this walked to
# October 2026 — the future — and reflect correctly reported the last owner as
# "scheduled", which the criteria then marked wrong. The corpus was the bug,
# not the answer, so the chain is pinned to dates that have already happened.
handovers = [("March", 2023), ("September", 2023), ("April", 2024), ("November", 2024)]
facts = [
HardFact(
f"own-{i + 1:03d}",
f"Ownership of the ranking service passed to {owner} in {month} {year}.",
"supersession",
)
for i, (owner, (month, year)) in enumerate(zip(owners, handovers))
]
# Filler that shares the vocabulary without answering: other systems moving
# between the same teams, so "ownership + team" retrieves plenty of wrong rows.
other = ["the ingestion pipeline", "the billing gateway", "the audit log", "the webhook dispatcher"]
for i, system in enumerate(other):
for j, owner in enumerate(owners):
facts.append(
HardFact(
f"own-x{i}{j}",
f"Ownership of {system} passed to {owner} in {_MONTHS[(i + j) * 2 % 12]} 202{3 + (j % 2)}.",
"supersession",
)
)
q = HardQuestion(
id="hq-owner-latest",
category="supersession",
question="Which team owns the ranking service now?",
ideal_query="ranking service ownership current team",
gold=[f.id for f in facts if f.cluster == "supersession" and f.id.startswith("own-0")],
answer_criteria=(
"States that the core infra team currently owns the ranking service. Mentioning the earlier "
"owners as history is fine; naming any of them as the current owner is not."
),
must_not_claim="the platform team, the retrieval team or the search guild currently owns the ranking service",
)
return HardCorpus(facts=facts, questions=[q])
def _counting() -> HardCorpus:
"""The answer is a count, so retrieving most of the evidence is still wrong.
Seven customers reported billing errors in Q2; eleven more reported other
things, or billing errors outside Q2. recall@k can look healthy while the
number in the answer is wrong.
"""
facts: list[HardFact] = []
gold: list[str] = []
billing_q2 = ["Aldridge", "Barrow", "Calloway", "Denning", "Ellery", "Fairbank", "Gadsden"]
for i, name in enumerate(billing_q2):
fid = f"cnt-b{i:02d}"
gold.append(fid)
facts.append(HardFact(fid, f"{name} Ltd reported a billing error in {_MONTHS[3 + i % 3]} 2026.", "counting"))
# Same shape, wrong quarter or wrong issue — retrieves just as well.
for i, name in enumerate(
[
"Harkness",
"Illingworth",
"Jarrow",
"Kelsey",
"Lomax",
"Mowbray",
"Ashcombe",
"Brightwell",
"Corfield",
"Dunmore",
"Everleigh",
"Fenwick",
"Garsdale",
"Halstead",
"Inglewood",
"Jessop",
"Kirkby",
"Langdon",
]
):
facts.append(
HardFact(f"cnt-o{i:02d}", f"{name} Ltd reported a billing error in {_MONTHS[9 + i % 3]} 2026.", "counting")
)
for i, name in enumerate(
[
"Norbury",
"Oakley",
"Pemberton",
"Quill",
"Ravensworth",
"Saltmarsh",
"Thirlwall",
"Underhill",
"Vance",
"Wetherby",
"Yarborough",
"Zouche",
"Alverton",
"Brackley",
"Cranmore",
]
):
facts.append(
HardFact(
f"cnt-n{i:02d}", f"{name} Ltd reported a latency problem in {_MONTHS[3 + i % 3]} 2026.", "counting"
)
)
q = HardQuestion(
id="hq-count-billing",
category="counting",
question="How many customers reported a billing error in the second quarter of 2026?",
ideal_query="customers billing error April May June 2026 count",
gold=gold,
answer_criteria="States that seven (7) customers reported a billing error in Q2 2026.",
must_not_claim="a number of Q2 billing-error reports other than seven",
)
return HardCorpus(facts=facts, questions=[q])
def _entity_confusion() -> HardCorpus:
"""Two services one character apart, with parallel facts about each."""
facts: list[HardFact] = []
gold: list[str] = []
for i, (svc, port, owner) in enumerate(
[
("payments-api", 8443, "the billing team"),
("payment-api", 8444, "the legacy platform team"),
("payments-apy", 8445, "the migration squad"),
]
):
for j, (attr, val) in enumerate(
[
("listens on port", port),
("is maintained by", owner),
("was last audited in", "2026"),
("exposes a health endpoint on port", port + 1000),
("was migrated off the legacy gateway in", "2025"),
("has a documented SLO of", "99.9%"),
]
):
fid = f"ent-{i}{j}"
facts.append(HardFact(fid, f"The {svc} service {attr} {val}.", "entity_confusion"))
if svc == "payments-api" and attr == "listens on port":
gold.append(fid)
q = HardQuestion(
id="hq-entity-port",
category="entity_confusion",
question="Which port does the payments-api service listen on?",
ideal_query="payments-api service listening port",
gold=gold,
answer_criteria="States that payments-api listens on port 8443.",
must_not_claim="payments-api listens on port 8444",
)
return HardCorpus(facts=facts, questions=[q])
def _scoped_truth() -> HardCorpus:
"""True in one region, false in another. An unqualified answer is wrong."""
facts = [
HardFact("scp-001", "Two-factor authentication is mandatory for all EU accounts.", "scoped_truth"),
HardFact("scp-002", "Two-factor authentication remains optional for US accounts.", "scoped_truth"),
HardFact("scp-003", "Two-factor authentication is mandatory for UK accounts from 2026.", "scoped_truth"),
HardFact("scp-004", "Single sign-on is mandatory for all enterprise accounts.", "scoped_truth"),
HardFact("scp-005", "Password rotation is optional for every region.", "scoped_truth"),
]
q = HardQuestion(
id="hq-scoped-2fa",
category="scoped_truth",
question="Is two-factor authentication mandatory for our accounts?",
ideal_query="two-factor authentication mandatory optional region accounts",
gold=["scp-001", "scp-002", "scp-003"],
answer_criteria=(
"Answers with the regional qualification rather than a flat yes or no: mandatory for EU "
"accounts and optional for US accounts. Any accurate treatment of the UK is acceptable — "
"the memory says UK accounts are mandatory FROM 2026, so stating it either as mandatory or "
"as mandatory starting in 2026 is correct, and so is omitting the UK entirely."
),
must_not_claim="two-factor authentication is mandatory everywhere, with no regional distinction",
)
return HardCorpus(facts=facts, questions=[q])
def _absence_dense() -> HardCorpus:
"""The topic is densely present; the asked-for year is not. Bait for a guess."""
facts = [
HardFact(
f"abs-{i:03d}",
f"Engineering headcount reached {40 + i * 7} at the end of {_MONTHS[i % 12]} 202{5 + i % 2}.",
"absence_dense",
)
for i in range(14)
]
q = HardQuestion(
id="hq-absent-2024",
category="absence_dense",
question="What was the engineering headcount at the end of 2024?",
ideal_query="engineering headcount end of 2024",
gold=[],
answer_criteria=(
"Says the memory holds no headcount figure for 2024, rather than giving or estimating one. "
"Citing 2025/2026 figures as context is fine only if it does not present one as the 2024 number."
),
must_not_claim="a specific engineering headcount figure for 2024",
)
return HardCorpus(facts=facts, questions=[q])
def _numeric_precision() -> HardCorpus:
"""A dozen similar magnitudes in one topic; only one matches the predicate."""
facts: list[HardFact] = []
gold_id = "num-006"
causes = [
"connection pool exhaustion",
"disk exhaustion",
"certificate expiry",
"DNS misconfiguration",
"a memory leak",
"connection pool exhaustion",
"thread starvation",
"a bad migration",
"clock skew",
"an upstream timeout",
"connection pool exhaustion",
"a failed failover",
"index corruption",
"a config rollout",
"disk exhaustion",
"connection pool exhaustion",
"a network partition",
"certificate expiry",
"queue backpressure",
"a bad deploy",
"connection pool exhaustion",
"replica lag",
"a memory leak",
"DNS misconfiguration",
]
# Each outage must own its (month, year). The first version cycled months with
# `i % 12` and years with `i // 12`, which produced a SECOND "April 2026
# outage" carrying different values — so the question had two contradictory
# answers, and reflect reporting a conflict was correct while the corpus was
# wrong. Near-misses must differ in their VALUES, never in what they claim to
# be. April 2026 is reserved for the gold row.
slots = [(m, y) for y in (2024, 2025, 2026) for m in _MONTHS if not (m == "April" and y == 2026)]
if len(causes) > len(slots):
raise RuntimeError("More outages than distinct (month, year) slots — they would collide")
for i, cause in enumerate(causes):
fid = f"num-{i:03d}"
limit = 100 + i * 50
month, year = slots[i]
text = (
f"The {month} {year} outage was caused by {cause} at {limit} connections and lasted {12 + i * 5} minutes."
)
if fid == gold_id:
text = (
"The April 2026 outage was caused by connection pool exhaustion at 200 connections "
"and lasted 47 minutes."
)
facts.append(HardFact(fid, text, "numeric_precision"))
q = HardQuestion(
id="hq-numeric-outage",
category="numeric_precision",
question="What caused the April 2026 outage, at what connection limit, and how long did it last?",
ideal_query="April 2026 outage connection pool exhaustion 200 connections 47 minutes",
gold=[gold_id],
answer_criteria=("States all three: connection pool exhaustion, a limit of 200 connections, and 47 minutes."),
must_not_claim="a connection limit other than 200 or a duration other than 47 minutes for the April 2026 outage",
)
return HardCorpus(facts=facts, questions=[q])
_BUILDERS = (
_releases,
_supersession,
_counting,
_entity_confusion,
_scoped_truth,
_absence_dense,
_numeric_precision,
)
def build() -> HardCorpus:
"""The whole hard corpus: facts and the questions labelled against them."""
facts: list[HardFact] = []
questions: list[HardQuestion] = []
for builder in _BUILDERS:
cluster = builder()
facts.extend(cluster.facts)
questions.extend(cluster.questions)
ids = [f.id for f in facts]
if len(set(ids)) != len(ids):
raise RuntimeError("Duplicate fact id in the hard corpus")
texts = [f.text for f in facts]
if len(set(texts)) != len(texts):
raise RuntimeError("Duplicate fact text in the hard corpus — gold labelling needs texts to be unique")
known = set(ids)
for q in questions:
missing = [g for g in q.gold if g not in known]
if missing:
raise RuntimeError(f"{q.id} references unknown gold ids: {missing}")
_assert_subjects_are_unique(facts)
return HardCorpus(facts=facts, questions=questions)
#: Phrases that name a single real-world thing. Two rows opening with the same
#: one are not near-misses, they are a contradiction: the question then has two
#: incompatible answers and a correct "the data conflicts" reply gets scored
#: wrong. This is exactly how the corpus once grew a second "April 2026 outage"
#: with different values and made reflect look broken. Near-misses must differ in
#: what they ASSERT, never in what they claim to BE.
_SUBJECT_PATTERNS = (
("numeric_precision", r"^The (\w+ \d{4}) outage"),
("releases", r"^Release (\S+) was (deployed to|promoted to|rolled back from) (production|staging|canary)"),
("supersession", r"^Ownership of (the [\w ]+?) passed to ([\w ]+?) in"),
)
def _assert_subjects_are_unique(facts: list[HardFact]) -> None:
"""Fail the build when two rows in a cluster claim to describe the same thing."""
import re
for cluster, pattern in _SUBJECT_PATTERNS:
seen: dict[tuple[str, ...], str] = {}
for fact in facts:
if fact.cluster != cluster:
continue
match = re.match(pattern, fact.text)
if match is None:
continue
key = match.groups()
if key in seen:
raise RuntimeError(
f"{cluster}: {fact.id} and {seen[key]} both describe {key!r}. "
"Two rows describing the same subject contradict rather than compete — "
"vary the values, not the identity."
)
seen[key] = fact.id
hindsight-system-evals/hindsight_system_evals/debug/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Debug tools for a failing eval. Not tests: run them by hand, see the README."""
hindsight-system-evals/hindsight_system_evals/debug/diagnose_page.py
+127
View File
@@ -0,0 +1,127 @@
"""Where does a wrong knowledge page come from: the synthesis, or the edit?
A page that is right after wave 1 and wrong after wave 2 has two candidate
causes, and they need opposite fixes:
* the **reflect synthesis** inside the refresh is already wrong, because the
delta window genuinely does not contain the fact — a scoping problem; or
* the synthesis is fine and the **delta operations** delete or overwrite the
good section anyway — a delta-application problem.
A dry-run refresh separates them: ``candidate_content`` is the synthesis before
any operation, ``preview_content`` the document after. It runs the production
pipeline and persists nothing, so it can be repeated on the same window — which
also says whether a failure is stable or one unlucky sample.
What the model was ASKED comes from the server's LLM request traces, dumped
verbatim with ``--dump`` so ``replay_delta_ops`` can replay them. All of it goes
through the public API, same as the evals.
Run with::
cd hindsight-system-evals
uv run python -m hindsight_system_evals.debug.diagnose_page --question hq-count-billing --dump captures/
"""
from __future__ import annotations
import argparse
import asyncio
import json
import tempfile
import uuid
from pathlib import Path
import httpx
from hindsight_client import Hindsight
from hindsight_client_api.models.llm_request_list_response import LLMRequestListResponse
from hindsight_system_evals.pages import create_page, facts, prepare_bank, questions, read_page, split_into_waves
from hindsight_system_evals.server import start_eval_server
from hindsight_system_evals.waiting import wait_until_settled
def _excerpt(text: str, limit: int = 400) -> str:
flat = " ".join((text or "").split())
return flat[:limit] + ("…" if len(flat) > limit else "")
async def _traces(url: str, bank_id: str) -> LLMRequestListResponse:
# Every call, not a scope filter: the refresh's reflect runs under several
# scopes (tool calls, final synthesis), and the one that misled the model is
# not always the one you would guess.
async with httpx.AsyncClient(base_url=url, timeout=60) as http:
response = await http.get(f"/v1/default/banks/{bank_id}/llm-requests", params={"limit": 500})
response.raise_for_status()
return LLMRequestListResponse.from_dict(response.json())
async def diagnose(url: str, question_id: str, repeats: int, dump: Path | None) -> None:
question = next(q for q in questions() if q.id == question_id)
waves = split_into_waves(facts())
client = Hindsight(base_url=url)
bank_id = f"syseval-diag-{uuid.uuid4().hex[:8]}"
try:
await prepare_bank(client, bank_id)
page = await create_page(client, bank_id, question)
mm_id = page.mental_model_id
await wait_until_settled(client, bank_id)
print(f"{question.id}: {question.question}\nbank {bank_id}, page {page.page_id}\n")
# Wave 1 for real, so the page holds the correct claim going in.
await client.aretain_batch(bank_id=bank_id, items=[{"content": f.text} for f in waves[0]])
await wait_until_settled(client, bank_id)
await client.arefresh_mental_model(bank_id=bank_id, mental_model_id=mm_id)
await wait_until_settled(client, bank_id)
print(f"after wave 1 (persisted): {_excerpt(await read_page(client, bank_id, mm_id))}\n")
# Wave 2 ingested but NOT refreshed: every dry run below previews the same
# second refresh over the same window, which is what makes them comparable.
await client.aretain_batch(bank_id=bank_id, items=[{"content": f.text} for f in waves[1]])
await wait_until_settled(client, bank_id)
for attempt in range(1, repeats + 1):
dry = await client.adry_run_refresh_mental_model(bank_id=bank_id, mental_model_id=mm_id)
print(f"dry run {attempt}: mode={dry.effective_mode} outcome={dry.outcome}")
print(f" window: {dry.window}")
print(f" candidate (raw synthesis): {_excerpt(dry.candidate_content, 300)}")
if dry.delta_operations:
for op in dry.delta_operations.applied or []:
print(f" applied {op.get('op', '?')}: {_excerpt(json.dumps(op, ensure_ascii=False), 160)}")
for op in dry.delta_operations.skipped or []:
print(f" skipped: {_excerpt(json.dumps(op, ensure_ascii=False), 160)}")
print(f" preview (after ops): {_excerpt(dry.preview_content, 300)}")
print(f" expected: {_excerpt(question.answer_criteria, 200)}\n")
listing = await _traces(url, bank_id)
print(f"{listing.total} traced LLM call(s): {sorted({entry.scope or '?' for entry in listing.items})}")
if dump is not None:
dump.mkdir(parents=True, exist_ok=True)
for index, entry in enumerate(listing.items):
target = dump / f"{question.id}-{entry.scope or 'unscoped'}-{index}.json"
target.write_text(json.dumps(entry.input, indent=2, ensure_ascii=False), encoding="utf-8")
print(f" wrote {target}")
finally:
await client.aclose()
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--question", default="hq-release-prod", help="corpus question id")
parser.add_argument("--repeats", type=int, default=3, help="dry runs of the second refresh")
parser.add_argument("--dump", type=Path, help="write each traced prompt here, for replay_delta_ops")
parser.add_argument("--url", help="an already-running server; by default one is started like the evals do")
args = parser.parse_args()
if args.url:
asyncio.run(diagnose(args.url, args.question, args.repeats, args.dump))
return
server = start_eval_server(log_path=Path(tempfile.mkdtemp(prefix="syseval-diag-")) / "server.log")
try:
print(f"server {server.url} (log {server.log_path}) — left running banks can be opened in the control plane\n")
asyncio.run(diagnose(server.url, args.question, args.repeats, args.dump))
finally:
server.stop()
if __name__ == "__main__":
main()
hindsight-system-evals/hindsight_system_evals/debug/replay_delta_ops.py
+193
View File
@@ -0,0 +1,193 @@
"""Replay one captured delta-ops request, try prompt variants on it, ask the model why.
An end-to-end page eval takes minutes, far too slow to iterate on a prompt. This
takes ONE captured ``mental_model_delta_ops`` request (``diagnose_page --dump``)
and replays it N times per system-prompt variant, so a change is judged in
seconds.
Scoring is mechanical, not judged: apply the emitted operations to the captured
CURRENT DOCUMENT and check the claim under test is still in the result. It scores
the resulting DOCUMENT, never each operation — an op that rewrites "a total of 3"
to "a total of 7" need not repeat the customer names, and scoring ops one by one
failed exactly that correct merge.
Every variant runs N times because these calls are not reproducible even at
temperature 0: the same request produced a destructive edit in production and a
correct one on replay. A variant's score is a rate; one clean run proves nothing.
``--interrogate`` then continues the SAME chat, telling the model what the right
result was and asking which lines of the prompt led it astray. That answer is a
lead about the input, not a result — any fix still has to win the replay and then
the eval.
Run with::
cd hindsight-system-evals
uv run python -m hindsight_system_evals.debug.replay_delta_ops \\
--input captures/hq-count-billing-mental_model_delta_ops-<n>.json \\
--must-keep "Denning|Barrow|Fairbank" --repeats 5 --variants variants.json
"""
from __future__ import annotations
import argparse
import asyncio
import json
import os
from dataclasses import dataclass, field
from pathlib import Path
from google import genai
from google.genai import types
from pydantic import BaseModel
class Message(BaseModel):
role: str
content: str
@dataclass
class CapturedRequest:
system: str
user: str
@property
def document(self) -> str:
"""The CURRENT DOCUMENT section, so an untouched claim counts as surviving."""
start = self.user.find("CURRENT DOCUMENT")
end = self.user.find("## NEW INFORMATION")
return self.user[start:end] if start != -1 and end != -1 else ""
@dataclass
class VariantScore:
kept: int
failures: list[str] = field(default_factory=list)
def load(path: Path) -> CapturedRequest:
raw = json.loads(path.read_text(encoding="utf-8"))
# A trace row stores the messages list; accept it bare or wrapped.
messages = [Message.model_validate(m) for m in (raw["messages"] if isinstance(raw, dict) else raw)]
return CapturedRequest(
system=next(m.content for m in messages if m.role == "system"),
user=next(m.content for m in messages if m.role == "user"),
)
def survives(reply: str, must_keep: list[str], document: str) -> bool:
"""Whether every must-keep token is in the document the operations produce."""
try:
ops = json.loads(reply[reply.index("{") : reply.rindex("}") + 1]).get("operations", [])
except ValueError:
return False
# Destructive ops drop what they target and we cannot resolve block ids from
# the prompt text alone, so any of them discards the captured document; the
# additive ops then contribute their own text. Conservative on purpose: a
# variant only scores if the claim is provably still there.
destructive = {"remove_block", "remove_section", "replace_block", "replace_section_blocks", "rename_section"}
surviving = "" if any(op.get("op") in destructive for op in ops) else document
added = " ".join(json.dumps(op.get("text") or op.get("blocks") or "", ensure_ascii=False) for op in ops)
final = f"{surviving} {added}"
return all(token in final for token in must_keep)
def _client() -> genai.Client:
key = os.getenv("HINDSIGHT_EVAL_LLM_API_KEY") or os.getenv("GEMINI_API_KEY") or ""
if not key:
raise SystemExit("Set HINDSIGHT_EVAL_LLM_API_KEY or GEMINI_API_KEY")
return genai.Client(api_key=key)
def _turn(role: str, text: str) -> types.Content:
return types.Content(role=role, parts=[types.Part(text=text)])
async def _generate(client: genai.Client, model: str, system: str, turns: list[types.Content]) -> str:
reply = await client.aio.models.generate_content(
model=model,
contents=turns,
config=types.GenerateContentConfig(system_instruction=system, temperature=0.0),
)
return reply.text or ""
async def score(
client: genai.Client, model: str, system: str, captured: CapturedRequest, must_keep: list[str], repeats: int
) -> VariantScore:
result = VariantScore(kept=0)
for _ in range(repeats):
reply = await _generate(client, model, system, [_turn("user", captured.user)])
if survives(reply, must_keep, captured.document):
result.kept += 1
else:
result.failures.append(reply[:200])
return result
async def interrogate(client: genai.Client, model: str, captured: CapturedRequest, expected: str) -> None:
chat = [_turn("user", captured.user)]
first = await _generate(client, model, captured.system, chat)
chat.append(_turn("model", first))
why = (
f"The correct result was: {expected}\n\n"
"Compare that with the operations you just emitted. Explain what in the prompt led you to your "
"choice, quoting the specific lines you relied on. Be concrete and do not be agreeable for its "
"own sake — if the prompt was unambiguous and you simply erred, say so."
)
chat.append(_turn("user", why))
answer = await _generate(client, model, captured.system, chat)
print(f"\n=== WHY ===\n{answer.strip()[:2500]}")
chat += [
_turn("model", answer),
_turn(
"user",
"Now propose the minimal edit to the SYSTEM PROMPT that would have produced the correct result. "
"It must not stop you superseding genuinely outdated content and must be a few lines. Give the "
"exact text, and which existing line it replaces if any.",
),
]
proposal = await _generate(client, model, captured.system, chat)
print(f"\n=== PROPOSED PROMPT FIX ===\n{proposal.strip()[:2500]}")
async def run(args: argparse.Namespace) -> None:
captured = load(args.input)
client = _client()
must_keep = args.must_keep.split("|")
candidates = {"captured (as the server sent it)": captured.system}
if args.variants:
candidates.update(json.loads(args.variants.read_text(encoding="utf-8")))
print(f"claim that must survive: {must_keep} ({args.repeats} runs each, {args.model})\n")
for name, system in candidates.items():
result = await score(client, args.model, system, captured, must_keep, args.repeats)
print(f"{result.kept}/{args.repeats} {name}")
if result.failures:
print(f" first failure: {result.failures[0][:160]}")
if args.interrogate:
await interrogate(client, args.model, captured, args.expected)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--input", type=Path, required=True, help="a prompt dumped by diagnose_page --dump")
parser.add_argument("--must-keep", required=True, help="pipe-separated tokens that must survive in the document")
parser.add_argument("--repeats", type=int, default=5)
parser.add_argument("--variants", type=Path, help="JSON file of {name: system_prompt} to try")
parser.add_argument("--model", default=os.getenv("HINDSIGHT_EVAL_LLM_MODEL", "gemini-3.7-flash"))
parser.add_argument("--interrogate", action="store_true", help="continue the chat: why, then how to fix")
parser.add_argument("--expected", default="", help="the correct result, for --interrogate")
args = parser.parse_args()
if args.interrogate and not args.expected:
parser.error("--interrogate needs --expected")
asyncio.run(run(args))
if __name__ == "__main__":
main()
hindsight-system-evals/hindsight_system_evals/judge.py
+130
View File
@@ -0,0 +1,130 @@
"""An independent LLM judge.
Two rules carried over from ``hindsight-api-slim/tests/llm_judge.py``, both
load-bearing:
* **The judge must not be the model under test.** Judging an answer with the same
model that wrote it measures agreement with itself. It is configured separately
from the server's provider; the fixtures warn when the two resolve to the same
model.
* **A single temperature-0 verdict flips on borderline phrasing.** A "not met" is
re-asked at a higher temperature and upheld only on majority agreement, so one
noisy sample cannot fail a run. Verdicts that pass first time cost one call.
Two backends, because the environments differ: a developer usually has a Gemini
API key, while the perf workflow authenticates every quality benchmark with a
VertexAI service account. Both go through ``google-genai`` — never through
``hindsight_api``, so nothing here can depend on engine internals.
"""
from __future__ import annotations
import asyncio
import json
import os
from dataclasses import dataclass
from functools import cache
from google import genai
from google.genai import types
_CONFIRMATIONS = int(os.getenv("HINDSIGHT_EVAL_JUDGE_CONFIRMATIONS", "2"))
_CONFIRM_TEMPERATURE = float(os.getenv("HINDSIGHT_EVAL_JUDGE_CONFIRM_TEMPERATURE", "0.5"))
_SYSTEM = (
"You grade whether an answer satisfies a stated criterion. Judge ONLY the criterion "
"given — not style, length, or anything else. Reply with a single JSON object: "
'{"meets_criteria": true|false, "reasoning": "<one sentence>"}. No prose outside it.'
)
@dataclass(frozen=True)
class Verdict:
meets_criteria: bool
reasoning: str
def judge_model() -> str:
# The perf workflow names Vertex models with a "google/" prefix; the SDK wants
# the bare name. Accepting both keeps one env convention across benchmarks.
return os.getenv("HINDSIGHT_EVAL_JUDGE_MODEL", "gemini-2.5-flash").removeprefix("google/")
def _judge_api_key() -> str:
return os.getenv("HINDSIGHT_EVAL_JUDGE_API_KEY") or os.getenv("GEMINI_API_KEY") or ""
@cache
def _client() -> genai.Client:
"""An API-key client when a key is set, otherwise VertexAI from a service account."""
provider = os.getenv("HINDSIGHT_EVAL_JUDGE_PROVIDER", "gemini" if _judge_api_key() else "vertexai")
if provider == "gemini":
key = _judge_api_key()
if not key:
raise RuntimeError(
"The judge needs HINDSIGHT_EVAL_JUDGE_API_KEY (or GEMINI_API_KEY), or "
"HINDSIGHT_EVAL_JUDGE_PROVIDER=vertexai with a service account. It is configured "
"separately from the server on purpose — a model grading its own output agrees with itself."
)
return genai.Client(api_key=key)
from google.oauth2 import service_account
key_file = os.getenv("HINDSIGHT_EVAL_JUDGE_VERTEXAI_SERVICE_ACCOUNT_KEY") or os.getenv(
"HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY", ""
)
project = os.getenv("HINDSIGHT_EVAL_JUDGE_VERTEXAI_PROJECT_ID") or os.getenv(
"HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID", ""
)
if not (key_file and project):
raise RuntimeError(
"A VertexAI judge needs a service-account key file and project id "
"(HINDSIGHT_EVAL_JUDGE_VERTEXAI_* or the HINDSIGHT_API_LLM_VERTEXAI_* equivalents)."
)
credentials = service_account.Credentials.from_service_account_file(
key_file, scopes=["https://www.googleapis.com/auth/cloud-platform"]
)
location = os.getenv("HINDSIGHT_EVAL_JUDGE_VERTEXAI_REGION") or os.getenv(
"HINDSIGHT_API_LLM_VERTEXAI_REGION", "us-central1"
)
return genai.Client(vertexai=True, project=project, location=location, credentials=credentials)
async def _judge_once(response: str, criteria: str, context: str | None, temperature: float) -> Verdict:
prompt = (
f"## Context\n{context}\n\n" if context else ""
) + f"## Criterion\n{criteria}\n\n## Answer to grade\n{response}"
reply = await _client().aio.models.generate_content(
model=judge_model(),
contents=prompt,
config=types.GenerateContentConfig(
system_instruction=_SYSTEM,
temperature=temperature,
response_mime_type="application/json",
),
)
text = reply.text or ""
try:
parsed = json.loads(text[text.index("{") : text.rindex("}") + 1])
except ValueError:
# A judge that returned something unreadable must not silently pass the
# thing it was asked to check.
return Verdict(False, f"judge reply was not JSON: {text[:200]}")
return Verdict(bool(parsed.get("meets_criteria")), str(parsed.get("reasoning", "")))
async def evaluate(response: str, criteria: str, context: str | None = None) -> Verdict:
"""Grade ``response`` against ``criteria``, smoothing single-call judge noise."""
primary = await _judge_once(response, criteria, context, temperature=0.0)
if primary.meets_criteria or _CONFIRMATIONS <= 0:
return primary
confirmations = await asyncio.gather(
*(_judge_once(response, criteria, context, _CONFIRM_TEMPERATURE) for _ in range(_CONFIRMATIONS)),
return_exceptions=True,
)
verdicts = [primary] + [v for v in confirmations if isinstance(v, Verdict)]
met = sum(1 for v in verdicts if v.meets_criteria)
if met > len(verdicts) - met:
return Verdict(True, f"majority of {len(verdicts)} judges met the criteria (primary overruled as noise)")
return Verdict(False, f"{len(verdicts) - met}/{len(verdicts)} judges agree: {primary.reasoning}")
hindsight-system-evals/hindsight_system_evals/pages.py
+177
View File
@@ -0,0 +1,177 @@
"""Build a knowledge page the way a caller does, and report what it converged to.
The scenario, end to end through the public API:
create page (source_query = the question)
-> retain wave 1 -> wait for the refresh -> snapshot
-> retain wave 2 -> wait for the refresh -> snapshot
-> grade the FINAL page
The waves are the point. A page is not written once; it accumulates, and each
delta refresh edits what is already stored. That is where a wrong answer stops
being a wrong answer and becomes a wrong *memory* — every later reflect reads it
back as fact. A single-wave build would never exercise a delta edit at all.
Waves never split a subject. Facts about one release travelling in different
waves make the later batch read as a correction of the earlier one, and a refresh
that supersedes on that basis is following its rules correctly — the corpus would
be the bug. ``split_into_waves`` keeps a subject together.
"""
from __future__ import annotations
from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field
from hindsight_client import Hindsight
from hindsight_client_api.models.create_knowledge_page_response import CreateKnowledgePageResponse
from hindsight_system_evals import corpus as corpus_module
from hindsight_system_evals.corpus import HardFact, HardQuestion
#: Waits until a bank has no pending work — the ``settled`` fixture.
SettleFn = Callable[[str], Awaitable[None]]
#: Two is the minimum that exercises a delta at all: the first wave writes the
#: page, the second has to edit it.
WAVES = 2
@dataclass
class WaveSnapshot:
wave: int
facts_ingested: int
content: str = ""
@property
def chars(self) -> int:
return len(self.content)
@dataclass
class PageOutcome:
"""One page, built across every wave, with the evidence to explain a failure."""
question_id: str
category: str
bank_id: str
page_id: str = ""
mental_model_id: str = ""
waves: list[WaveSnapshot] = field(default_factory=list)
final_content: str = ""
correct: bool = False
correct_reason: str = ""
hit_trap: bool = False
trap_reason: str = ""
def split_into_waves(facts: list[HardFact], count: int = WAVES) -> list[list[HardFact]]:
"""Split the corpus into waves without separating any subject's facts."""
by_subject: dict[str, list[HardFact]] = {}
for index, fact in enumerate(facts):
# A fact with no subject cannot contradict anything, so it is its own group.
key = f"{fact.cluster}:{fact.subject}" if fact.subject else f"_solo:{index}"
by_subject.setdefault(key, []).append(fact)
waves: list[list[HardFact]] = [[] for _ in range(count)]
for position, key in enumerate(sorted(by_subject)):
waves[position % count].extend(by_subject[key])
return waves
def questions() -> list[HardQuestion]:
return corpus_module.build().questions
def facts() -> list[HardFact]:
return corpus_module.build().facts
async def prepare_bank(client: Hindsight, bank_id: str) -> None:
"""Configure the bank so seeding the corpus costs no model calls.
Measured on the first blackbox run: of 269s wall time per page, 789s of LLM
time went to fact extraction (one call per one-sentence fact) and 199s to
consolidation, against 21s for the reflect and delta calls actually being
evaluated. Neither is under test here:
* ``chunks`` stores each item as written, with no extraction call. That also
removes a confound: extraction may paraphrase a fact, and the gold labels
point at the exact authored text.
* consolidation and observations produce rows the page never reads — its
trigger reads raw ``world``/``experience`` facts.
This is ordinary per-bank configuration through the public config endpoint,
so the eval is still blackbox.
"""
await client.aupdate_bank_config(
bank_id,
retain_extraction_mode="chunks",
enable_observations=False,
enable_auto_consolidation=False,
)
async def create_page(client: Hindsight, bank_id: str, question: HardQuestion) -> CreateKnowledgePageResponse:
"""Create the page for one question, reading raw facts and refreshed explicitly."""
return await client.knowledge_base.create_knowledge_page(
bank_id,
{
"name": f"eval {question.id}",
"source_query": question.question,
# Pages default to observation-only, which would make the eval depend
# on consolidation writing usable observations first. Reading raw
# facts keeps the corpus text exactly as authored, which is what the
# gold labels point at.
"trigger": {
"mode": "delta",
"fact_types": ["world", "experience"],
"exclude_mental_models": True,
# Refreshed explicitly below instead of after consolidation, because
# consolidation is switched off for this bank — see prepare_bank.
"refresh_after_consolidation": False,
},
},
)
async def build_page(
client: Hindsight,
bank_id: str,
question: HardQuestion,
waves: list[list[HardFact]],
settle: SettleFn,
) -> PageOutcome:
"""Create the page, feed it the corpus wave by wave, and snapshot each time."""
outcome = PageOutcome(question_id=question.id, category=question.category, bank_id=bank_id)
await prepare_bank(client, bank_id)
created = await create_page(client, bank_id, question)
outcome.page_id = created.page_id
outcome.mental_model_id = created.mental_model_id
await settle(bank_id)
for index, wave in enumerate(waves, start=1):
await client.aretain_batch(bank_id=bank_id, items=[{"content": fact.text} for fact in wave])
await settle(bank_id)
# The refresh is the thing under test, so it goes through the same public
# endpoint and the same worker as any caller's — only the trigger is
# explicit rather than consolidation-driven.
await client.arefresh_mental_model(bank_id=bank_id, mental_model_id=outcome.mental_model_id)
await settle(bank_id)
content = await read_page(client, bank_id, outcome.mental_model_id)
outcome.waves.append(WaveSnapshot(wave=index, facts_ingested=len(wave), content=content))
outcome.final_content = outcome.waves[-1].content if outcome.waves else ""
return outcome
async def read_page(client: Hindsight, bank_id: str, mental_model_id: str) -> str:
"""The page's stored content, read through the public API.
A page is a tree node backed by a mental model; the content lives on the
model, so that is what gets read. Still blackbox — this is the same endpoint
any caller uses.
"""
model = await client.aget_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
return getattr(model, "content", "") or ""
hindsight-system-evals/hindsight_system_evals/report.py
+78
View File
@@ -0,0 +1,78 @@
"""The JSON a run publishes to the continuous performance monitor.
The field names are the dashboard's contract (``system-evals.html`` and
``publish-system-evals-results.sh`` read them), so they live in one typed place
rather than as dict literals spread across the test modules and conftest.
"""
from __future__ import annotations
from typing import Literal
from pydantic import BaseModel, Field
EvalKind = Literal["knowledge_page", "reflect"]
class EvalRecord(BaseModel):
kind: EvalKind
question_id: str
category: str
correct: bool
hit_trap: bool
bank_id: str
reason: str
page_id: str = ""
#: The page's size after each wave; empty for a one-shot reflect answer.
sizes: list[int] = Field(default_factory=list)
class KindSummary(BaseModel):
total: int
correct: int
trap_count: int
class ModelRef(BaseModel):
provider: str | None = None
model: str
class ModelConfig(BaseModel):
hindsight: ModelRef
judge: ModelRef
class RunReport(BaseModel):
timestamp: str
suite: str
mode: str
# ``model_config`` is reserved on pydantic models; the dashboard reads that
# key, so the field is renamed only on the way out.
llm_config: ModelConfig = Field(serialization_alias="model_config")
total: int
correct: int
#: The headline pair. The correct rate can dip on an incomplete answer; a trap
#: is a stored falsehood, so it is reported on its own and never averaged in.
correct_rate: float | None
trap_count: int
#: The same counts per suite, keyed by ``EvalKind``: a page and a one-shot
#: answer fail for different reasons and should not hide inside one rate.
by_kind: dict[str, KindSummary]
items: list[EvalRecord]
def to_json(self) -> str:
return self.model_dump_json(by_alias=True, indent=2)
def summarise(records: list[EvalRecord]) -> KindSummary:
return KindSummary(
total=len(records),
correct=sum(1 for r in records if r.correct),
trap_count=sum(1 for r in records if r.hit_trap),
)
#: Filled by the evals as they finish; written once at session end. A failed
#: assertion still records its outcome first, so a red run publishes what it saw.
RECORDED: list[EvalRecord] = []
hindsight-system-evals/hindsight_system_evals/server.py
+159
View File
@@ -0,0 +1,159 @@
"""Start a real ``hindsight-api`` against a REAL model provider.
Mirrors ``hindsight_system_tests.server`` deliberately — same scratch-directory
trick, same environment hygiene, same health wait — with one difference that is
the whole reason this package is separate: the system tests point every LLM call
at a stub, because a test wants determinism. An eval measuring answer quality
cannot do that. Stub the model and you score the stub.
The consequences follow from that and are not worked around:
* provider credentials are REQUIRED, so these evals cannot run on fork PRs, and
the CI job has to be gated on secrets — unlike ``test-system``;
* results are rates, not equalities, because the same request can produce a
different answer twice running.
"""
from __future__ import annotations
import contextlib
import os
import subprocess
import time
from dataclasses import dataclass
from pathlib import Path
import httpx
REPO_ROOT = Path(__file__).resolve().parents[2]
API_DIR = REPO_ROOT / "hindsight-api-slim"
#: Its own pg0 instance. Sharing one with the developer's server (or with the
#: system tests) means a run competes for connections and can be refused mid-way
#: with "sorry, too many clients already".
PG0_INSTANCE = "hindsight-system-evals"
SERVER_STARTUP_TIMEOUT = 180.0
@dataclass
class EvalServer:
url: str
log_path: Path
_process: subprocess.Popen
def logs(self) -> str:
return self.log_path.read_text(encoding="utf-8", errors="replace")
def stop(self) -> None:
self._process.terminate()
with contextlib.suppress(subprocess.TimeoutExpired):
self._process.wait(timeout=30)
if self._process.poll() is None:
self._process.kill()
def free_port() -> int:
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
return int(sock.getsockname()[1])
def provider_environment() -> dict[str, str]:
"""The model settings for the server under test, from this process's env.
Read explicitly rather than inherited: the server's environment is wiped
below, so anything not named here does not reach it. ``HINDSIGHT_EVAL_*``
wins over ``HINDSIGHT_API_*`` so a developer can point the evals at a model
other than the one their own server uses.
VertexAI authenticates with a service-account key file rather than an API
key — that is how the perf workflow runs every quality benchmark — so its
settings pass through as a group instead of being required individually.
"""
def pick(suffix: str) -> str:
return os.getenv(f"HINDSIGHT_EVAL_{suffix}") or os.getenv(f"HINDSIGHT_API_{suffix}") or ""
provider, model = pick("LLM_PROVIDER"), pick("LLM_MODEL")
env = {"HINDSIGHT_API_LLM_PROVIDER": provider, "HINDSIGHT_API_LLM_MODEL": model}
if provider == "vertexai":
for suffix in ("LLM_VERTEXAI_SERVICE_ACCOUNT_KEY", "LLM_VERTEXAI_PROJECT_ID", "LLM_VERTEXAI_REGION"):
if value := pick(suffix):
env[f"HINDSIGHT_API_{suffix}"] = value
credentialed = "HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY" in env
else:
if api_key := pick("LLM_API_KEY"):
env["HINDSIGHT_API_LLM_API_KEY"] = api_key
credentialed = "HINDSIGHT_API_LLM_API_KEY" in env
if not (provider and model and credentialed):
raise RuntimeError(
"System evals need a real model. Set HINDSIGHT_EVAL_LLM_PROVIDER / _MODEL and either "
"_API_KEY or, for vertexai, _VERTEXAI_SERVICE_ACCOUNT_KEY (HINDSIGHT_API_ equivalents "
"also work). A stub cannot be used here: it would score the stub."
)
if base_url := pick("LLM_BASE_URL"):
env["HINDSIGHT_API_LLM_BASE_URL"] = base_url
return env
def start_eval_server(*, log_path: Path) -> EvalServer:
port = free_port()
env = os.environ.copy()
# The repo .env is for the developer's own server. Left in place it would
# decide which model these evals measure, and a run would mean something
# different on every machine.
for key in list(env):
if key.startswith("HINDSIGHT_API_"):
del env[key]
env.update(provider_environment())
env.update(
{
"HINDSIGHT_API_DATABASE_URL": f"pg0://{PG0_INSTANCE}",
"HINDSIGHT_API_HOST": "127.0.0.1",
"HINDSIGHT_API_PORT": str(port),
"HINDSIGHT_API_LOG_LEVEL": "info",
# The debug tools replay a captured delta-ops request verbatim, and the
# default 50k-char cap truncates a large one mid-document. A replay of a
# truncated prompt is a different prompt.
"HINDSIGHT_API_LLM_TRACE_MAX_CHARS": "1000000",
}
)
# `hindsight-api` loads a discovered .env with override=True (#2961), so
# started from inside the repo it would ignore everything set above. An empty
# .env in a scratch directory stops the upward walk. Same trick as the system
# tests, and for the same reason.
run_dir = log_path.parent / "run"
run_dir.mkdir(parents=True, exist_ok=True)
(run_dir / ".env").write_text("# intentionally empty: see start_eval_server\n")
log_file = log_path.open("w")
process = subprocess.Popen(
["uv", "run", "--project", str(API_DIR), "hindsight-api"],
cwd=run_dir,
env=env,
stdout=log_file,
stderr=subprocess.STDOUT,
)
server = EvalServer(url=f"http://127.0.0.1:{port}", log_path=log_path, _process=process)
_wait_until_healthy(server)
return server
def _wait_until_healthy(server: EvalServer) -> None:
deadline = time.monotonic() + SERVER_STARTUP_TIMEOUT
while time.monotonic() < deadline:
if server._process.poll() is not None:
raise RuntimeError(f"hindsight-api exited during startup:\n{server.logs()}")
with contextlib.suppress(httpx.HTTPError):
if httpx.get(f"{server.url}/health", timeout=5).status_code == 200:
return
time.sleep(0.5)
server.stop()
raise RuntimeError(f"hindsight-api was not healthy within {SERVER_STARTUP_TIMEOUT}s:\n{server.logs()}")
hindsight-system-evals/hindsight_system_evals/waiting.py
+91
View File
@@ -0,0 +1,91 @@
"""Wait for the asynchronous half of the system to finish.
A retain returns as soon as the facts are stored, but the server is not done: the
worker still has observation extraction and possibly consolidation to run. Those
are not noise to be suppressed — they are where a large share of this project's
composition bugs live — so tests let them run and wait for them here.
The wait is expressed through the public operations API, like everything else in
this suite. It settles when the bank has nothing pending or processing, and it
fails loudly on a failed operation rather than letting a test assert against a
half-built bank and call it a pass.
"""
from __future__ import annotations
import asyncio
import os
import time
from hindsight_client import Hindsight
# Far longer than the system tests use, and for a concrete reason: there a stub
# answers instantly, so 90s is generous. Here a settle waits on real fact
# extraction AND consolidation over the whole corpus, each a real model call. The
# first run of this suite failed at 90s with `consolidation(processing)` still in
# flight — the eval was fine, the inherited timeout was not.
SETTLE_TIMEOUT_SECONDS = float(os.getenv("HINDSIGHT_EVAL_SETTLE_TIMEOUT", "900"))
# Polling five times a second against a real server for fifteen minutes is a lot
# of pointless requests; a settle here takes minutes, not milliseconds.
POLL_INTERVAL_SECONDS = 1.0
# The bank must look idle for two consecutive polls before we believe it. One
# quiet poll proves nothing: a worker that has just claimed the next task, or an
# operation enqueued a moment after the previous one completed, both read as
# "nothing in flight" for an instant.
_CONSECUTIVE_QUIET_POLLS = 2
_BUSY_STATUSES = ("pending", "processing")
async def wait_until_settled(
client: Hindsight,
bank_id: str,
*,
timeout: float = SETTLE_TIMEOUT_SECONDS,
) -> None:
"""Block until the bank's background work has finished.
Raises on a failed operation, and on timeout reports what was still in flight
— a bare "timed out" would send the reader to the server log for something the
API could have told them.
"""
deadline = time.monotonic() + timeout
quiet_polls = 0
in_flight: list[str] = []
while time.monotonic() < deadline:
failed = await client.operations.list_operations(bank_id, status="failed", limit=100)
if failed.operations:
summary = ", ".join(
f"{op.task_type}: {op.error_message or 'no error recorded'}" for op in failed.operations
)
raise AssertionError(f"bank {bank_id} has failed background operations — {summary}")
busy = [
op
for status in _BUSY_STATUSES
for op in (await client.operations.list_operations(bank_id, status=status, limit=100)).operations
]
# An operation carrying an error is already lost, even while its status
# still reads `pending`: that is the worker holding it for a retry. With a
# deterministic stub the retry gets the same answer, so waiting out the
# backoff only delays the same failure — report it now, with the message.
errored = [op for op in busy if op.error_message]
if errored:
summary = ", ".join(f"{op.task_type}: {op.error_message}" for op in errored)
raise AssertionError(f"bank {bank_id} has a background operation that failed — {summary}")
in_flight = [f"{op.task_type}({op.status})" for op in busy]
if in_flight:
quiet_polls = 0
else:
quiet_polls += 1
if quiet_polls >= _CONSECUTIVE_QUIET_POLLS:
return
await asyncio.sleep(POLL_INTERVAL_SECONDS)
raise AssertionError(f"bank {bank_id} did not settle within {timeout}s; still in flight: {in_flight or 'nothing'}")
hindsight-system-evals/pyproject.toml
+54
View File
@@ -0,0 +1,54 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "hindsight-system-evals"
version = "0.1.0"
description = "Blackbox quality evals: a real Hindsight server and a real model, scored by an independent judge"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"pytest>=7.0.0",
"pytest-asyncio>=0.21.0",
"pytest-timeout>=2.4.0",
"httpx>=0.27.0",
"pydantic>=2",
"pyyaml>=6",
# The judge only — never the server under test. Covers both a developer's
# Gemini API key and the perf workflow's VertexAI service account.
"google-genai>=1.0.0",
"google-auth>=2.0.0",
"hindsight-client",
]
[tool.uv.sources]
# editable, for the same reason the system tests give: a copied client silently
# freezes the API surface at whenever the venv was last built.
hindsight-client = { path = "../hindsight-clients/python", editable = true }
[tool.hatch.build.targets.wheel]
packages = ["hindsight_system_evals"]
[tool.pytest.ini_options]
log_cli = true
log_cli_level = "INFO"
log_cli_format = "%(asctime)s - %(levelname)s - %(name)s - %(message)s"
log_cli_date_format = "%Y-%m-%d %H:%M:%S"
# Must exceed HINDSIGHT_EVAL_SETTLE_TIMEOUT (900s default), or pytest kills a
# test in the middle of the settle it is legitimately waiting on. One eval waits
# on fact extraction, consolidation and a refresh, all real model calls.
addopts = "--timeout 2400 -v"
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"
markers = [
"full: the complete eval set; deselected in minimum-acceptance runs",
]
[tool.ruff]
line-length = 120
target-version = "py311"
[tool.ruff.lint]
select = ["E", "W", "F", "I"]
ignore = ["E501"]
hindsight-system-evals/uv.lock
Generated
+1594
View File
File diff suppressed because it is too large Load Diff
scripts/benchmarks/publish-system-evals-results.sh
Executable
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env bash
# Publish system-evals (knowledge-page convergence) results to the dashboard repo's gh-pages branch.
#
# Reads a system-evals JSON (from `pytest evals --output`), enriches it with
# commit + workflow metadata, then pushes:
# data/system-evals/<timestamp>-<short_sha>.json
# data/system-evals-index.json (manifest, newest first)
# to vectorize-io/hindsight-continuous-performance-monitor (gh-pages). The static
# site's system-evals.html reads data/system-evals-index.json + the run JSONs and
# charts the correct rate (higher is better) and the trap count (must be 0).
#
# Required env:
# PERF_DASHBOARD_TOKEN PAT with Contents:write on the dashboard repo
#
# Usage:
# ./scripts/benchmarks/publish-system-evals-results.sh path/to/system-evals-results.json
set -euo pipefail
INPUT_JSON="${1:?usage: $0 <system-evals-results.json>}"
DASHBOARD_REPO="${DASHBOARD_REPO:-vectorize-io/hindsight-continuous-performance-monitor}"
HINDSIGHT_REPO="${HINDSIGHT_REPO:-vectorize-io/hindsight}"
if [ ! -f "$INPUT_JSON" ]; then
echo "Input JSON not found: $INPUT_JSON" >&2
exit 1
fi
: "${PERF_DASHBOARD_TOKEN:?PERF_DASHBOARD_TOKEN must be set}"
# ─────────────────────────────────────────────────────────────────────────
# Capture commit + workflow metadata
# ─────────────────────────────────────────────────────────────────────────
SHA=$(git rev-parse HEAD)
SHORT_SHA=$(git rev-parse --short=8 HEAD)
SUBJECT=$(git log -1 --pretty=%s)
AUTHOR=$(git log -1 --pretty=%an)
AUTHOR_DATE=$(git log -1 --pretty=%aI)
COMMIT_URL="https://github.com/${HINDSIGHT_REPO}/commit/${SHA}"
PR_NUMBER=""
PR_URL=""
if command -v gh >/dev/null 2>&1; then
PR_NUMBER=$(gh api "repos/${HINDSIGHT_REPO}/commits/${SHA}/pulls" \
--jq '.[0].number // empty' 2>/dev/null || true)
if [ -n "$PR_NUMBER" ]; then
PR_URL="https://github.com/${HINDSIGHT_REPO}/pull/${PR_NUMBER}"
fi
fi
RUN_ID="${GITHUB_RUN_ID:-}"
RUN_URL=""
if [ -n "$RUN_ID" ]; then
RUN_REPO="${GITHUB_REPOSITORY:-$HINDSIGHT_REPO}"
RUN_URL="https://github.com/${RUN_REPO}/actions/runs/${RUN_ID}"
fi
TIMESTAMP_FILE=$(date -u +%Y%m%dT%H%M%SZ)
DATA_FILE="data/system-evals/${TIMESTAMP_FILE}-${SHORT_SHA}.json"
echo "Publishing system-evals run for ${SHORT_SHA} → ${DATA_FILE}"
# ─────────────────────────────────────────────────────────────────────────
# Enrich the input JSON (the benchmark already carries timestamp + metrics)
# ─────────────────────────────────────────────────────────────────────────
ENRICHED_TMP=$(mktemp)
trap 'rm -f "$ENRICHED_TMP"' EXIT
jq \
--arg sha "$SHA" \
--arg short_sha "$SHORT_SHA" \
--arg subject "$SUBJECT" \
--arg author "$AUTHOR" \
--arg author_date "$AUTHOR_DATE" \
--arg commit_url "$COMMIT_URL" \
--arg pr_number "$PR_NUMBER" \
--arg pr_url "$PR_URL" \
--arg run_id "$RUN_ID" \
--arg run_url "$RUN_URL" \
'. + {
commit: {
sha: $sha,
short_sha: $short_sha,
subject: $subject,
author: $author,
author_date: $author_date,
url: $commit_url,
pr_number: ($pr_number | if . == "" then null else tonumber end),
pr_url: (if $pr_url == "" then null else $pr_url end)
},
workflow_run: (if $run_url == "" then null else {id: $run_id, url: $run_url} end)
}' "$INPUT_JSON" > "$ENRICHED_TMP"
RUN_TIMESTAMP=$(jq -r '.timestamp' "$ENRICHED_TMP")
CORRECT_RATE=$(jq -r '.correct_rate' "$ENRICHED_TMP")
TRAP_COUNT=$(jq -r '.trap_count // 0' "$ENRICHED_TMP")
TOTAL=$(jq -r '.total // 0' "$ENRICHED_TMP")
MODE=$(jq -r '.mode // "full"' "$ENRICHED_TMP")
# ─────────────────────────────────────────────────────────────────────────
# Clone dashboard repo's gh-pages branch
# ─────────────────────────────────────────────────────────────────────────
WORK=$(mktemp -d)
trap 'rm -f "$ENRICHED_TMP"; rm -rf "$WORK"' EXIT
git clone --quiet --depth 1 --branch gh-pages \
"https://x-access-token:${PERF_DASHBOARD_TOKEN}@github.com/${DASHBOARD_REPO}.git" \
"$WORK"
mkdir -p "$WORK/data/system-evals"
cp "$ENRICHED_TMP" "$WORK/$DATA_FILE"
# ─────────────────────────────────────────────────────────────────────────
# Update manifest (data/system-evals-index.json)
# ─────────────────────────────────────────────────────────────────────────
NEW_ENTRY=$(jq -n \
--arg sha "$SHA" \
--arg short_sha "$SHORT_SHA" \
--arg subject "$SUBJECT" \
--arg author "$AUTHOR" \
--arg author_date "$AUTHOR_DATE" \
--arg commit_url "$COMMIT_URL" \
--arg pr_url "$PR_URL" \
--arg run_url "$RUN_URL" \
--arg data_file "$DATA_FILE" \
--arg timestamp "$RUN_TIMESTAMP" \
--argjson correct_rate "$CORRECT_RATE" \
--argjson trap_count "$TRAP_COUNT" \
--argjson total "$TOTAL" \
--arg mode "$MODE" \
'{
sha: $sha,
short_sha: $short_sha,
subject: $subject,
author: $author,
author_date: $author_date,
commit_url: $commit_url,
pr_url: (if $pr_url == "" then null else $pr_url end),
run_url: (if $run_url == "" then null else $run_url end),
data_file: $data_file,
timestamp: $timestamp,
correct_rate: $correct_rate,
trap_count: $trap_count,
total: $total,
mode: $mode
}')
INDEX_FILE="$WORK/data/system-evals-index.json"
if [ ! -f "$INDEX_FILE" ]; then
echo '{"runs": []}' > "$INDEX_FILE"
fi
UPDATED_INDEX=$(jq \
--argjson entry "$NEW_ENTRY" \
'.runs = ([$entry] + (.runs // [])) | .updated_at = (now | todateiso8601)' \
"$INDEX_FILE")
echo "$UPDATED_INDEX" > "$INDEX_FILE"
# ─────────────────────────────────────────────────────────────────────────
# Commit and push
# ─────────────────────────────────────────────────────────────────────────
cd "$WORK"
git config user.name 'hindsight-perf-bot'
git config user.email 'hindsight-perf-bot@users.noreply.github.com'
git add data/
if git diff --cached --quiet; then
echo "No changes to commit (this shouldn't happen — skipping push)" >&2
exit 0
fi
git commit --quiet -m "system-evals: add results for ${SHORT_SHA}"
if ! git push --quiet origin gh-pages; then
echo "Push rejected, pulling and retrying..." >&2
git pull --quiet --rebase origin gh-pages
git push --quiet origin gh-pages
fi
echo "Published ${DATA_FILE} to ${DASHBOARD_REPO} gh-pages"