mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
feat(coding-agents): fall back to knowledge pages, then observations, when the session reflect fails (#4324)
A reflect that times out or 5xxs used to leave the session with no memory. Now the hook searches knowledge pages for the goal and injects the matches; if none match, it injects a raw recall of the bank's observations. 4xx and transport errors still skip memory, since every endpoint would fail the same way. The reflect cap drops from 25s to 20s so the 7s fallback budget still fits the host's 30s hook window.
This commit is contained in:
Nicolò Boschi
@@ -519,7 +519,7 @@ hook by Codex...), so one shared config serves several agents side by side:
|
||||
| `manageBankConfig` | `true` | let the plugin shape the bank's own configuration — the retain strategies it writes under, the `knowledge` entity-label group, and, on a bank that has none, the missions. Writing is strictly **additive**: it adds what the bank does not define and never overwrites what is there, so your control-plane edits survive. Set `false` to keep it out of the bank config entirely — see **A bank you shape yourself** below |
|
||||
| `observationScopes` | `"shared"` | how consolidation groups observations: `"shared"` (default) = ONE global scope per bank, so every agent on a repo builds one set of beliefs; also `"combined"` (the server default), `"per_tag"`, `"all_combinations"`, `[["t"]]`; `"per_source"` adds a scope per `source:` kind alongside the global one, so commit knowledge and conversation knowledge consolidate apart |
|
||||
| `disabled` | `false` | hard off-switch (inert plugin/hook — a no-memory baseline) |
|
||||
| `reflectTimeoutMs` | `120000` | **automatic** session-reflect timeout (hook harnesses additionally cap it at 25s to fit the host's hook window); on timeout the session runs without reflect (recorded) |
|
||||
| `reflectTimeoutMs` | `120000` | **automatic** session-reflect timeout (hook harnesses additionally cap it at 20s to fit the host's hook window); on timeout or a 5xx the hook falls back to knowledge-page search, then to a raw recall of observations (recorded) |
|
||||
| `reflectToolTimeoutMs` | `330000` | timeout for the agent-invoked `hindsight_reflect` tool — a call the agent waits on, whose high-budget synthesis on a populated bank runs for minutes. Defaults above the server's own reflect wall timeout (`HINDSIGHT_API_REFLECT_WALL_TIMEOUT`, 300s) so the server decides when to give up. Unset, it inherits an explicitly raised `reflectTimeoutMs`, but a short one never lowers it |
|
||||
| `reflectBudget` | `"high"` | reflect budget for the `hindsight_reflect` tool: `"low"`, `"mid"` or `"high"`. Drop it on a large bank where high-budget synthesis exceeds the server's wall timeout. The automatic session-start reflect always uses `"low"` to fit its hook window and is unaffected |
|
||||
| `autoReflect` | `true` | inject a one-time reflect synthesis on the session's **first prompt**. `false` = tool-only reflect: nothing is injected; the agent searches knowledge pages first and reflects only when they are too shallow |
|
||||
@@ -797,7 +797,10 @@ as a JSON line to `/tmp/hindsight-plugin.log` (override with `HINDSIGHT_DIAG_FIL
|
||||
```
|
||||
|
||||
`reflect_failed` / `pages_failed` record the error; if you're comparing memory-on vs memory-off,
|
||||
check this file — a run whose reflects failed is a no-memory run. Seed starts are logged as
|
||||
check this file — a run whose reflects failed is a no-memory run. When the failure was a timeout or
|
||||
a 5xx, the hook falls back to knowledge-page search and, if no page matches, to a raw recall of the
|
||||
bank's observations: `reflect_fallback_pages` / `reflect_fallback_observations` record what each
|
||||
step returned (`*_failed` when it errored). Seed starts are logged as
|
||||
`seed_started`.
|
||||
|
||||
### Is the memory ready yet?
|
||||
|
||||
@@ -522,7 +522,7 @@ hook by Codex...), so one shared config serves several agents side by side:
|
||||
| `manageBankConfig` | `true` | let the plugin shape the bank's own configuration — the retain strategies it writes under, the `knowledge` entity-label group, and, on a bank that has none, the missions. Writing is strictly **additive**: it adds what the bank does not define and never overwrites what is there, so your control-plane edits survive. Set `false` to keep it out of the bank config entirely — see **A bank you shape yourself** below |
|
||||
| `observationScopes` | `"shared"` | how consolidation groups observations: `"shared"` (default) = ONE global scope per bank, so every agent on a repo builds one set of beliefs; also `"combined"` (the server default), `"per_tag"`, `"all_combinations"`, `[["t"]]`; `"per_source"` adds a scope per `source:` kind alongside the global one, so commit knowledge and conversation knowledge consolidate apart |
|
||||
| `disabled` | `false` | hard off-switch (inert plugin/hook — a no-memory baseline) |
|
||||
| `reflectTimeoutMs` | `120000` | **automatic** session-reflect timeout (hook harnesses additionally cap it at 25s to fit the host's hook window); on timeout the session runs without reflect (recorded) |
|
||||
| `reflectTimeoutMs` | `120000` | **automatic** session-reflect timeout (hook harnesses additionally cap it at 20s to fit the host's hook window); on timeout or a 5xx the hook falls back to knowledge-page search, then to a raw recall of observations (recorded) |
|
||||
| `reflectToolTimeoutMs` | `330000` | timeout for the agent-invoked `hindsight_reflect` tool — a call the agent waits on, whose high-budget synthesis on a populated bank runs for minutes. Defaults above the server's own reflect wall timeout (`HINDSIGHT_API_REFLECT_WALL_TIMEOUT`, 300s) so the server decides when to give up. Unset, it inherits an explicitly raised `reflectTimeoutMs`, but a short one never lowers it |
|
||||
| `reflectBudget` | `"high"` | reflect budget for the `hindsight_reflect` tool: `"low"`, `"mid"` or `"high"`. Drop it on a large bank where high-budget synthesis exceeds the server's wall timeout. The automatic session-start reflect always uses `"low"` to fit its hook window and is unaffected |
|
||||
| `autoReflect` | `true` | inject a one-time reflect synthesis on the session's **first prompt**. `false` = tool-only reflect: nothing is injected; the agent searches knowledge pages first and reflects only when they are too shallow |
|
||||
@@ -845,7 +845,10 @@ as a JSON line to `/tmp/hindsight-plugin.log` (override with `HINDSIGHT_DIAG_FIL
|
||||
```
|
||||
|
||||
`reflect_failed` / `pages_failed` record the error; if you're comparing memory-on vs memory-off,
|
||||
check this file — a run whose reflects failed is a no-memory run. Seed starts are logged as
|
||||
check this file — a run whose reflects failed is a no-memory run. When the failure was a timeout or
|
||||
a 5xx, the hook falls back to knowledge-page search and, if no page matches, to a raw recall of the
|
||||
bank's observations: `reflect_fallback_pages` / `reflect_fallback_observations` record what each
|
||||
step returned (`*_failed` when it errored). Seed starts are logged as
|
||||
`seed_started`.
|
||||
|
||||
### Is the memory ready yet?
|
||||
|
||||
@@ -204,7 +204,7 @@ hook by Codex...), so one shared config serves several agents side by side:
|
||||
| `manageBankConfig` | `true` | let the plugin shape the bank's own configuration — the retain strategies it writes under, the `knowledge` entity-label group, and, on a bank that has none, the missions. Writing is strictly **additive**: it adds what the bank does not define and never overwrites what is there, so your control-plane edits survive. Set `false` to keep it out of the bank config entirely — see **A bank you shape yourself** below |
|
||||
| `observationScopes` | `"shared"` | how consolidation groups observations: `"shared"` (default) = ONE global scope per bank, so every agent on a repo builds one set of beliefs; also `"combined"` (the server default), `"per_tag"`, `"all_combinations"`, `[["t"]]`; `"per_source"` adds a scope per `source:` kind alongside the global one, so commit knowledge and conversation knowledge consolidate apart |
|
||||
| `disabled` | `false` | hard off-switch (inert plugin/hook — a no-memory baseline) |
|
||||
| `reflectTimeoutMs` | `120000` | **automatic** session-reflect timeout (hook harnesses additionally cap it at 25s to fit the host's hook window); on timeout the session runs without reflect (recorded) |
|
||||
| `reflectTimeoutMs` | `120000` | **automatic** session-reflect timeout (hook harnesses additionally cap it at 20s to fit the host's hook window); on timeout or a 5xx the hook falls back to knowledge-page search, then to a raw recall of observations (recorded) |
|
||||
| `reflectToolTimeoutMs` | `330000` | timeout for the agent-invoked `hindsight_reflect` tool — a call the agent waits on, whose high-budget synthesis on a populated bank runs for minutes. Defaults above the server's own reflect wall timeout (`HINDSIGHT_API_REFLECT_WALL_TIMEOUT`, 300s) so the server decides when to give up. Unset, it inherits an explicitly raised `reflectTimeoutMs`, but a short one never lowers it |
|
||||
| `reflectBudget` | `"high"` | reflect budget for the `hindsight_reflect` tool: `"low"`, `"mid"` or `"high"`. Drop it on a large bank where high-budget synthesis exceeds the server's wall timeout. The automatic session-start reflect always uses `"low"` to fit its hook window and is unaffected |
|
||||
| `autoReflect` | `true` | inject a one-time reflect synthesis on the session's **first prompt**. `false` = tool-only reflect: nothing is injected; the agent searches knowledge pages first and reflects only when they are too shallow |
|
||||
@@ -482,7 +482,10 @@ as a JSON line to `/tmp/hindsight-plugin.log` (override with `HINDSIGHT_DIAG_FIL
|
||||
```
|
||||
|
||||
`reflect_failed` / `pages_failed` record the error; if you're comparing memory-on vs memory-off,
|
||||
check this file — a run whose reflects failed is a no-memory run. Seed starts are logged as
|
||||
check this file — a run whose reflects failed is a no-memory run. When the failure was a timeout or
|
||||
a 5xx, the hook falls back to knowledge-page search and, if no page matches, to a raw recall of the
|
||||
bank's observations: `reflect_fallback_pages` / `reflect_fallback_observations` record what each
|
||||
step returned (`*_failed` when it errored). Seed starts are logged as
|
||||
`seed_started`.
|
||||
|
||||
### Is the memory ready yet?
|
||||
|
||||
@@ -91,7 +91,7 @@ export interface RawConfig {
|
||||
reflectTimeoutMs?: number; // session-start reflect timeout (default 120000; hooks cap lower internally)
|
||||
/** Timeout for the agent-invoked `hindsight_reflect` tool (default 330000). Deliberately its own
|
||||
* knob and much larger than `reflectTimeoutMs`: that one bounds an automatic hook that must fit
|
||||
* the host's 25s window, whereas this one bounds a call the agent made on purpose and waits on,
|
||||
* the host's 20s reflect slot, whereas this one bounds a call the agent made on purpose and waits on,
|
||||
* whose `budget: "high"` synthesis on a populated bank can run for minutes. The default sits
|
||||
* ABOVE the server's own reflect wall timeout (HINDSIGHT_API_REFLECT_WALL_TIMEOUT, 300s) so the
|
||||
* server decides when to give up, not an arbitrary client deadline (#3590). Unset, it inherits
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
DEFAULT_MAX_PARALLEL_RETAINS,
|
||||
DEFAULT_OBSERVATION_SCOPES,
|
||||
HindsightClient,
|
||||
ReflectError,
|
||||
retryAfterMs,
|
||||
} from "./hindsight";
|
||||
|
||||
@@ -548,4 +549,60 @@ describe("HindsightClient.reflect failures", () => {
|
||||
"reflect timed out after 10ms"
|
||||
);
|
||||
});
|
||||
|
||||
it("types the failure so the hook can tell a fallback-worthy one apart", async () => {
|
||||
const client = new HindsightClient({ apiUrl: "http://x", bank: "b" });
|
||||
const failWith = async (status: number) => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => jsonResponse(status, { detail: "x" }))
|
||||
);
|
||||
return client.reflect("why?", { timeoutMs: 5_000 }).catch((e: unknown) => e);
|
||||
};
|
||||
|
||||
const e503 = await failWith(503);
|
||||
expect(e503).toBeInstanceOf(ReflectError);
|
||||
expect((e503 as ReflectError).status).toBe(503);
|
||||
expect((e503 as ReflectError).fallbackEligible).toBe(true);
|
||||
// A 4xx fails the same way on every endpoint (auth, missing bank): no fallback.
|
||||
expect(((await failWith(401)) as ReflectError).fallbackEligible).toBe(false);
|
||||
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(
|
||||
(_url: string | URL | Request, init?: RequestInit) =>
|
||||
new Promise<Response>((_resolve, reject) => {
|
||||
init?.signal?.addEventListener("abort", () => reject(init.signal?.reason));
|
||||
})
|
||||
)
|
||||
);
|
||||
const timeout = (await client
|
||||
.reflect("why?", { timeoutMs: 10 })
|
||||
.catch((e) => e)) as ReflectError;
|
||||
expect(timeout.timedOut).toBe(true);
|
||||
expect(timeout.fallbackEligible).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("HindsightClient.recallObservations", () => {
|
||||
it("recalls only observations, low budget, no entities, and returns their texts in order", async () => {
|
||||
const fetchMock = vi.fn(async (_url: string | URL | Request, _init?: RequestInit) =>
|
||||
jsonResponse(200, { results: [{ text: " first " }, { text: "" }, { text: "second" }] })
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
const client = new HindsightClient({ apiUrl: "http://x", bank: "b" });
|
||||
|
||||
const out = await client.recallObservations("goal", { maxTokens: 2000, timeoutMs: 5_000 });
|
||||
|
||||
expect(out).toEqual(["first", "second"]);
|
||||
const [url, init] = fetchMock.mock.calls[0];
|
||||
expect(String(url)).toBe("http://x/v1/default/banks/b/memories/recall");
|
||||
expect(JSON.parse(String(init?.body))).toEqual({
|
||||
query: "goal",
|
||||
types: ["observation"],
|
||||
budget: "low",
|
||||
max_tokens: 2000,
|
||||
include: { entities: null },
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -169,6 +169,31 @@ export class KnowledgePagesUnavailableError extends Error {
|
||||
const TERMINAL = new Set(["completed", "failed", "cancelled", "error"]);
|
||||
|
||||
/** Default cap on concurrent retain-related requests; configurable via `maxParallelRetains`. */
|
||||
/**
|
||||
* A reflect that failed on the SERVER's side of the wire: our deadline expired or the server
|
||||
* answered non-2xx. `status` is undefined for a timeout. Transport errors (connection refused,
|
||||
* DNS) are NOT wrapped — they mean the server is unreachable, so no other endpoint would answer
|
||||
* either.
|
||||
*/
|
||||
export class ReflectError extends Error {
|
||||
constructor(
|
||||
message: string,
|
||||
readonly status: number | undefined,
|
||||
readonly timedOut: boolean,
|
||||
options?: ErrorOptions
|
||||
) {
|
||||
super(message, options);
|
||||
this.name = "ReflectError";
|
||||
}
|
||||
|
||||
/** A timeout or a 5xx means reflect's synthesis (the slow LLM path) broke, while the cheap
|
||||
* retrieval endpoints may still answer — worth falling back. A 4xx will fail the same way on
|
||||
* every endpoint (auth, missing bank), so it is not. */
|
||||
get fallbackEligible(): boolean {
|
||||
return this.timedOut || (this.status !== undefined && this.status >= 500);
|
||||
}
|
||||
}
|
||||
|
||||
export const DEFAULT_MAX_PARALLEL_RETAINS = 10;
|
||||
|
||||
/** How long drain() pauses between poll cycles when the API did not rate-limit (429). */
|
||||
@@ -279,14 +304,15 @@ export class HindsightClient {
|
||||
method: string,
|
||||
url: string,
|
||||
body?: unknown,
|
||||
tolerate: number[] = []
|
||||
tolerate: number[] = [],
|
||||
timeoutMs = 15_000
|
||||
): Promise<Response> {
|
||||
// Hard cap on EVERY request: a stalled server (pool deadlock, network) must degrade to a
|
||||
// memoryless turn — never hang a host that awaits us (opencode blocks its BOOT on plugin init).
|
||||
const r = await this.fetchWithAuth(url, {
|
||||
method,
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
signal: AbortSignal.timeout(15_000),
|
||||
signal: AbortSignal.timeout(timeoutMs),
|
||||
});
|
||||
if (r.status === 429 && !tolerate.includes(429))
|
||||
throw new RateLimitedError(retryAfterMs(r.headers.get("retry-after")));
|
||||
@@ -512,7 +538,7 @@ export class HindsightClient {
|
||||
/**
|
||||
* Reflect: synthesized, root-cause answer over the bank. Bounded so a slow server never hangs a
|
||||
* caller — but `timeoutMs` is REQUIRED, deliberately: the right deadline differs by an order of
|
||||
* magnitude between the automatic hook (25s, to fit the host's window) and the agent-invoked
|
||||
* magnitude between the automatic hook (20s, to fit the host's window) and the agent-invoked
|
||||
* tool (minutes, on a populated bank). This used to default to 120s, which silently overrode the
|
||||
* tool's configured window and aborted every high-budget synthesis mid-flight (#3590).
|
||||
*/
|
||||
@@ -527,21 +553,51 @@ export class HindsightClient {
|
||||
});
|
||||
// Keep the server's body: a bare "reflect 500" in the diag trail is undebuggable after the fact.
|
||||
if (!resp.ok)
|
||||
throw new Error(
|
||||
`reflect ${resp.status} ${(await resp.text()).slice(0, 1000)}${this.authHint(resp.status)}`
|
||||
throw new ReflectError(
|
||||
`reflect ${resp.status} ${(await resp.text()).slice(0, 1000)}${this.authHint(resp.status)}`,
|
||||
resp.status,
|
||||
false
|
||||
);
|
||||
const data = (await resp.json()) as { text?: string };
|
||||
return (data.text || "").trim();
|
||||
} catch (e) {
|
||||
// Our own deadline surfaces as a generic "This operation was aborted"; name it.
|
||||
if (ctrl.signal.aborted)
|
||||
throw new Error(`reflect timed out after ${opts.timeoutMs}ms`, { cause: e });
|
||||
throw new ReflectError(`reflect timed out after ${opts.timeoutMs}ms`, undefined, true, {
|
||||
cause: e,
|
||||
});
|
||||
throw e;
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Raw recall restricted to consolidated observations — no LLM in the loop, so it still answers
|
||||
* when reflect's synthesis times out or 5xxs. Returns the observation texts in rank order.
|
||||
*/
|
||||
async recallObservations(
|
||||
query: string,
|
||||
opts: { maxTokens: number; timeoutMs: number }
|
||||
): Promise<string[]> {
|
||||
const r = await this.req(
|
||||
"POST",
|
||||
this.bankUrl("/memories/recall"),
|
||||
{
|
||||
query,
|
||||
types: ["observation"],
|
||||
budget: "low",
|
||||
max_tokens: opts.maxTokens,
|
||||
include: { entities: null },
|
||||
},
|
||||
[],
|
||||
opts.timeoutMs
|
||||
);
|
||||
if (r.status === 404) return [];
|
||||
const j = (await r.json()) as { results?: { text?: string }[] };
|
||||
return (j.results ?? []).map((x) => (x.text ?? "").trim()).filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* The bank's knowledge-base tree (folders + pages, nested). The tree carries names, source
|
||||
* queries and staleness but NOT synthesized content, so it is cheap enough to poll.
|
||||
@@ -611,11 +667,18 @@ export class HindsightClient {
|
||||
* hindsight_search_knowledge_pages. */
|
||||
async searchKnowledgePages(
|
||||
query: string,
|
||||
limit = 3
|
||||
limit = 3,
|
||||
timeoutMs?: number
|
||||
): Promise<{ id: string; name: string; snippet: string; score: number }[]> {
|
||||
if (this.knowledgePagesSupported === false) throw new KnowledgePagesUnavailableError();
|
||||
const q = `?q=${encodeURIComponent(query)}&limit=${limit}`;
|
||||
const r = await this.req("GET", this.bankUrl(`/knowledge-base/search${q}`));
|
||||
const r = await this.req(
|
||||
"GET",
|
||||
this.bankUrl(`/knowledge-base/search${q}`),
|
||||
undefined,
|
||||
[],
|
||||
timeoutMs
|
||||
);
|
||||
const j = (await r.json()) as {
|
||||
results?: { id: string; name: string; snippet?: string; score?: number }[];
|
||||
};
|
||||
|
||||
@@ -5,6 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { resolveConfig } from "./config";
|
||||
import { buildHookOutput, runHook } from "./hook";
|
||||
import { diagFilePath } from "./diag";
|
||||
import { ReflectError } from "./hindsight";
|
||||
import { buildReflectQuery } from "./inject";
|
||||
|
||||
let root: string;
|
||||
@@ -42,12 +43,23 @@ function makeClient(
|
||||
reflect: (query: string, opts: { budget?: string; timeoutMs: number }) => Promise<string>;
|
||||
listPages: () => Promise<unknown>;
|
||||
getPage: (pageId: string) => Promise<unknown>;
|
||||
searchKnowledgePages: (
|
||||
query: string,
|
||||
limit: number,
|
||||
timeoutMs?: number
|
||||
) => Promise<{ id: string; name: string; snippet: string }[]>;
|
||||
recallObservations: (
|
||||
query: string,
|
||||
opts: { maxTokens: number; timeoutMs: number }
|
||||
) => Promise<string[]>;
|
||||
}> = {}
|
||||
) {
|
||||
return {
|
||||
reflect: vi.fn(async () => "REFLECT_ANSWER"),
|
||||
listPages: vi.fn(async () => ({ items: [{ id: "p1", name: "Uploader guide" }] })),
|
||||
getPage: vi.fn(async () => ({ content: PAGE_CONTENT })),
|
||||
searchKnowledgePages: vi.fn(async () => [] as { id: string; name: string; snippet: string }[]),
|
||||
recallObservations: vi.fn(async () => [] as string[]),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
@@ -182,7 +194,7 @@ describe("buildHookOutput", () => {
|
||||
.map((line) => JSON.parse(line))
|
||||
.find((entry) => entry.event === "reflect_failed");
|
||||
expect(failed.bank).toBe("coding-agent::demo");
|
||||
expect(failed.timeoutMs).toBe(25_000);
|
||||
expect(failed.timeoutMs).toBe(20_000);
|
||||
// Past describeError's default 200-char cut, where the root cause usually sits.
|
||||
expect(failed.error).toContain("upstream LLM rejected the turn");
|
||||
});
|
||||
@@ -215,7 +227,7 @@ describe("buildHookOutput", () => {
|
||||
expect(result.context).toBeUndefined();
|
||||
});
|
||||
|
||||
it("uses a bounded low-budget reflect and caps its timeout at 25000ms", async () => {
|
||||
it("uses a bounded low-budget reflect and caps its timeout at 20000ms", async () => {
|
||||
const cfg = resolveConfig({}); // reflectTimeoutMs default 120000
|
||||
const client = makeClient();
|
||||
await buildHookOutput({
|
||||
@@ -227,11 +239,11 @@ describe("buildHookOutput", () => {
|
||||
});
|
||||
expect(client.reflect).toHaveBeenCalledWith(buildReflectQuery("the prompt"), {
|
||||
budget: "low",
|
||||
timeoutMs: 25000,
|
||||
timeoutMs: 20000,
|
||||
});
|
||||
});
|
||||
|
||||
it("uses the configured reflect timeout when it is below the 25s cap", async () => {
|
||||
it("uses the configured reflect timeout when it is below the 20s cap", async () => {
|
||||
const cfg = resolveConfig({ reflectTimeoutMs: 5000 });
|
||||
const client = makeClient();
|
||||
await buildHookOutput({
|
||||
@@ -247,6 +259,167 @@ describe("buildHookOutput", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("reflect fallback (timeout / 5xx)", () => {
|
||||
const timedOut = () => new ReflectError("reflect timed out after 20000ms", undefined, true);
|
||||
const serverError = () => new ReflectError("reflect 502 bad gateway", 502, false);
|
||||
const HIT = { id: "kp-1", name: "Upload retries", snippet: "200ms jitter\nwindow" };
|
||||
|
||||
it("timeout -> injects matching knowledge pages, skips recall, caches it (no retry)", async () => {
|
||||
const client = makeClient({
|
||||
reflect: vi.fn(async () => {
|
||||
throw timedOut();
|
||||
}),
|
||||
searchKnowledgePages: vi.fn(async () => [HIT]),
|
||||
});
|
||||
const args = {
|
||||
harness: "claude-code",
|
||||
prompt: MATCHING_PROMPT,
|
||||
cfg: resolveConfig({}),
|
||||
client,
|
||||
cacheFile,
|
||||
};
|
||||
|
||||
const t1 = await buildHookOutput(args);
|
||||
|
||||
expect(client.searchKnowledgePages).toHaveBeenCalledWith(
|
||||
MATCHING_PROMPT,
|
||||
3,
|
||||
expect.any(Number)
|
||||
);
|
||||
expect(client.recallObservations).not.toHaveBeenCalled();
|
||||
expect(t1.context).toContain("<hindsight_memory>");
|
||||
expect(t1.context).toContain("- Upload retries (kp-1): 200ms jitter window");
|
||||
expect(t1.context).toContain("hindsight_read_knowledge_page");
|
||||
expect(t1.notice).toContain("reflect unavailable — fell back to 1 knowledge page ");
|
||||
expect(t1.notice).not.toContain("no memory this turn");
|
||||
|
||||
const t2 = await buildHookOutput(args);
|
||||
expect(client.reflect).toHaveBeenCalledTimes(1);
|
||||
expect(client.searchKnowledgePages).toHaveBeenCalledTimes(1);
|
||||
// Injected once, on the turn it ran — like a reflect answer.
|
||||
expect(t2.context ?? "").not.toContain("Upload retries");
|
||||
});
|
||||
|
||||
it("5xx with no matching page -> raw recall of observations is injected", async () => {
|
||||
const client = makeClient({
|
||||
reflect: vi.fn(async () => {
|
||||
throw serverError();
|
||||
}),
|
||||
recallObservations: vi.fn(async () => [
|
||||
"Retries back off exponentially.",
|
||||
"Tokens rotate daily.",
|
||||
]),
|
||||
});
|
||||
|
||||
const out = await buildHookOutput({
|
||||
harness: "claude-code",
|
||||
prompt: MATCHING_PROMPT,
|
||||
cfg: resolveConfig({}),
|
||||
client,
|
||||
cacheFile,
|
||||
});
|
||||
|
||||
expect(client.searchKnowledgePages).toHaveBeenCalledTimes(1);
|
||||
expect(client.recallObservations).toHaveBeenCalledWith(MATCHING_PROMPT, {
|
||||
maxTokens: 2000,
|
||||
timeoutMs: expect.any(Number),
|
||||
});
|
||||
expect(out.context).toContain("consolidated observations");
|
||||
expect(out.context).toContain("- Retries back off exponentially.\n- Tokens rotate daily.");
|
||||
expect(out.notice).toContain("fell back to 2 observations");
|
||||
});
|
||||
|
||||
it("a failing page search still falls through to observations", async () => {
|
||||
const client = makeClient({
|
||||
reflect: vi.fn(async () => {
|
||||
throw timedOut();
|
||||
}),
|
||||
searchKnowledgePages: vi.fn(async () => {
|
||||
throw new Error("knowledge pages unavailable");
|
||||
}),
|
||||
recallObservations: vi.fn(async () => ["An observation."]),
|
||||
});
|
||||
|
||||
const out = await buildHookOutput({
|
||||
harness: "claude-code",
|
||||
prompt: MATCHING_PROMPT,
|
||||
cfg: resolveConfig({}),
|
||||
client,
|
||||
cacheFile,
|
||||
});
|
||||
|
||||
expect(out.context).toContain("- An observation.");
|
||||
});
|
||||
|
||||
it("both empty -> no context, the usual no-memory notice, failure cached", async () => {
|
||||
const client = makeClient({
|
||||
reflect: vi.fn(async () => {
|
||||
throw serverError();
|
||||
}),
|
||||
});
|
||||
|
||||
const out = await buildHookOutput({
|
||||
harness: "claude-code",
|
||||
prompt: MATCHING_PROMPT,
|
||||
cfg: resolveConfig({}),
|
||||
client,
|
||||
cacheFile,
|
||||
});
|
||||
|
||||
expect(client.recallObservations).toHaveBeenCalledTimes(1);
|
||||
expect(out.context).toBeUndefined();
|
||||
expect(out.notice).toContain("no memory this turn");
|
||||
expect(JSON.parse(readFileSync(cacheFile, "utf8")).reflectAnswer).toBe("");
|
||||
});
|
||||
|
||||
it("4xx or a transport error does NOT fall back: every endpoint would fail the same way", async () => {
|
||||
for (const err of [new ReflectError("reflect 401", 401, false), new Error("fetch failed")]) {
|
||||
rmSync(cacheFile, { force: true });
|
||||
const client = makeClient({
|
||||
reflect: vi.fn(async () => {
|
||||
throw err;
|
||||
}),
|
||||
});
|
||||
const out = await buildHookOutput({
|
||||
harness: "claude-code",
|
||||
prompt: MATCHING_PROMPT,
|
||||
cfg: resolveConfig({}),
|
||||
client,
|
||||
cacheFile,
|
||||
});
|
||||
expect(client.searchKnowledgePages).not.toHaveBeenCalled();
|
||||
expect(client.recallObservations).not.toHaveBeenCalled();
|
||||
expect(out.notice).toContain("no memory this turn");
|
||||
}
|
||||
});
|
||||
|
||||
it("records each fallback step in the diag trail", async () => {
|
||||
const diagFile = join(root, "diag.log");
|
||||
vi.stubEnv("HINDSIGHT_DIAG_FILE", diagFile);
|
||||
const client = makeClient({
|
||||
reflect: vi.fn(async () => {
|
||||
throw timedOut();
|
||||
}),
|
||||
recallObservations: vi.fn(async () => ["An observation."]),
|
||||
});
|
||||
|
||||
await buildHookOutput({
|
||||
harness: "claude-code",
|
||||
prompt: MATCHING_PROMPT,
|
||||
cfg: resolveConfig({}),
|
||||
client,
|
||||
cacheFile,
|
||||
});
|
||||
|
||||
const events = readFileSync(diagFile, "utf8")
|
||||
.trim()
|
||||
.split("\n")
|
||||
.map((line) => JSON.parse(line));
|
||||
expect(events.find((e) => e.event === "reflect_fallback_pages")?.count).toBe(0);
|
||||
expect(events.find((e) => e.event === "reflect_fallback_observations")?.count).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
it("autoReflect false: never calls reflect, injects no memory block", async () => {
|
||||
const cfg = resolveConfig({ autoReflect: false });
|
||||
const client = makeClient();
|
||||
|
||||
@@ -25,9 +25,14 @@ import { diag, diagFilePath } from "./diag";
|
||||
import { describeError, log, setLogLevel } from "./log";
|
||||
import { startBackgroundSeed } from "./seed";
|
||||
import type { ClientOpts } from "./hindsight";
|
||||
import { HindsightClient } from "./hindsight";
|
||||
import { HindsightClient, ReflectError } from "./hindsight";
|
||||
import { brandWord } from "./brand";
|
||||
import { buildReflectQuery, buildSystemInjection } from "./inject";
|
||||
import {
|
||||
buildReflectQuery,
|
||||
buildSystemInjection,
|
||||
formatObservationFallback,
|
||||
formatPageFallback,
|
||||
} from "./inject";
|
||||
import type { PageRef } from "./knowledge-injection";
|
||||
import { buildRosterRefresh, parsePageList } from "./knowledge-injection";
|
||||
import {
|
||||
@@ -66,19 +71,95 @@ export interface HookSpec {
|
||||
interface HookClient {
|
||||
reflect(query: string, opts: { budget?: string; timeoutMs: number }): Promise<string>;
|
||||
listPages(): Promise<unknown>;
|
||||
searchKnowledgePages(
|
||||
query: string,
|
||||
limit: number,
|
||||
timeoutMs?: number
|
||||
): Promise<{ id: string; name: string; snippet: string }[]>;
|
||||
recallObservations(
|
||||
query: string,
|
||||
opts: { maxTokens: number; timeoutMs: number }
|
||||
): Promise<string[]>;
|
||||
knowledgePagesSupported?: boolean;
|
||||
/** Recorded on reflect failures so the diag trail says which bank to look at server-side. */
|
||||
readonly bank?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Cap on the once-per-session reflect. INVARIANT: this MUST stay below every harness's
|
||||
* UserPromptSubmit/PreInvocation hook timeout (currently 30s in the supported hook harnesses) —
|
||||
* otherwise the host kills the hook mid-reflect before the
|
||||
* result is cached, so the injection is discarded AND the reflect re-fires (uncached) every turn.
|
||||
* Raise the harness timeout in lockstep if you raise this.
|
||||
* Cap on the once-per-session reflect. INVARIANT: this plus HOOK_FALLBACK_BUDGET_MS MUST stay
|
||||
* below every harness's UserPromptSubmit/PreInvocation hook timeout (currently 30s in the
|
||||
* supported hook harnesses) — otherwise the host kills the hook before the result is cached, so
|
||||
* the injection is discarded AND the reflect re-fires (uncached) every turn.
|
||||
* Raise the harness timeout in lockstep if you raise either.
|
||||
*/
|
||||
const HOOK_REFLECT_CAP_MS = 25_000;
|
||||
const HOOK_REFLECT_CAP_MS = 20_000;
|
||||
/** Shared deadline for the whole fallback chain (page search, then observation recall) that runs
|
||||
* after a reflect timeout/5xx. Both are retrieval-only endpoints — no LLM — so seconds suffice. */
|
||||
const HOOK_FALLBACK_BUDGET_MS = 7_000;
|
||||
const FALLBACK_PAGE_LIMIT = 3;
|
||||
const FALLBACK_RECALL_MAX_TOKENS = 2_000;
|
||||
|
||||
/** What the reflect fallback produced: the memory body to inject and where it came from. */
|
||||
interface FallbackResult {
|
||||
memory: string;
|
||||
source: "knowledge_pages" | "observations";
|
||||
count: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reflect timed out or 5xx'd: the synthesis path broke, but retrieval may still answer. Try the
|
||||
* curated knowledge pages first (search), and only when none match fall back to a raw recall
|
||||
* over consolidated observations. Undefined when both came back empty or failed. Never throws.
|
||||
*/
|
||||
async function reflectFallback(
|
||||
harness: string,
|
||||
prompt: string,
|
||||
client: HookClient
|
||||
): Promise<FallbackResult | undefined> {
|
||||
const deadline = Date.now() + HOOK_FALLBACK_BUDGET_MS;
|
||||
const remaining = () => Math.max(deadline - Date.now(), 1);
|
||||
// The search query rides in a GET query string; the goal's opening carries its keywords.
|
||||
const query = prompt.slice(0, 500);
|
||||
|
||||
let t0 = Date.now();
|
||||
try {
|
||||
const hits = await client.searchKnowledgePages(query, FALLBACK_PAGE_LIMIT, remaining());
|
||||
diag(harness, "reflect_fallback_pages", { ms: Date.now() - t0, count: hits.length });
|
||||
if (hits.length) {
|
||||
return { memory: formatPageFallback(hits), source: "knowledge_pages", count: hits.length };
|
||||
}
|
||||
} catch (e) {
|
||||
diag(harness, "reflect_fallback_pages_failed", {
|
||||
ms: Date.now() - t0,
|
||||
error: describeError(e),
|
||||
});
|
||||
}
|
||||
|
||||
t0 = Date.now();
|
||||
try {
|
||||
const observations = await client.recallObservations(prompt.slice(0, 2000), {
|
||||
maxTokens: FALLBACK_RECALL_MAX_TOKENS,
|
||||
timeoutMs: remaining(),
|
||||
});
|
||||
diag(harness, "reflect_fallback_observations", {
|
||||
ms: Date.now() - t0,
|
||||
count: observations.length,
|
||||
});
|
||||
if (observations.length) {
|
||||
return {
|
||||
memory: formatObservationFallback(observations),
|
||||
source: "observations",
|
||||
count: observations.length,
|
||||
};
|
||||
}
|
||||
} catch (e) {
|
||||
diag(harness, "reflect_fallback_observations_failed", {
|
||||
ms: Date.now() - t0,
|
||||
error: describeError(e),
|
||||
});
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export interface HookOutput {
|
||||
/** The model-facing injection block, or undefined when there's nothing to inject. */
|
||||
@@ -114,6 +195,8 @@ export async function buildHookOutput(args: {
|
||||
// nothing to say on a sparse bank (diag records that as reflect_empty), and reporting it as a
|
||||
// failure would tell the user the plugin broke on exactly the sessions where it did not.
|
||||
let reflectFailed = false;
|
||||
// Set when reflect timed out / 5xx'd and a retrieval-only fallback supplied the memory instead.
|
||||
let fallback: FallbackResult | undefined;
|
||||
const deferInitialReflect = cached.deferInitialReflect === true;
|
||||
if (deferInitialReflect) {
|
||||
// A new bank has no useful history yet. Do not burn the once-per-session synthesis on prompt
|
||||
@@ -125,7 +208,7 @@ export async function buildHookOutput(args: {
|
||||
const timeoutMs = Math.min(cfg.reflectTimeoutMs, HOOK_REFLECT_CAP_MS);
|
||||
try {
|
||||
reflectAnswer = await client.reflect(buildReflectQuery(prompt), {
|
||||
// Automatic reflection runs inside a hard 25s hook window. Hindsight's low budget is the
|
||||
// Automatic reflection runs inside a hard 20s slot of the hook window. Hindsight's low budget is the
|
||||
// supported default for bounded reflect calls; callers that explicitly invoke the MCP
|
||||
// tool still get the deeper high-budget path.
|
||||
budget: "low",
|
||||
@@ -153,6 +236,11 @@ export async function buildHookOutput(args: {
|
||||
error: describeError(e, 1500),
|
||||
query: prompt.slice(0, 80),
|
||||
});
|
||||
if (e instanceof ReflectError && e.fallbackEligible) {
|
||||
fallback = await reflectFallback(harness, prompt, client);
|
||||
// The fallback body is cached exactly like a reflect answer: injected once, not retried.
|
||||
if (fallback) reflectAnswer = fallback.memory;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -203,7 +291,14 @@ export async function buildHookOutput(args: {
|
||||
// preview of what came back). Ordinary turns stay silent — page knowledge is now pulled via
|
||||
// the hindsight_search_knowledge_pages tool, which is visible as a real tool call.
|
||||
let notice: string | undefined;
|
||||
if (reflectRanThisTurn && reflectAnswer) {
|
||||
if (fallback) {
|
||||
// Say which degraded source answered: a page list or raw observations is not a synthesis,
|
||||
// and the user deserves to know the session got less than the usual memory.
|
||||
const what = fallback.source === "knowledge_pages" ? "knowledge page" : "observation";
|
||||
notice =
|
||||
`${brandWord()} · reflect unavailable — fell back to ${fallback.count} ` +
|
||||
`${what}${fallback.count === 1 ? "" : "s"} (see ${diagFilePath()})`;
|
||||
} else if (reflectRanThisTurn && reflectAnswer) {
|
||||
const q = prompt.replace(/\s+/g, " ").trim();
|
||||
const excerpt = q.length > 48 ? `${q.slice(0, 48)}…` : q;
|
||||
const preview = reflectAnswer.replace(/\s+/g, " ").trim();
|
||||
|
||||
@@ -32,6 +32,30 @@ export function buildReflectQuery(prompt: string): string {
|
||||
);
|
||||
}
|
||||
|
||||
/** The memory body injected when reflect failed and knowledge-page search answered instead. */
|
||||
export function formatPageFallback(hits: { id: string; name: string; snippet: string }[]): string {
|
||||
return (
|
||||
"(Hindsight's synthesis was unavailable this turn; these knowledge pages matched the goal by " +
|
||||
"search. Read one with hindsight_read_knowledge_page(<id>) if it looks relevant.)\n" +
|
||||
hits
|
||||
.map((h) => {
|
||||
const snippet = h.snippet.replace(/\s+/g, " ").trim();
|
||||
return `- ${h.name} (${h.id})${snippet ? `: ${snippet}` : ""}`;
|
||||
})
|
||||
.join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
/** The memory body injected when reflect failed and no knowledge page matched, so raw recall
|
||||
* over the bank's consolidated observations answered instead. */
|
||||
export function formatObservationFallback(observations: string[]): string {
|
||||
return (
|
||||
"(Hindsight's synthesis was unavailable this turn; these consolidated observations were " +
|
||||
"recalled from the bank for the goal, unsynthesized.)\n" +
|
||||
observations.map((o) => `- ${o.replace(/\s+/g, " ").trim()}`).join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
export function buildSystemInjection(memory: string): string {
|
||||
// The <hindsight_memory> wrapper is LOAD-BEARING: the transcript readers strip this exact tag
|
||||
// (transcript-util MEMORY_TAG_RE) so the session write-back never re-ingests the injected
|
||||
|
||||
@@ -209,6 +209,8 @@ describe("ZCode prompt hook", () => {
|
||||
await runHook(zcode.prompt, () => ({
|
||||
reflect: async () => "",
|
||||
listPages: async () => ({ items: [] }),
|
||||
searchKnowledgePages: async () => [],
|
||||
recallObservations: async () => [],
|
||||
knowledgePagesSupported: false,
|
||||
}));
|
||||
} finally {
|
||||
|
||||
@@ -514,7 +514,7 @@ hook by Codex...), so one shared config serves several agents side by side:
|
||||
| `manageBankConfig` | `true` | let the plugin shape the bank's own configuration — the retain strategies it writes under, the `knowledge` entity-label group, and, on a bank that has none, the missions. Writing is strictly **additive**: it adds what the bank does not define and never overwrites what is there, so your control-plane edits survive. Set `false` to keep it out of the bank config entirely — see **A bank you shape yourself** below |
|
||||
| `observationScopes` | `"shared"` | how consolidation groups observations: `"shared"` (default) = ONE global scope per bank, so every agent on a repo builds one set of beliefs; also `"combined"` (the server default), `"per_tag"`, `"all_combinations"`, `[["t"]]`; `"per_source"` adds a scope per `source:` kind alongside the global one, so commit knowledge and conversation knowledge consolidate apart |
|
||||
| `disabled` | `false` | hard off-switch (inert plugin/hook — a no-memory baseline) |
|
||||
| `reflectTimeoutMs` | `120000` | **automatic** session-reflect timeout (hook harnesses additionally cap it at 25s to fit the host's hook window); on timeout the session runs without reflect (recorded) |
|
||||
| `reflectTimeoutMs` | `120000` | **automatic** session-reflect timeout (hook harnesses additionally cap it at 20s to fit the host's hook window); on timeout or a 5xx the hook falls back to knowledge-page search, then to a raw recall of observations (recorded) |
|
||||
| `reflectToolTimeoutMs` | `330000` | timeout for the agent-invoked `hindsight_reflect` tool — a call the agent waits on, whose high-budget synthesis on a populated bank runs for minutes. Defaults above the server's own reflect wall timeout (`HINDSIGHT_API_REFLECT_WALL_TIMEOUT`, 300s) so the server decides when to give up. Unset, it inherits an explicitly raised `reflectTimeoutMs`, but a short one never lowers it |
|
||||
| `reflectBudget` | `"high"` | reflect budget for the `hindsight_reflect` tool: `"low"`, `"mid"` or `"high"`. Drop it on a large bank where high-budget synthesis exceeds the server's wall timeout. The automatic session-start reflect always uses `"low"` to fit its hook window and is unaffected |
|
||||
| `autoReflect` | `true` | inject a one-time reflect synthesis on the session's **first prompt**. `false` = tool-only reflect: nothing is injected; the agent searches knowledge pages first and reflects only when they are too shallow |
|
||||
@@ -792,7 +792,10 @@ as a JSON line to `/tmp/hindsight-plugin.log` (override with `HINDSIGHT_DIAG_FIL
|
||||
```
|
||||
|
||||
`reflect_failed` / `pages_failed` record the error; if you're comparing memory-on vs memory-off,
|
||||
check this file — a run whose reflects failed is a no-memory run. Seed starts are logged as
|
||||
check this file — a run whose reflects failed is a no-memory run. When the failure was a timeout or
|
||||
a 5xx, the hook falls back to knowledge-page search and, if no page matches, to a raw recall of the
|
||||
bank's observations: `reflect_fallback_pages` / `reflect_fallback_observations` record what each
|
||||
step returned (`*_failed` when it errored). Seed starts are logged as
|
||||
`seed_started`.
|
||||
|
||||
### Is the memory ready yet?
|
||||
|
||||
Reference in New Issue
Block a user