mirror of
https://github.com/lobehub/lobehub.git
synced 2026-09-20 04:56:13 +08:00
✨ feat(expertise): distil delivery standards from rejected acceptance checks, and restate them as they repeat (#19629)
* ✨ feat(expertise): distil delivery standards from the checks a reviewer rejected The reviewer's rejections are the only structured record of what they actually require, and nothing read them: expertise ingestion mined conversations only. This turns a settled acceptance round into expertise the same way a topic completion already does, reusing the whole lesson dedup / hit / snapshot path. Trigger: a new round attaching to the acceptance settles the previous one. The product's reject flow ends at a clipboard copy (`onCopyReview`), so no server-side event marks "I finished reviewing" — but the next round landing proves the previous round's judgement is final. Folding into a draft round is deliberately not a trigger: no new round opened, so nothing settled. - `persistDomainRun` now takes an explicit run descriptor instead of assuming a topic/agent, so both ingestion paths share it. - One observation writes one hit per rejection it cites, so `hitCount` counts violations rather than analysis passes. - The prompt is a separate chain: it must generalize ("this divider isn't needed" → "decorative dividers are not added"), cite its sources, and state when the standard does not apply. Restating one rejection verbatim yields a rule that never fires again, which is the failure this pipeline exists to avoid. - Domains resolve through the project the acceptance belongs to, falling back to the owner's always-on domains; when neither exists, a delivery-standards domain is created, owned by the user and merely mounted on the project so a sibling project can mount the same one later. Distillation runs out of band and never throws into the attach path: losing a lesson is cheaper than losing the reviewer's round. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 🐛 fix(expertise): read the circled frames, and stop losing rounds to `{}` nulls Replayed one real round (7 rejections, all circled) against the pinned model and found two things unit tests cannot see, because a unit test hands the parser a constructed object while the model hands it whatever it feels like. 1. Every `['string', 'null']` field came back as `{}` under a strict json_schema, so `RejectionAnalysisSchema.parse` threw and the entire round was lost. The nullable fields are plain strings now, with "" as the "none" the service maps back to null. A test walks the schema and fails on any resurrected union. 2. The distillation was text-only, which is close to useless here: 616 of this owner's 876 rejections circle a region on a screenshot, and "this isn't aligned" / "this cyan is too light" cannot be understood from the words. One round's frames are now inlined as data URIs, circled ones first, and each circled region is rendered with its label and normalized position so the prompt can point at the frame it belongs to. Inlined rather than linked because a link makes the provider fetch from our storage, which is unreachable for a local or private bucket. Frames are capped and the overflow is named in the prompt, so a withheld frame is never read as proof that nothing was wrong there. Also makes `limits` mandatory prose rather than an optional null. On the first replay the model left it null on all 7 observations; a standard with no stated exemption gets applied everywhere and becomes noise. Verified on the replayed round: parse clean, all 7 limits filled with real exemptions, and re-feeding the same rejections with those 7 lessons present attaches 7/7 with 0 new lessons — the dedup path holds, so repeats do not fragment the library. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ✨ feat(expertise): keep a distilled standard to what the reviewer actually said Reviewing the first replay's output against the rejections it came from showed the distillation inventing the two things that matter most. `reasoning` was manufacturing design rationale. "This order is reversed" became "follows common messaging interaction conventions and does not match user visual habits" — plausible prose the reviewer never wrote, which a later compile step would then enforce as if they had. It now carries the reviewer's own words, or says outright that they gave no reason. `limits` was inventing exemptions. "Except on mobile or narrow containers", "except in large rich-text editors" — none of it stated by anyone. An invented exemption silently narrows a standard the reviewer stated without limit, so the field is now filled only from a boundary they drew or a contradicting rejection in the same round, and otherwise says so. Generalization also gets a test it can fail rather than the vague "strip the screen names": could a delivery on a completely unrelated screen violate this? A new `subject` field forces the answer to be written down — a model that cannot say what the concrete thing is an example of has not generalized at all. The prompt also caps the climb at what was actually objected to, so one rejected divider does not become a rule about every visual element. Replayed the same production round (7 rejections, all circled) on v2: reasons are the reviewer's own words on all 7, limits honestly empty on all 7, and the abstraction climbs where it should — "the annotation border on a screenshot" became "highlight and annotation lines drawn over content". Known cost, not yet addressed: climbing a level can drop the direction out of a standard. "Place the add-reaction control after the existing reactions" became "maintain the correct order", which is no longer something a delivery can be judged against. Two of seven regressed this way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ✨ feat(expertise): explain why a standard holds, and say who supplied the reason The previous pass cut `reasoning` down to the reviewer's own words to stop the distillation inventing rationale. That over-corrected: quoting the complaint back carries no information, duplicates the evidence stored on the hit, and strips out the one thing that lets a standard reach a screen nobody has built yet — the mechanism. "A pale line disappears against a white background, so the reader cannot tell which region was circled" is what makes the rule portable; "the reviewer said the cyan is too light" is not. So the reason is explanatory again, and the real hazard is handled where it belongs: `reasonSource` records whether the reviewer stated it or the distillation supplied it. The compile step downstream turns a lesson's reason into an enforced criterion, so a reason the model invented must never read as something the reviewer said. That field only works if it is decided by an observable rather than a judgement call. Asked "did the reviewer say why", the model marked 3 of 7 as `reviewer` on rejections that were pure instructions — "put it in one row, annotations left, actions right", "there's an extra line here" — which is exactly the direction of error the field exists to prevent. Asked instead "does their text state a consequence or a cause", all 7 came back correct on replay: only "the cyan is too light, I can't see it" kept `reviewer`, because it names the consequence. Inferred reasons are marked in the lesson body so the distinction survives into the UI, and the prompt now says to answer `inferred` when in doubt: under-claiming costs nothing, over-claiming puts words in the reviewer's mouth. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ✨ feat(expertise): judge a distilled reason by whether it settles a case nobody described "The mechanism is what lets a rule transfer to a new situation" is not just guidance — it is a test the model can run on its own sentence, so the prompt now states it that way: could someone facing a case this round never described settle it using your reason alone? That test is what separates a mechanism from the objection restated in nicer words. "Harms visual tidiness", "feels inelegant", "adds cognitive load" all fail it, because tidiness and elegance are the judgement under review — they settle nothing. The prompt names those phrasings and asks for the observable underneath: what does a person fail to see, misread, mis-click, or have to do twice. Replayed on the same production round, 5 of 7 reasons moved from verdict to mechanism: "loses the value of an inline screenshot" became "the reader cannot make out the content and must click to enlarge every time"; "misleads the user into thinking it is a network fault" gained the action that follows, "and retries blindly". Two did not, and the reason is worth recording rather than tuning away: a purely aesthetic standard has no functional mechanism underneath it. Pressed for one, the model can only produce a synonym of the verdict ("visual noise" for "untidy"). Those lessons want an honest "this is the owner's taste" more than they want an invented mechanism — a distinction worth making explicit once there is enough volume to see how common they are. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ✨ feat(expertise): let a standard admit it is taste rather than fake a mechanism Not every standard has a mechanism underneath it. Some of what a reviewer requires is simply what they prefer, and pressing the distillation for a reason anyway produces a synonym of the objection — "visual noise" for "untidy", "fragments related actions" for "I want it in one row". Those read as objective and would be enforced as if they were, which is worse than an admitted preference. `reasonKind` records which one a standard is, orthogonal to `reasonSource`: one says who supplied the reason, the other says whether the reason rests on something observable. The model's own failure is the detector — if the only reason it can produce is a synonym of the complaint, that is the signal to stop and answer "taste". A taste reason still has to be actionable: "the owner does not accept dividers that were not asked for; regions are separated by spacing and container edges alone" says what to do next without pretending to be physics. It is also load bearing downstream. A taste standard has no objective test, so the compile step must not turn it into a criterion that blocks a delivery on its own — noted where the field is declared, since that step lands later. Replayed on the same production round: 4 mechanism / 3 taste, and the three are exactly the ones that had been fabricating mechanisms. The four keep the reasons they had, one of them improving ("also leaks how the backend handles errors"). Both qualifiers are rendered into the lesson body, so the distinction survives into the UI rather than living only in the analysis. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ✨ feat(expertise): restate a standard once the reviewer has rejected it several times Ingestion writes a lesson from the first rejection that produces it and, on every rejection after that, only bumps its counters. So a standard that has fired eight times is still worded for the screen it was born on, and still carries no boundary — both of which are measurable on this owner's history. Replayed against 898 circled rejections: of 44 rejections in the same category as a distilled standard but from other acceptances, the standard reproduced the reviewer's actual objection on 4, while firing on 39% of deliveries they had accepted. A rule written from one case and bounded by nothing does exactly that. It is worth fixing rather than routing around, because the repeats are real: grouping those rejections by the standard behind them puts 62% of the spacing complaints and 40% of the unrequested-decoration ones into groups of 2+, largest 12. Placement reaches only 11%, which is why refusing has to be a first-class answer rather than a failure. The pass reads every instance at once, so the wording can climb to what they share instead of what one of them said, and it reads deliveries the reviewer ACCEPTED alongside them, which is the only input that can justify a limit. A boundary inferred from the instances is invented; one read off a delivery they shipped is theirs. On the divider group this is the difference between "输入框内部 工具栏不应添加多余的分隔线" and a standard that exempts danger-action separators in menus, table headers and sibling list rows, each citing the accepted delivery it was read from — verified against those frames rather than taken on trust. Two design points came out of running it on real rows instead of fixtures: `generalized: false` is an answer, not an error. Instances that share a category do not share a standard: "put the status next to the title" and "move topics above the profile" are both placement and are not one rule. The refusal is recorded as a revision anyway, or the same instances are re-read on every subsequent rejection. Taste survives consolidation. Given six spacing rejections the model answered `taste` and left the boundary unwritten rather than manufacturing a mechanism from six data points — which is the behaviour that keeps a preference from being enforced as if it were objective. Consolidation runs after the write and never inside it: a round of rejections that is safely recorded must not roll back because the rewrite of an unrelated standard failed. No schema change — `revision.kind = 'generalize'` and `prevTitle` were designed for this and had no writer until now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ✨ feat(expertise): make every generalized boundary name the delivery it rests on The consolidation prompt asked the model never to invent an exemption, but nothing checked. Its `limits` was one free-text sentence, and the deliveries it cited ("S2") were per-request labels that meant nothing once the request was gone — so "why is this allowed" had no answer, and an invented exemption looked exactly like a real one. `limits` is now a list, each entry citing the shipped deliveries it was read from. `resolveLimits` maps those labels back to check result ids and drops any entry that resolves to nothing: no refs, a rejected instance's ref, a label that was never listed. What survives is written into the lesson and, with its ids, into `revision.evidence` alongside the instances and the shipped set the pass was offered — refusals included, so "not one standard" can be checked against the same rows. Fixes a second way to get a boundary wrong, in the other direction: a rewrite used to replace the lesson's limits wholesale, so a boundary the reviewer had stated could be silently dropped, widening the standard past what they said. A carried-over limit is now marked as such and kept only if its wording is already in the standard; when nothing survives, the existing limits section stays. Replayed on the divider group from production: two exemptions, citing S2 (the profile "⋯" menu that isolates Delete) and S4 (the documents table), resolved to their check result ids; both frames checked by eye. Spacing wrote none, and the mixed placement group still refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 🐛 fix(expertise): distil the final round, and stop two rounds racing without a queue Three fixes from review, all on paths the replays never exercised. **The last round of every acceptance was never learned from.** Distillation was scheduled only when a later round attached, on the reasoning that a new round is the first server-side proof the previous one is settled. But a terminal accept or reject settles a round just as surely, and it is the only thing that can settle the last one — no later round ever follows it. `stampDecision` now returns the round it stamped and both terminal paths hand it to distillation. Accepting still settles it: a reviewer can accept overall while individual checks were rejected along the way, and those rejections are the material. Triggering a round twice is harmless — the reflection key makes the second pass a no-op. **Without a queue, two rounds settling together raced.** The queue path buys serialization with `flowControl`; the local path had nothing, so both runs could see no bound domain and create duplicate defaults, or distil against the same stale lesson catalog and fork a lesson that should have attached — the persistence lock only serializes the writes, and would record both. Local runs now chain per reviewer, on the same key the queue uses. A failed distillation does not cancel the rounds queued behind it. **Consolidation may no longer promote taste to mechanism.** Replaying one group twice with an unchanged prompt returned `taste` once and `mechanism` once. The two are not symmetric: `mechanism` is the permissive value — it is what lets a standard compile into a criterion that blocks a delivery on its own. Ingestion decides this at the lesson's birth from the reviewer's own words, so a later pass promoting it means model noise can arm the gate with a mechanism nobody stated. Admitting taste needs no such proof, so downgrades still apply. Also corrects what `expertise_lessons.backtest` claims. Its doc described scoring a standard's precision against past decisions as the pre-compile gate; measuring that on 898 circled rejections showed the labels are too incomplete to gate on — a reviewer circles the worst thing in a delivery, so a standard correctly spotting a defect they did not circle reads as a false alarm. The column keeps its place and its emptiness, with the numbers written down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 🐛 fix(expertise): keep the reviewer's own limits, and move the revision counter with a system pass Three more review findings, all in the consolidation pass. **The next user edit of a consolidated lesson would have failed.** A system pass numbered its revision from the log's maximum but left `expertise_lessons. currentRevision` behind, while `reviseLesson` numbers from `currentRevision + 1` against a unique `(lesson, revision)`. Two system passes, or one after a user edit, and the reviewer's next rewrite collides. The counter now moves inside the same transaction as the revision — including on a refusal, which records a revision too — and it is derived from the larger of the log and the counter, so a lesson whose counter has already drifted repairs itself. **A rewrite could silently drop a boundary the reviewer drew.** `limits` was the model's complete list, so carrying an existing limit depended on it echoing that limit back: omit one of two and the exemption disappears, widening a standard the reviewer had bounded. Existing limits are now kept verbatim by the service and never re-derived from the answer; the model returns only what it is ADDING. The one exception is ingestion's "边界未由评审者说明" placeholder, which only a reader can tell from a real boundary — so the model answers that as a flag, and the placeholder is dropped only when a validated exemption arrives to replace it. **Serialization was keyed too narrowly.** Domains and their bindings are workspace-wide, so two members reviewing at once race exactly as one member's two rounds do, and a per-user key lets each mint a different default domain and split the catalog. Both the local chain and the queue's `flowControl` now key on the catalog: the workspace when there is one, the user otherwise. Replayed against production rows again, since the limits contract changed: the divider group wrote two exemptions citing the profile "⋯" menu and the documents table, both resolving to real check result ids; a variant carrying a pre-existing reviewer limit kept that limit and appended to it. One of those replays came back degenerate — the model emitted collapsed text — and the pass contained it: the invented refs resolved to nothing, so no boundary was written and the reviewer's limit survived untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * 🐛 fix(expertise): sample only the accepted rounds the caller may read **A teammate's creator-only round could end up quoted in a shared standard.** The contrast set was every accepted check in the workspace, but a workspace shares its lesson catalog, not every round in it: a round is creator-only unless it is public. This pass puts a delivery's title, the reviewer's comment and their circled notes into a prompt, then persists the result as a limit the whole workspace reads — so the sample is now restricted the way a reader would be. The caller's own checks, plus other members' only where the round they belong to is public; a check with no round is the caller's own by the scope predicate. **One rejection could count as two.** A repeated label in the model's answer ("R1", "R1") wrote two hits for the same rejection and incremented `hitCount` twice. That count ranks the standards list and decides core-versus-niche, so the distortion is durable even though consolidation later deduplicates the ids. Deduplicated at the write itself rather than at one caller, since that is the only place hits are made and the topic path shares it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Arvin Xu
@@ -0,0 +1,21 @@
|
||||
import { withOtelMetricsForUpstashWorkflows } from '@lobechat/observability-otel/modules/upstash-workflow';
|
||||
import { serve } from '@upstash/workflow/hono';
|
||||
import { Hono } from 'hono';
|
||||
|
||||
import { runExpertiseRejectionWorkflow } from '@/server/workflows/expertiseRejection';
|
||||
|
||||
import { createWorkflowQstashClient } from '../qstashClient';
|
||||
|
||||
const app = new Hono();
|
||||
|
||||
app.post(
|
||||
'/run',
|
||||
serve(
|
||||
withOtelMetricsForUpstashWorkflows(runExpertiseRejectionWorkflow, {
|
||||
url: '/api/workflows/expertise-rejection/run',
|
||||
}),
|
||||
{ qstashClient: createWorkflowQstashClient() },
|
||||
),
|
||||
);
|
||||
|
||||
export default app;
|
||||
@@ -3,6 +3,7 @@ import { Hono } from 'hono';
|
||||
import agentEvalRunApp from './agent-eval-run';
|
||||
import agentSignalApp from './agent-signal';
|
||||
import expertiseHistoryApp from './expertise-history';
|
||||
import expertiseRejectionApp from './expertise-rejection';
|
||||
import goalApp from './goal';
|
||||
import memoryUserMemoryApp from './memory-user-memory';
|
||||
import onboardingTaskRecommendationApp from './onboarding-task-recommendation';
|
||||
@@ -16,6 +17,7 @@ const app = new Hono().basePath('/api/workflows');
|
||||
app.route('/agent-eval-run', agentEvalRunApp);
|
||||
app.route('/agent-signal', agentSignalApp);
|
||||
app.route('/expertise-history', expertiseHistoryApp);
|
||||
app.route('/expertise-rejection', expertiseRejectionApp);
|
||||
app.route('/goal', goalApp);
|
||||
app.route('/memory-user-memory', memoryUserMemoryApp);
|
||||
app.route('/onboarding/understanding', onboardingUnderstandingApp);
|
||||
|
||||
@@ -0,0 +1,444 @@
|
||||
// @vitest-environment node
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { ExpertiseConsolidationService, resolveLimits } from './consolidation';
|
||||
|
||||
const { resolveExpertiseModelConfig } = vi.hoisted(() => ({
|
||||
resolveExpertiseModelConfig: vi.fn().mockResolvedValue({ model: 'm', provider: 'p' }),
|
||||
}));
|
||||
const generateObject = vi.fn();
|
||||
const listByCheckResult = vi.fn().mockResolvedValue([]);
|
||||
|
||||
vi.mock('@/server/services/aiGeneration', () => ({
|
||||
AiGenerationService: class {
|
||||
generateObject = generateObject;
|
||||
},
|
||||
}));
|
||||
vi.mock('@/database/models/verifyEvidence', () => ({
|
||||
VerifyEvidenceModel: class {
|
||||
listByCheckResult = listByCheckResult;
|
||||
},
|
||||
}));
|
||||
vi.mock('@/database/models/file', () => ({
|
||||
FileModel: class {
|
||||
findById = vi.fn().mockResolvedValue(null);
|
||||
},
|
||||
}));
|
||||
vi.mock('@/server/services/file', () => ({ FileService: class {} }));
|
||||
vi.mock('./modelConfig', () => ({ resolveExpertiseModelConfig }));
|
||||
|
||||
const LESSON = {
|
||||
code: 'P-07',
|
||||
domainId: 'domain_1',
|
||||
id: 'lesson_1',
|
||||
reasonKind: 'taste',
|
||||
sections: [
|
||||
{ body: '输入框内部工具栏不应添加多余的分隔线', key: 'rule' },
|
||||
{ body: '分割线破坏容器整体性', key: 'why' },
|
||||
{ body: '评论输入框底部多了一条横线', key: 'how' },
|
||||
],
|
||||
title: '输入框内部工具栏不应添加多余的分隔线',
|
||||
};
|
||||
|
||||
const instance = (id: string, comment: string) => ({
|
||||
detail: {
|
||||
annotations: [
|
||||
{ comment, evidenceId: `ev_${id}`, rect: { height: 0.1, width: 0.2, x: 0.1, y: 0.2 } },
|
||||
],
|
||||
},
|
||||
example: `多了一条线（${id}）`,
|
||||
id,
|
||||
title: `check ${id}`,
|
||||
});
|
||||
|
||||
/**
|
||||
* A boundary fake for the reads `consolidate` makes, in order: the lesson row, its instances, the
|
||||
* accepted deliveries, then — inside the transaction — the latest revision number. `inserts` and
|
||||
* `updates` capture what would be written.
|
||||
*/
|
||||
const createDb = (
|
||||
instances: unknown[],
|
||||
shipped: unknown[] = [],
|
||||
priorRevision = 0,
|
||||
lesson: typeof LESSON = LESSON,
|
||||
) => {
|
||||
const inserts: Record<string, unknown>[] = [];
|
||||
const updates: Record<string, unknown>[] = [];
|
||||
const wheres: unknown[] = [];
|
||||
const results: unknown[][] = [
|
||||
[lesson],
|
||||
instances,
|
||||
shipped,
|
||||
priorRevision > 0 ? [{ revision: priorRevision }] : [],
|
||||
];
|
||||
let index = 0;
|
||||
|
||||
const chain = (): Record<string, unknown> => {
|
||||
const value = results[index++] ?? [];
|
||||
const self: Record<string, unknown> = {
|
||||
as: () => self,
|
||||
from: () => self,
|
||||
groupBy: () => self,
|
||||
innerJoin: () => self,
|
||||
leftJoin: () => self,
|
||||
limit: () => self,
|
||||
orderBy: () => self,
|
||||
// eslint-disable-next-line unicorn/no-thenable
|
||||
then: (resolve: (v: unknown) => void) => resolve(value),
|
||||
where: (condition: unknown) => {
|
||||
wheres.push(condition);
|
||||
return self;
|
||||
},
|
||||
};
|
||||
return self;
|
||||
};
|
||||
|
||||
const db: Record<string, unknown> = {
|
||||
insert: () => ({
|
||||
values: async (value: Record<string, unknown>) => {
|
||||
inserts.push(value);
|
||||
},
|
||||
}),
|
||||
select: chain,
|
||||
transaction: async (fn: (tx: unknown) => Promise<unknown>) => fn(db),
|
||||
update: () => ({
|
||||
set: (value: Record<string, unknown>) => ({
|
||||
where: async () => {
|
||||
updates.push(value);
|
||||
},
|
||||
}),
|
||||
}),
|
||||
};
|
||||
|
||||
return { db: db as never, inserts, updates, wheres };
|
||||
};
|
||||
|
||||
afterEach(() => {
|
||||
vi.clearAllMocks();
|
||||
generateObject.mockReset();
|
||||
});
|
||||
|
||||
describe('ExpertiseConsolidationService.consolidate', () => {
|
||||
it('restates the standard at the level its instances share and keeps the wording it replaced', async () => {
|
||||
const { db, inserts, updates } = createDb([
|
||||
instance('c1', '这里多了没必要的一条'),
|
||||
instance('c2', '这个卡片间的分割线去掉'),
|
||||
instance('c3', '表格头下面这条线不要'),
|
||||
]);
|
||||
generateObject.mockResolvedValue({
|
||||
currentLimitsArePlaceholder: true,
|
||||
generalized: true,
|
||||
limits: [],
|
||||
note: 'I1–I3 都是未被要求的分隔线',
|
||||
reasonKind: 'taste',
|
||||
reasoning: '分隔线与留白表达同一件事，重复的边界让人多读一层结构',
|
||||
subject: '未被要求的分隔装饰',
|
||||
title: '不要添加未被要求的分隔线，区域之间只靠留白与容器边界区分',
|
||||
});
|
||||
|
||||
const result = await new ExpertiseConsolidationService(db, 'user_1').consolidate('lesson_1');
|
||||
|
||||
expect(result.generalized).toBe(true);
|
||||
expect(updates[0].title).toBe('不要添加未被要求的分隔线，区域之间只靠留白与容器边界区分');
|
||||
const sections = updates[0].sections as { body: string; key: string }[];
|
||||
expect(sections.find((section) => section.key === 'rule')?.body).toContain(
|
||||
'适用对象：未被要求的分隔装饰',
|
||||
);
|
||||
// The worked example has no better source than the one already on the lesson, so it survives.
|
||||
expect(sections.find((section) => section.key === 'how')?.body).toBe(
|
||||
'评论输入框底部多了一条横线',
|
||||
);
|
||||
expect(inserts[0]).toMatchObject({
|
||||
changedBy: 'system',
|
||||
evidence: { boundaries: [], instances: ['c1', 'c2', 'c3'], shipped: [] },
|
||||
kind: 'generalize',
|
||||
prevTitle: '输入框内部工具栏不应添加多余的分隔线',
|
||||
revision: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it('leaves the standard alone when the instances do not share one, but still records the pass', async () => {
|
||||
const { db, inserts, updates } = createDb([
|
||||
instance('c1', '状态应该放在标题边上'),
|
||||
instance('c2', '话题放到助理档案上方去'),
|
||||
instance('c3', '这两个我觉得应该并排放'),
|
||||
]);
|
||||
generateObject.mockResolvedValue({
|
||||
currentLimitsArePlaceholder: true,
|
||||
generalized: false,
|
||||
limits: [],
|
||||
note: '三条都是位置问题，但不是同一条标准',
|
||||
reasonKind: 'taste',
|
||||
reasoning: '',
|
||||
subject: '',
|
||||
title: '',
|
||||
});
|
||||
|
||||
const result = await new ExpertiseConsolidationService(db, 'user_1').consolidate('lesson_1');
|
||||
|
||||
expect(result).toMatchObject({ generalized: false, reason: 'nothing-new' });
|
||||
// Nothing is rewritten — the only write is the revision counter moving with the logged pass,
|
||||
// which a refusal still records, or the same instances get re-read every round.
|
||||
expect(updates).toEqual([{ currentRevision: 1 }]);
|
||||
// …and records what it looked at, so "not one standard" can be checked against the same rows.
|
||||
expect(inserts[0]).toMatchObject({
|
||||
evidence: { instances: ['c1', 'c2', 'c3'] },
|
||||
kind: 'generalize',
|
||||
prevTitle: LESSON.title,
|
||||
});
|
||||
});
|
||||
|
||||
it('stores which accepted delivery each boundary was read from, and drops the ones that name none', async () => {
|
||||
const { db, inserts, updates } = createDb(
|
||||
[instance('c1', 'a'), instance('c2', 'b'), instance('c3', 'c')],
|
||||
[
|
||||
{ detail: null, example: null, id: 'ok_menu', title: 'menu' },
|
||||
{ detail: null, example: null, id: 'ok_table', title: 'table' },
|
||||
],
|
||||
);
|
||||
generateObject.mockResolvedValue({
|
||||
currentLimitsArePlaceholder: true,
|
||||
generalized: true,
|
||||
limits: [
|
||||
{
|
||||
shippedRefs: ['S1'],
|
||||
text: '下拉菜单里隔离危险操作的分隔线不受此限',
|
||||
},
|
||||
// Cites nothing the reviewer shipped — an invented exemption.
|
||||
{ shippedRefs: [], text: '卡片内部的分隔线不受此限' },
|
||||
// Cites a rejected instance, which can never justify an exemption.
|
||||
{ shippedRefs: ['I2'], text: '表单里的分隔线不受此限' },
|
||||
],
|
||||
note: '',
|
||||
reasonKind: 'mechanism',
|
||||
reasoning: 'r',
|
||||
subject: 's',
|
||||
title: 't',
|
||||
});
|
||||
|
||||
await new ExpertiseConsolidationService(db, 'user_1').consolidate('lesson_1');
|
||||
|
||||
const sections = updates[0].sections as { body: string; key: string }[];
|
||||
expect(sections.find((section) => section.key === 'limits')?.body).toBe(
|
||||
'下拉菜单里隔离危险操作的分隔线不受此限',
|
||||
);
|
||||
expect(inserts[0].evidence).toEqual({
|
||||
boundaries: [
|
||||
{ checkResultIds: ['ok_menu'], limit: '下拉菜单里隔离危险操作的分隔线不受此限' },
|
||||
],
|
||||
instances: ['c1', 'c2', 'c3'],
|
||||
shipped: ['ok_menu', 'ok_table'],
|
||||
});
|
||||
});
|
||||
|
||||
it('keeps the limits the standard already had when the pass adds none', async () => {
|
||||
const stated = '表格表头下方的线不受此限';
|
||||
const { db, updates } = createDb(
|
||||
[instance('c1', 'a'), instance('c2', 'b'), instance('c3', 'c')],
|
||||
[],
|
||||
0,
|
||||
{ ...LESSON, sections: [...LESSON.sections, { body: stated, key: 'limits' }] },
|
||||
);
|
||||
generateObject.mockResolvedValue({
|
||||
currentLimitsArePlaceholder: true,
|
||||
generalized: true,
|
||||
limits: [],
|
||||
note: '',
|
||||
reasonKind: 'taste',
|
||||
reasoning: 'r',
|
||||
subject: 's',
|
||||
title: 't',
|
||||
});
|
||||
|
||||
await new ExpertiseConsolidationService(db, 'user_1').consolidate('lesson_1');
|
||||
|
||||
// Dropping a boundary the reviewer stated widens the standard past what they said.
|
||||
const sections = updates[0].sections as { body: string; key: string }[];
|
||||
expect(sections.find((section) => section.key === 'limits')?.body).toBe(stated);
|
||||
});
|
||||
|
||||
it('samples only the accepted deliveries the caller may read', async () => {
|
||||
const { db, wheres } = createDb([
|
||||
instance('c1', 'a'),
|
||||
instance('c2', 'b'),
|
||||
instance('c3', 'c'),
|
||||
]);
|
||||
generateObject.mockResolvedValue({
|
||||
currentLimitsArePlaceholder: true,
|
||||
generalized: false,
|
||||
limits: [],
|
||||
note: '',
|
||||
reasonKind: 'taste',
|
||||
reasoning: '',
|
||||
subject: '',
|
||||
title: '',
|
||||
});
|
||||
|
||||
await new ExpertiseConsolidationService(db, 'user_1', 'ws_1').consolidate('lesson_1');
|
||||
|
||||
// A workspace shares its lesson catalog but not every round in it. This pass reads a
|
||||
// delivery's title and the reviewer's notes into a prompt, then persists the result where the
|
||||
// whole workspace sees it — so a teammate's creator-only round must not be sampled.
|
||||
const columns = new Set<string>();
|
||||
const seen = new Set<unknown>();
|
||||
const walk = (node: unknown) => {
|
||||
if (!node || typeof node !== 'object' || seen.has(node)) return;
|
||||
seen.add(node);
|
||||
const record = node as Record<string, unknown>;
|
||||
if (typeof record.name === 'string' && record.table) columns.add(record.name);
|
||||
for (const value of Object.values(record)) walk(value);
|
||||
};
|
||||
walk(wheres);
|
||||
|
||||
expect([...columns]).toEqual(expect.arrayContaining(['visibility', 'user_id']));
|
||||
});
|
||||
|
||||
it('will not promote a taste standard to a mechanism', async () => {
|
||||
// LESSON is taste. Replaying the same group twice returned both answers, so an upgrade here is
|
||||
// model noise arming the compile gate with a mechanism the reviewer never gave.
|
||||
const { db, updates } = createDb([
|
||||
instance('c1', 'a'),
|
||||
instance('c2', 'b'),
|
||||
instance('c3', 'c'),
|
||||
]);
|
||||
generateObject.mockResolvedValue({
|
||||
currentLimitsArePlaceholder: true,
|
||||
generalized: true,
|
||||
limits: [],
|
||||
note: '',
|
||||
reasonKind: 'mechanism',
|
||||
reasoning: 'r',
|
||||
subject: 's',
|
||||
title: 't',
|
||||
});
|
||||
|
||||
await new ExpertiseConsolidationService(db, 'user_1').consolidate('lesson_1');
|
||||
|
||||
expect(updates[0].reasonKind).toBe('taste');
|
||||
});
|
||||
|
||||
it('lets a mechanism standard admit it was taste all along', async () => {
|
||||
const { db, updates } = createDb(
|
||||
[instance('c1', 'a'), instance('c2', 'b'), instance('c3', 'c')],
|
||||
[],
|
||||
0,
|
||||
{ ...LESSON, reasonKind: 'mechanism' },
|
||||
);
|
||||
generateObject.mockResolvedValue({
|
||||
currentLimitsArePlaceholder: true,
|
||||
generalized: true,
|
||||
limits: [],
|
||||
note: '',
|
||||
reasonKind: 'taste',
|
||||
reasoning: 'r',
|
||||
subject: 's',
|
||||
title: 't',
|
||||
});
|
||||
|
||||
await new ExpertiseConsolidationService(db, 'user_1').consolidate('lesson_1');
|
||||
|
||||
expect(updates[0].reasonKind).toBe('taste');
|
||||
});
|
||||
|
||||
it('counts a delivery once even when it backs the standard through several hits', async () => {
|
||||
const { db } = createDb([instance('c1', 'a'), instance('c1', 'a again'), instance('c2', 'b')]);
|
||||
|
||||
const result = await new ExpertiseConsolidationService(db, 'user_1').consolidate('lesson_1');
|
||||
|
||||
// Two distinct deliveries is below the threshold, so the pass must not reach the model.
|
||||
expect(result.reason).toBe('below-threshold');
|
||||
expect(generateObject).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not call the model for a standard that has fired once', async () => {
|
||||
const { db } = createDb([instance('c1', 'a')]);
|
||||
|
||||
const result = await new ExpertiseConsolidationService(db, 'user_1').consolidate('lesson_1');
|
||||
|
||||
expect(result.reason).toBe('below-threshold');
|
||||
expect(generateObject).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('ExpertiseConsolidationService.dueForConsolidation', () => {
|
||||
/** Only the aggregate row matters here, so the fake answers one query with whatever it is given. */
|
||||
const dueDb = (rows: { id: string; pending: number; total: number }[]) =>
|
||||
({
|
||||
select: () => {
|
||||
const self: Record<string, unknown> = {
|
||||
as: () => self,
|
||||
from: () => self,
|
||||
groupBy: () => self,
|
||||
leftJoin: () => self,
|
||||
// eslint-disable-next-line unicorn/no-thenable
|
||||
then: (resolve: (v: unknown) => void) => resolve(rows),
|
||||
where: () => self,
|
||||
};
|
||||
return self;
|
||||
},
|
||||
}) as never;
|
||||
|
||||
it('picks the standards that have taken on instances since they were last restated', async () => {
|
||||
const service = new ExpertiseConsolidationService(
|
||||
dueDb([
|
||||
// Enough instances and one arrived after the last pass.
|
||||
{ id: 'grown', pending: 1, total: 4 },
|
||||
// Consolidated at three and still at three — nothing new to read.
|
||||
{ id: 'settled', pending: 0, total: 3 },
|
||||
// Two rejections can agree on wording by coincidence; three is where it stops being one.
|
||||
{ id: 'young', pending: 2, total: 2 },
|
||||
]),
|
||||
'user_1',
|
||||
);
|
||||
|
||||
await expect(service.dueForConsolidation('domain_1')).resolves.toEqual(['grown']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveLimits', () => {
|
||||
it('keeps every limit the standard already carries, even ones the model never mentions', () => {
|
||||
const { texts } = resolveLimits([{ shippedRefs: ['S1'], text: '新的豁免' }], ['ok_1'], {
|
||||
isPlaceholder: false,
|
||||
text: '表格表头下方的线不受此限\n侧栏分组之间的线不受此限',
|
||||
});
|
||||
|
||||
// The reviewer drew these. A pass that had to echo them back to keep them would widen the
|
||||
// standard the first time the model forgot one.
|
||||
expect(texts).toEqual(['表格表头下方的线不受此限', '侧栏分组之间的线不受此限', '新的豁免']);
|
||||
});
|
||||
|
||||
it('drops the "no boundary stated" placeholder only once a real exemption replaces it', () => {
|
||||
const placeholder = { isPlaceholder: true, text: '边界未由评审者说明' };
|
||||
|
||||
expect(
|
||||
resolveLimits([{ shippedRefs: ['S1'], text: '真的豁免' }], ['ok_1'], placeholder).texts,
|
||||
).toEqual(['真的豁免']);
|
||||
// Nothing valid arrived, so the standard keeps the section it had.
|
||||
expect(
|
||||
resolveLimits([{ shippedRefs: [], text: '编的豁免' }], ['ok_1'], placeholder).texts,
|
||||
).toEqual(['边界未由评审者说明']);
|
||||
});
|
||||
|
||||
it('ignores labels that were never listed and does not repeat an id', () => {
|
||||
const { boundaries } = resolveLimits(
|
||||
[{ shippedRefs: ['S1', 'S1', 'S9'], text: 'x' }],
|
||||
['ok_1'],
|
||||
{ isPlaceholder: false, text: '' },
|
||||
);
|
||||
|
||||
expect(boundaries).toEqual([{ checkResultIds: ['ok_1'], limit: 'x' }]);
|
||||
});
|
||||
|
||||
it('does not add a boundary the standard already states', () => {
|
||||
const { texts } = resolveLimits(
|
||||
[{ shippedRefs: ['S1'], text: '表格表头下方的线 不受此限' }],
|
||||
['ok_1'],
|
||||
{
|
||||
isPlaceholder: false,
|
||||
text: '表格表头下方的线不受此限',
|
||||
},
|
||||
);
|
||||
|
||||
expect(texts).toEqual(['表格表头下方的线不受此限']);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,528 @@
|
||||
import { TRACING_SCENARIOS } from '@lobechat/const';
|
||||
import {
|
||||
expertiseHits,
|
||||
expertiseLessonRevisions,
|
||||
expertiseLessons,
|
||||
verifyCheckResults,
|
||||
verifyRuns,
|
||||
} from '@lobechat/database/schemas';
|
||||
import {
|
||||
chainExpertiseConsolidation,
|
||||
EXPERTISE_CONSOLIDATION_JSON_SCHEMA,
|
||||
EXPERTISE_CONSOLIDATION_PROMPT_VERSION,
|
||||
} from '@lobechat/prompts';
|
||||
import type {
|
||||
ExpertiseLessonSection,
|
||||
ExpertiseRevisionEvidence,
|
||||
VerifyCheckDecisionDetail,
|
||||
} from '@lobechat/types';
|
||||
import debug from 'debug';
|
||||
import { and, desc, eq, gt, inArray, isNull, notInArray, or, sql } from 'drizzle-orm';
|
||||
import pMap from 'p-map';
|
||||
import { z } from 'zod';
|
||||
|
||||
import { FileModel } from '@/database/models/file';
|
||||
import { VerifyEvidenceModel } from '@/database/models/verifyEvidence';
|
||||
import type { LobeChatDatabase } from '@/database/type';
|
||||
import { AiGenerationService } from '@/server/services/aiGeneration';
|
||||
import { FileService } from '@/server/services/file';
|
||||
import { resolveModelReadableFrameUrl } from '@/server/services/verify/modelFrames';
|
||||
|
||||
import { resolveExpertiseModelConfig } from './modelConfig';
|
||||
|
||||
const log = debug('lobe-server:expertise-consolidation');
|
||||
|
||||
/**
|
||||
* Instances a standard needs before it is worth restating.
|
||||
*
|
||||
* Two is a coincidence — the second rejection can attach to the first on wording alone. Measured
|
||||
* on this owner's 898 circled rejections, grouping by the standard behind them puts 40% of the
|
||||
* "unrequested decoration" complaints and 62% of the spacing ones into groups of 2+ (largest: 12),
|
||||
* while placement complaints reach only 11%. So the material exists, and most of it arrives in
|
||||
* threes and up.
|
||||
*/
|
||||
const MIN_INSTANCES = 3;
|
||||
|
||||
/** Instances read per pass. Beyond this the frames, not the reasoning, are what caps the request. */
|
||||
const MAX_INSTANCES = 8;
|
||||
|
||||
/**
|
||||
* Accepted deliveries offered as the contrast set.
|
||||
*
|
||||
* This is the only input that can justify a `limits`, so it is not optional padding: without it
|
||||
* the pass can restate a standard but can never bound one. Measured on 23 accepted deliveries,
|
||||
* standards distilled this way fired on 39% of them — every one of those is a boundary the
|
||||
* reviewer drew and nobody wrote down.
|
||||
*/
|
||||
const MAX_SHIPPED = 6;
|
||||
|
||||
/** Total frames inlined. Each is a full base64 body, so this is a payload budget. */
|
||||
const MAX_FRAMES = 10;
|
||||
|
||||
const pct = (value: number) => `${Math.round(value * 100)}%`;
|
||||
|
||||
const ConsolidationSchema = z.object({
|
||||
currentLimitsArePlaceholder: z.boolean(),
|
||||
generalized: z.boolean(),
|
||||
limits: z.array(z.object({ shippedRefs: z.array(z.string()), text: z.string() })),
|
||||
note: z.string(),
|
||||
reasonKind: z.enum(['mechanism', 'taste']),
|
||||
reasoning: z.string(),
|
||||
subject: z.string(),
|
||||
title: z.string(),
|
||||
});
|
||||
|
||||
/** Whitespace-insensitive, so the same limit written twice is not kept twice. */
|
||||
const squash = (text: string) => text.replaceAll(/\s+/g, '');
|
||||
|
||||
/**
|
||||
* The limits a consolidated standard ends up with, and what each new one rests on.
|
||||
*
|
||||
* Existing limits are kept verbatim and are never re-derived from the model's answer: they were
|
||||
* drawn by the reviewer, and a pass that forgot to echo one back would silently widen a standard
|
||||
* they had bounded. The single exception is ingestion's "no boundary stated" placeholder, which is
|
||||
* dropped only when a real exemption arrives to replace it — and only the model can tell the two
|
||||
* apart, since the placeholder is written in whatever language the reviewer used.
|
||||
*
|
||||
* A new exemption survives only if it names a shipped delivery that was actually offered. No refs,
|
||||
* an instance's ref, a label that was never listed — all dropped. This is where "never invent a
|
||||
* boundary" stops being a request to the model.
|
||||
*/
|
||||
export const resolveLimits = (
|
||||
limits: z.infer<typeof ConsolidationSchema>['limits'],
|
||||
shippedIds: string[],
|
||||
current: { isPlaceholder: boolean; text: string },
|
||||
) => {
|
||||
const byRef = new Map(shippedIds.map((id, index) => [`S${index + 1}`, id]));
|
||||
const boundaries: ExpertiseRevisionEvidence['boundaries'] = [];
|
||||
|
||||
for (const limit of limits) {
|
||||
const text = limit.text.trim();
|
||||
if (!text) continue;
|
||||
const checkResultIds = [
|
||||
...new Set(
|
||||
limit.shippedRefs
|
||||
.map((ref) => byRef.get(ref.trim()))
|
||||
.filter((id): id is string => Boolean(id)),
|
||||
),
|
||||
];
|
||||
if (checkResultIds.length === 0) continue;
|
||||
boundaries.push({ checkResultIds, limit: text });
|
||||
}
|
||||
|
||||
const existing = current.text
|
||||
.split('\n')
|
||||
.map((line) => line.trim())
|
||||
.filter(Boolean);
|
||||
// The placeholder only goes away once something real replaces it; with no additions the standard
|
||||
// keeps exactly the limits section it had.
|
||||
const kept = current.isPlaceholder && boundaries.length > 0 ? [] : existing;
|
||||
const seen = new Set(kept.map((line) => squash(line)));
|
||||
const added = boundaries
|
||||
.map((boundary) => boundary.limit)
|
||||
.filter((text) => !seen.has(squash(text)) && seen.add(squash(text)));
|
||||
|
||||
return { boundaries, texts: [...kept, ...added] };
|
||||
};
|
||||
|
||||
export interface ConsolidationResult {
|
||||
generalized: boolean;
|
||||
lessonId: string;
|
||||
note: string;
|
||||
reason?: 'below-threshold' | 'no-instances' | 'no-lesson' | 'nothing-new';
|
||||
title?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Restates a standard at the level its instances share, once it has several.
|
||||
*
|
||||
* Ingestion writes a lesson from the first rejection that produced it and, on every rejection
|
||||
* after that, only bumps its counters — so a standard that has fired eight times is still worded
|
||||
* for the screen it was born on, and still carries no boundary. Both gaps are measurable: of 44
|
||||
* rejections in the same category as a distilled standard but from other acceptances, the standard
|
||||
* reproduced the reviewer's actual objection on 4, while firing on 39% of deliveries they had
|
||||
* accepted. A rule written from one case and bounded by nothing does both of those.
|
||||
*
|
||||
* This pass is the missing half. It reads every instance at once (so the wording can climb to what
|
||||
* they share instead of what one of them said) together with deliveries the reviewer accepted (so
|
||||
* a limit can be read off a case they let through, rather than invented).
|
||||
*/
|
||||
export class ExpertiseConsolidationService {
|
||||
constructor(
|
||||
private readonly db: LobeChatDatabase,
|
||||
private readonly userId: string,
|
||||
private readonly workspaceId?: string,
|
||||
) {}
|
||||
|
||||
/**
|
||||
* Standards in this domain that have taken on instances since they were last restated.
|
||||
*
|
||||
* The "since" is read off the revision log rather than a counter column: a lesson consolidated
|
||||
* at three instances and sitting at three has nothing to learn, while the same lesson at four
|
||||
* does. Comparing timestamps keeps that true without a column that can drift out of step.
|
||||
*/
|
||||
dueForConsolidation = async (domainId: string, lessonIds?: string[]): Promise<string[]> => {
|
||||
const lastGeneralized = this.db
|
||||
.select({
|
||||
at: sql<Date>`max(${expertiseLessonRevisions.createdAt})`.as('at'),
|
||||
lessonId: expertiseLessonRevisions.lessonId,
|
||||
})
|
||||
.from(expertiseLessonRevisions)
|
||||
.where(eq(expertiseLessonRevisions.kind, 'generalize'))
|
||||
.groupBy(expertiseLessonRevisions.lessonId)
|
||||
.as('last_generalized');
|
||||
|
||||
const rows = await this.db
|
||||
.select({
|
||||
id: expertiseLessons.id,
|
||||
pending: sql<number>`count(${expertiseHits.id}) filter (
|
||||
where ${lastGeneralized.at} is null or ${expertiseHits.createdAt} > ${lastGeneralized.at}
|
||||
)::int`,
|
||||
total: sql<number>`count(${expertiseHits.id})::int`,
|
||||
})
|
||||
.from(expertiseLessons)
|
||||
.leftJoin(expertiseHits, eq(expertiseHits.lessonId, expertiseLessons.id))
|
||||
.leftJoin(lastGeneralized, eq(lastGeneralized.lessonId, expertiseLessons.id))
|
||||
.where(
|
||||
and(
|
||||
eq(expertiseLessons.domainId, domainId),
|
||||
eq(expertiseLessons.status, 'active'),
|
||||
lessonIds?.length ? inArray(expertiseLessons.id, lessonIds) : undefined,
|
||||
),
|
||||
)
|
||||
.groupBy(expertiseLessons.id);
|
||||
|
||||
return rows.filter((row) => row.total >= MIN_INSTANCES && row.pending > 0).map((row) => row.id);
|
||||
};
|
||||
|
||||
consolidate = async (lessonId: string): Promise<ConsolidationResult> => {
|
||||
const [lesson] = await this.db
|
||||
.select({
|
||||
code: expertiseLessons.code,
|
||||
currentRevision: expertiseLessons.currentRevision,
|
||||
domainId: expertiseLessons.domainId,
|
||||
id: expertiseLessons.id,
|
||||
reasonKind: expertiseLessons.reasonKind,
|
||||
sections: expertiseLessons.sections,
|
||||
title: expertiseLessons.title,
|
||||
})
|
||||
.from(expertiseLessons)
|
||||
.where(eq(expertiseLessons.id, lessonId))
|
||||
.limit(1);
|
||||
if (!lesson) return { generalized: false, lessonId, note: '', reason: 'no-lesson' };
|
||||
|
||||
const instances = await this.loadInstances(lessonId);
|
||||
if (instances.length === 0)
|
||||
return { generalized: false, lessonId, note: '', reason: 'no-instances' };
|
||||
if (instances.length < MIN_INSTANCES)
|
||||
return { generalized: false, lessonId, note: '', reason: 'below-threshold' };
|
||||
|
||||
const shipped = await this.loadShipped(instances.map((instance) => instance.id));
|
||||
const { visuals, withheld } = await this.resolveFrames(instances, shipped);
|
||||
const frameLabel = new Map(visuals.map((visual, index) => [visual.key, `frame ${index + 1}`]));
|
||||
|
||||
const section = (key: string) => lesson.sections.find((entry) => entry.key === key)?.body ?? '';
|
||||
const rendered = [
|
||||
`code: ${lesson.code}`,
|
||||
`title: ${lesson.title}`,
|
||||
section('why') && `reasoning: ${section('why')}`,
|
||||
section('limits') && `limits: ${section('limits')}`,
|
||||
`reasonKind: ${lesson.reasonKind ?? 'unknown'}`,
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join('\n');
|
||||
|
||||
const ai = new AiGenerationService(this.db, this.userId, this.workspaceId);
|
||||
const modelConfig = await resolveExpertiseModelConfig(this.db, this.userId);
|
||||
const raw = await ai.generateObject(
|
||||
{
|
||||
...chainExpertiseConsolidation({
|
||||
instances: instances
|
||||
.map((instance, index) => this.renderDelivery(`I${index + 1}`, instance, frameLabel))
|
||||
.join('\n\n'),
|
||||
lesson: rendered,
|
||||
shipped: shipped
|
||||
.map((delivery, index) => this.renderDelivery(`S${index + 1}`, delivery, frameLabel))
|
||||
.join('\n\n'),
|
||||
visuals: visuals.map(({ accessUrl, label }) => ({ accessUrl, label })),
|
||||
withheldEvidence: withheld,
|
||||
}),
|
||||
...modelConfig,
|
||||
schema: EXPERTISE_CONSOLIDATION_JSON_SCHEMA,
|
||||
},
|
||||
{
|
||||
metadata: { trigger: 'expertise_consolidation' },
|
||||
tracing: {
|
||||
promptVersion: EXPERTISE_CONSOLIDATION_PROMPT_VERSION,
|
||||
scenario: TRACING_SCENARIOS.ExpertiseConsolidation,
|
||||
schemaName: EXPERTISE_CONSOLIDATION_JSON_SCHEMA.name,
|
||||
},
|
||||
},
|
||||
);
|
||||
const result = ConsolidationSchema.parse(raw);
|
||||
// Only ever downgrade. Replaying one group twice with an unchanged prompt returned `taste`
|
||||
// once and `mechanism` once, and the two are not symmetric: `mechanism` is the permissive
|
||||
// value — it is what lets a standard compile into a criterion that blocks a delivery on its
|
||||
// own. Ingestion decided this at the lesson's birth from the reviewer's own words, so letting
|
||||
// a later pass promote it means a mechanism nobody stated can arm the gate through model
|
||||
// noise. Admitting taste needs no such proof.
|
||||
const reasonKind =
|
||||
lesson.reasonKind === 'taste' && result.reasonKind === 'mechanism'
|
||||
? 'taste'
|
||||
: result.reasonKind;
|
||||
const shippedIds = shipped.map((delivery) => delivery.id);
|
||||
const { boundaries, texts } = resolveLimits(result.limits, shippedIds, {
|
||||
isPlaceholder: result.currentLimitsArePlaceholder,
|
||||
text: section('limits'),
|
||||
});
|
||||
const evidence: ExpertiseRevisionEvidence = {
|
||||
boundaries,
|
||||
instances: instances.map((instance) => instance.id),
|
||||
shipped: shippedIds,
|
||||
};
|
||||
|
||||
// A refusal is still a pass: recording it is what stops the same instances being re-read on
|
||||
// every subsequent rejection, and "these do not share a standard" is the honest answer for a
|
||||
// category like placement, where only 11% of complaints repeat.
|
||||
if (!result.generalized || !result.title.trim()) {
|
||||
await this.db.transaction(async (tx) =>
|
||||
this.recordRevision(lesson, lesson.sections, result.note, {
|
||||
evidence,
|
||||
generalized: false,
|
||||
tx,
|
||||
}),
|
||||
);
|
||||
return { generalized: false, lessonId, note: result.note, reason: 'nothing-new' };
|
||||
}
|
||||
|
||||
const sections: ExpertiseLessonSection[] = [
|
||||
{
|
||||
body: result.subject.trim()
|
||||
? `${result.title.trim()}\n\n适用对象：${result.subject.trim()}`
|
||||
: result.title.trim(),
|
||||
key: 'rule',
|
||||
},
|
||||
{ body: result.reasoning, key: 'why' },
|
||||
// The worked example is the one section this pass has no better source for than the lesson
|
||||
// already has, so it is carried over rather than regenerated.
|
||||
...lesson.sections.filter((entry) => entry.key === 'how'),
|
||||
// Nothing survived resolution → the standard keeps the limits section it already had (the
|
||||
// reviewer's own boundary, or ingestion's "not stated" placeholder), rather than losing it.
|
||||
...(texts.length > 0
|
||||
? [{ body: texts.join('\n'), key: 'limits' as const }]
|
||||
: lesson.sections.filter((entry) => entry.key === 'limits')),
|
||||
];
|
||||
|
||||
await this.db.transaction(async (tx) => {
|
||||
await tx
|
||||
.update(expertiseLessons)
|
||||
.set({ reasonKind, sections, title: result.title.trim() })
|
||||
.where(eq(expertiseLessons.id, lessonId));
|
||||
await this.recordRevision(lesson, sections, result.note, { evidence, generalized: true, tx });
|
||||
});
|
||||
|
||||
log('consolidated %s: %s -> %s', lesson.code, lesson.title, result.title);
|
||||
return { generalized: true, lessonId, note: result.note, title: result.title.trim() };
|
||||
};
|
||||
|
||||
/** The rejected deliveries behind this standard, newest first so a growing rule reads current. */
|
||||
private loadInstances = async (lessonId: string) => {
|
||||
const rows = await this.db
|
||||
.select({
|
||||
detail: verifyCheckResults.userDecisionDetail,
|
||||
example: expertiseHits.example,
|
||||
id: verifyCheckResults.id,
|
||||
title: verifyCheckResults.checkItemTitle,
|
||||
})
|
||||
.from(expertiseHits)
|
||||
.innerJoin(verifyCheckResults, eq(verifyCheckResults.id, expertiseHits.sourceCheckResultId))
|
||||
.where(eq(expertiseHits.lessonId, lessonId))
|
||||
.orderBy(desc(expertiseHits.createdAt))
|
||||
.limit(MAX_INSTANCES);
|
||||
|
||||
// One check can back a standard through several hits; reading it twice would let a single
|
||||
// delivery look like agreement between two.
|
||||
const seen = new Set<string>();
|
||||
return rows.filter((row) => !seen.has(row.id) && seen.add(row.id));
|
||||
};
|
||||
|
||||
/**
|
||||
* Deliveries accepted in the same scope, excluding the instances themselves.
|
||||
*
|
||||
* Readable rounds only. A workspace shares its lesson catalog but not every round in it: a round
|
||||
* is creator-only unless it is public, and this pass puts a delivery's title, the reviewer's
|
||||
* comments and their circled notes into a prompt — then persists the result as a limit the whole
|
||||
* workspace reads. So the sample is restricted the way a reader would be: the caller's own
|
||||
* checks, plus other members' only where the round they belong to is public. A check with no
|
||||
* round is the caller's own by the scope predicate.
|
||||
*
|
||||
* Deterministic per owner rather than newest-first: a boundary that appears only because the
|
||||
* sample moved is not a boundary, and a re-run has to be comparable with the last one.
|
||||
*/
|
||||
private loadShipped = async (excludeIds: string[]) => {
|
||||
const scope = this.workspaceId
|
||||
? eq(verifyCheckResults.workspaceId, this.workspaceId)
|
||||
: and(eq(verifyCheckResults.userId, this.userId), isNull(verifyCheckResults.workspaceId));
|
||||
|
||||
return this.db
|
||||
.select({
|
||||
detail: verifyCheckResults.userDecisionDetail,
|
||||
example: sql<null>`null`,
|
||||
id: verifyCheckResults.id,
|
||||
title: verifyCheckResults.checkItemTitle,
|
||||
})
|
||||
.from(verifyCheckResults)
|
||||
.leftJoin(verifyRuns, eq(verifyRuns.id, verifyCheckResults.verifyRunId))
|
||||
.where(
|
||||
and(
|
||||
scope,
|
||||
or(eq(verifyCheckResults.userId, this.userId), eq(verifyRuns.visibility, 'public')),
|
||||
eq(verifyCheckResults.userDecision, 'accepted'),
|
||||
excludeIds.length > 0 ? notInArray(verifyCheckResults.id, excludeIds) : undefined,
|
||||
gt(sql`length(coalesce(${verifyCheckResults.checkItemTitle}, ''))`, 0),
|
||||
),
|
||||
)
|
||||
.orderBy(sql`md5(${verifyCheckResults.id}::text)`)
|
||||
.limit(MAX_SHIPPED);
|
||||
};
|
||||
|
||||
private renderDelivery = (
|
||||
ref: string,
|
||||
delivery: {
|
||||
detail: VerifyCheckDecisionDetail | null;
|
||||
example?: null | string;
|
||||
id: string;
|
||||
title: null | string;
|
||||
},
|
||||
frameLabel: Map<string, string>,
|
||||
) => {
|
||||
const regions = (delivery.detail?.annotations ?? []).map((annotation) => {
|
||||
const frame = frameLabel.get(`${delivery.id}:${annotation.evidenceId}`);
|
||||
const at = annotation.rect
|
||||
? ` at ${pct(annotation.rect.x)},${pct(annotation.rect.y)} sized ${pct(annotation.rect.width)}×${pct(annotation.rect.height)}`
|
||||
: '';
|
||||
return ` circled${frame ? ` on ${frame}` : ''}${at}: ${annotation.comment?.trim() || '(no note)'}`;
|
||||
});
|
||||
const frames = [...frameLabel]
|
||||
.filter(([key]) => key.startsWith(`${delivery.id}:`))
|
||||
.map(([, label]) => label);
|
||||
|
||||
return [
|
||||
`[${ref}] promised: ${delivery.title ?? '(untitled check)'}`,
|
||||
delivery.detail?.comment?.trim() && ` said: ${delivery.detail.comment.trim()}`,
|
||||
...regions,
|
||||
regions.length === 0 && frames.length > 0 && ` shown on ${frames.join(', ')}`,
|
||||
delivery.example?.trim() && ` recorded as: ${delivery.example.trim()}`,
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join('\n');
|
||||
};
|
||||
|
||||
/**
|
||||
* Frames for both sets, instances first.
|
||||
*
|
||||
* The order is the budget: dropping a shipped frame costs a boundary this pass might have
|
||||
* found, while dropping an instance frame costs the restatement itself.
|
||||
*/
|
||||
private resolveFrames = async (
|
||||
instances: { detail: VerifyCheckDecisionDetail | null; id: string }[],
|
||||
shipped: { detail: VerifyCheckDecisionDetail | null; id: string }[],
|
||||
) => {
|
||||
const fileModel = new FileModel(this.db, this.userId, this.workspaceId);
|
||||
const fileService = new FileService(this.db, this.userId, this.workspaceId);
|
||||
const evidenceModel = new VerifyEvidenceModel(this.db, this.userId, this.workspaceId);
|
||||
|
||||
const collect = async (
|
||||
deliveries: { detail: VerifyCheckDecisionDetail | null; id: string }[],
|
||||
kind: 'instance' | 'shipped',
|
||||
) => {
|
||||
const out: { fileId: string; key: string; label: string }[] = [];
|
||||
for (const [index, delivery] of deliveries.entries()) {
|
||||
const circled = new Set(
|
||||
(delivery.detail?.annotations ?? []).map((annotation) => annotation.evidenceId),
|
||||
);
|
||||
const rows = await evidenceModel.listByCheckResult(delivery.id);
|
||||
// One frame per delivery on each side: this pass compares many deliveries against one
|
||||
// standard, so breadth buys more than a second angle on any single delivery.
|
||||
const chosen =
|
||||
rows.find((row) => row.type === 'screenshot' && row.fileId && circled.has(row.id)) ??
|
||||
rows.find((row) => row.type === 'screenshot' && row.fileId);
|
||||
if (!chosen?.fileId) continue;
|
||||
out.push({
|
||||
fileId: chosen.fileId,
|
||||
key: `${delivery.id}:${chosen.id}`,
|
||||
label: `${kind === 'instance' ? 'I' : 'S'}${index + 1} — ${kind === 'instance' ? 'rejected' : 'accepted'}`,
|
||||
});
|
||||
}
|
||||
return out;
|
||||
};
|
||||
|
||||
const ordered = [
|
||||
...(await collect(instances, 'instance')),
|
||||
...(await collect(shipped, 'shipped')),
|
||||
];
|
||||
const selected = ordered.slice(0, MAX_FRAMES);
|
||||
const resolved = await pMap(
|
||||
selected,
|
||||
async (item) => {
|
||||
const file = await fileModel.findById(item.fileId);
|
||||
if (!file) return null;
|
||||
return {
|
||||
accessUrl: await resolveModelReadableFrameUrl(fileService, file),
|
||||
key: item.key,
|
||||
label: item.label,
|
||||
};
|
||||
},
|
||||
{ concurrency: 4 },
|
||||
);
|
||||
|
||||
const visuals = resolved.filter((item): item is NonNullable<typeof item> => Boolean(item));
|
||||
const dropped = ordered.length - selected.length;
|
||||
return {
|
||||
visuals,
|
||||
withheld:
|
||||
dropped > 0
|
||||
? `${dropped} further screenshot(s) were not attached. A boundary cannot be read off a frame you were not given, so treat their absence as missing information rather than as proof the subject is not there.`
|
||||
: undefined,
|
||||
};
|
||||
};
|
||||
|
||||
private recordRevision = async (
|
||||
lesson: { currentRevision?: null | number; id: string; title: string },
|
||||
sections: ExpertiseLessonSection[],
|
||||
note: string,
|
||||
options: { evidence: ExpertiseRevisionEvidence; generalized: boolean; tx: LobeChatDatabase },
|
||||
) => {
|
||||
const db = options.tx;
|
||||
const [prior] = await db
|
||||
.select({ revision: expertiseLessonRevisions.revision })
|
||||
.from(expertiseLessonRevisions)
|
||||
.where(eq(expertiseLessonRevisions.lessonId, lesson.id))
|
||||
.orderBy(desc(expertiseLessonRevisions.revision))
|
||||
.limit(1);
|
||||
|
||||
// Both the log's own maximum and the lesson's counter, because a user edit numbers from
|
||||
// `currentRevision + 1` against a unique `(lesson, revision)`: leaving the counter behind makes
|
||||
// the reviewer's NEXT edit collide, and taking the larger of the two also repairs a lesson
|
||||
// whose counter has already drifted.
|
||||
const revision = Math.max(prior?.revision ?? 0, lesson.currentRevision ?? 0) + 1;
|
||||
|
||||
await db.insert(expertiseLessonRevisions).values({
|
||||
changedBy: 'system',
|
||||
evidence: options.evidence,
|
||||
feedback: note,
|
||||
kind: 'generalize',
|
||||
lessonId: lesson.id,
|
||||
// The title before the rewrite is what makes a pass auditable at a glance; a refusal keeps
|
||||
// the same title on both sides, which is exactly how a refusal should read.
|
||||
prevTitle: lesson.title,
|
||||
revision,
|
||||
sections,
|
||||
});
|
||||
await db
|
||||
.update(expertiseLessons)
|
||||
.set({ currentRevision: revision })
|
||||
.where(eq(expertiseLessons.id, lesson.id));
|
||||
log('recorded %s revision for %s', options.generalized ? 'generalize' : 'no-change', lesson.id);
|
||||
};
|
||||
}
|
||||
@@ -128,8 +128,8 @@ describe('ExpertiseDomainService', () => {
|
||||
|
||||
expect(createDomain).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
agentId: 'agent_1',
|
||||
canonEntries: draft.canonEntries,
|
||||
carrier: { id: 'agent_1', type: 'agent' },
|
||||
layerCanonRef: 'Google SRE incident lifecycle',
|
||||
layerSource: 'canonical',
|
||||
layers: [
|
||||
|
||||
@@ -112,9 +112,9 @@ export class ExpertiseDomainService {
|
||||
/** Persists a reviewed draft as the chosen anchor. The user has seen and possibly edited every field by now. */
|
||||
create = async (input: DomainDraft & { agentId: string; brief: string }) =>
|
||||
new ExpertiseModel(this.db, this.userId, this.workspaceId).createDomain({
|
||||
agentId: input.agentId,
|
||||
brief: input.brief,
|
||||
canonEntries: input.canonEntries,
|
||||
carrier: { id: input.agentId, type: 'agent' },
|
||||
domainFilter: input.domainFilter,
|
||||
layerCanonRef: input.layerCanonRef ?? undefined,
|
||||
layers: input.layers.map((l) => ({
|
||||
|
||||
@@ -10,6 +10,8 @@ const { resolveExpertiseModelConfig } = vi.hoisted(() => ({
|
||||
resolveExpertiseModelConfig: vi.fn(),
|
||||
}));
|
||||
const generateObject = vi.fn();
|
||||
const dueForConsolidation = vi.fn();
|
||||
const consolidate = vi.fn();
|
||||
const listDomainsForAgent = vi.fn();
|
||||
const listLessons = vi.fn();
|
||||
|
||||
@@ -25,6 +27,12 @@ vi.mock('@/server/services/aiGeneration', () => ({
|
||||
},
|
||||
}));
|
||||
vi.mock('./modelConfig', () => ({ resolveExpertiseModelConfig }));
|
||||
vi.mock('./consolidation', () => ({
|
||||
ExpertiseConsolidationService: class {
|
||||
consolidate = consolidate;
|
||||
dueForConsolidation = dueForConsolidation;
|
||||
},
|
||||
}));
|
||||
|
||||
const completion = (selfIteration: SelfIterationCompletionPayload) => ({
|
||||
agentId: 'agent-signal-reflection',
|
||||
@@ -247,8 +255,10 @@ const createTx = (persistedLessons: Record<string, unknown>[]): TxFake => {
|
||||
inserted,
|
||||
tx: {
|
||||
insert: (table: unknown) => ({
|
||||
values: async (value: Record<string, unknown>) => {
|
||||
inserted.set(table, [...(inserted.get(table) ?? []), value]);
|
||||
// Drizzle accepts one row or many; hit writes fan out, so flatten both into one list.
|
||||
values: async (value: Record<string, unknown> | Record<string, unknown>[]) => {
|
||||
const rows = Array.isArray(value) ? value : [value];
|
||||
inserted.set(table, [...(inserted.get(table) ?? []), ...rows]);
|
||||
},
|
||||
}),
|
||||
select: selectChain,
|
||||
@@ -277,19 +287,25 @@ const observation = (overrides: Record<string, unknown> = {}) => ({
|
||||
const persistRun = async (
|
||||
fake: TxFake,
|
||||
observations: ReturnType<typeof observation>[],
|
||||
): Promise<void> => {
|
||||
): Promise<string[]> => {
|
||||
const service = new ExpertiseIngestionService(
|
||||
{
|
||||
transaction: async (callback: (value: unknown) => Promise<void>) => callback(fake.tx),
|
||||
} as never,
|
||||
'user_1',
|
||||
);
|
||||
await service['persistDomainRun']({
|
||||
agentId: 'agent_1',
|
||||
return service['persistDomainRun']({
|
||||
domain: { id: 'domain_1' },
|
||||
observations: observations as never,
|
||||
operationId: 'operation_1',
|
||||
topicId: 'topic_1',
|
||||
run: {
|
||||
actorId: 'agent_1',
|
||||
actorType: 'agent',
|
||||
hadHumanInLoop: false,
|
||||
operationId: 'operation_1',
|
||||
reflectionKey: 'topic:topic_1:operation:operation_1',
|
||||
subjectId: 'topic_1',
|
||||
subjectType: 'topic',
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
@@ -318,6 +334,76 @@ describe('ExpertiseIngestionService.persistDomainRun', () => {
|
||||
expect(hit?.runId).toBe(run?.id);
|
||||
});
|
||||
|
||||
it('writes one hit per rejection a standard was distilled from', async () => {
|
||||
const fake = createTx([]);
|
||||
await persistRun(fake, [
|
||||
observation({
|
||||
outcome: 'violation',
|
||||
sourceCheckResultIds: ['check_a', 'check_b', 'check_c'],
|
||||
}),
|
||||
]);
|
||||
|
||||
// hitCount has to read as "violated three times", not "analysed once", or the frequency
|
||||
// the standards list ranks by is a count of analysis passes.
|
||||
const hits = fake.inserted.get(expertiseHits) ?? [];
|
||||
expect(hits).toHaveLength(3);
|
||||
expect(hits.map((hit) => hit.sourceCheckResultId)).toEqual(['check_a', 'check_b', 'check_c']);
|
||||
expect(fake.inserted.get(expertiseLessons)?.[0].hitCount).toBe(3);
|
||||
expect(fake.inserted.get(expertiseLessons)?.[0].exampleCount).toBe(1);
|
||||
});
|
||||
|
||||
it('counts a rejection once when the model cites its label twice', async () => {
|
||||
const fake = createTx([]);
|
||||
await persistRun(fake, [
|
||||
observation({
|
||||
outcome: 'violation',
|
||||
sourceCheckResultIds: ['check_a', 'check_a', 'check_b'],
|
||||
}),
|
||||
]);
|
||||
|
||||
// hitCount ranks the standards list and decides core-versus-niche, so a label the model
|
||||
// repeated must not make one rejection look like two.
|
||||
const hits = fake.inserted.get(expertiseHits) ?? [];
|
||||
expect(hits.map((hit) => hit.sourceCheckResultId)).toEqual(['check_a', 'check_b']);
|
||||
expect(fake.inserted.get(expertiseLessons)?.[0].hitCount).toBe(2);
|
||||
});
|
||||
|
||||
it('stores what a standard rests on as columns, not as prose in its body', async () => {
|
||||
const fake = createTx([]);
|
||||
await persistRun(fake, [
|
||||
observation({
|
||||
reasonKind: 'taste',
|
||||
reasonSource: 'inferred',
|
||||
reasoning: '属主不接受多余分割线',
|
||||
}),
|
||||
]);
|
||||
|
||||
const lesson = fake.inserted.get(expertiseLessons)?.[0];
|
||||
// The compile step refuses to compile taste, and it cannot do that by matching a phrase: the
|
||||
// body is written in whatever language the reviewer used.
|
||||
expect(lesson).toMatchObject({ reasonKind: 'taste', reasonSource: 'inferred' });
|
||||
expect(
|
||||
(lesson?.sections as { body: string; key: string }[]).find((s) => s.key === 'why'),
|
||||
).toEqual({ body: '属主不接受多余分割线', key: 'why' });
|
||||
});
|
||||
|
||||
it('keeps the limits section only when the model stated one', async () => {
|
||||
const withLimits = createTx([]);
|
||||
await persistRun(withLimits, [observation({ limits: 'Not inside chart internals' })]);
|
||||
expect(withLimits.inserted.get(expertiseLessons)?.[0].sections).toContainEqual({
|
||||
body: 'Not inside chart internals',
|
||||
key: 'limits',
|
||||
});
|
||||
|
||||
const withoutLimits = createTx([]);
|
||||
await persistRun(withoutLimits, [observation({ limits: ' ' })]);
|
||||
expect(
|
||||
(withoutLimits.inserted.get(expertiseLessons)?.[0].sections as { key: string }[]).map(
|
||||
(section) => section.key,
|
||||
),
|
||||
).toEqual(['rule', 'why', 'how']);
|
||||
});
|
||||
|
||||
it('attaches by code when the model returns a real lesson code', async () => {
|
||||
const fake = createTx([
|
||||
{ code: 'P-07', id: 'lesson_7', status: 'active', title: 'Separate the runtime plane' },
|
||||
@@ -370,3 +456,43 @@ describe('ExpertiseIngestionService.persistDomainRun', () => {
|
||||
expect(runUpdate).toMatchObject({ instanceCount: 1, newCount: 1 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('ExpertiseIngestionService.consolidateTouched', () => {
|
||||
it('reports every standard a round touched, whether it was created or attached to', async () => {
|
||||
const fake = createTx([
|
||||
{ code: 'P-01', id: 'lesson_existing', status: 'active', title: '既有标准' },
|
||||
]);
|
||||
|
||||
const touched = await persistRun(fake, [
|
||||
observation({ existingLessonCode: 'P-01' }),
|
||||
observation({ title: '一条新的标准' }),
|
||||
]);
|
||||
|
||||
// The attached lesson is the one consolidation exists for, so it must survive the round even
|
||||
// though nothing about it was inserted.
|
||||
expect(touched).toContain('lesson_existing');
|
||||
expect(touched).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('keeps a recorded round when restating one of its standards fails', async () => {
|
||||
const service = new ExpertiseIngestionService({} as never, 'user_1');
|
||||
dueForConsolidation.mockResolvedValue(['lesson_1', 'lesson_2']);
|
||||
consolidate
|
||||
.mockRejectedValueOnce(new Error('provider down'))
|
||||
.mockResolvedValueOnce({ generalized: true, lessonId: 'lesson_2', note: '' });
|
||||
|
||||
const results = await service['consolidateTouched']('domain_1', ['lesson_1', 'lesson_2']);
|
||||
|
||||
// The rejections are already committed; a failed rewrite leaves that standard worded as it
|
||||
// was, which is exactly the state before this pass existed.
|
||||
expect(consolidate).toHaveBeenCalledTimes(2);
|
||||
expect(results).toEqual([{ generalized: true, lessonId: 'lesson_2', note: '' }]);
|
||||
});
|
||||
|
||||
it('does not reach for the model when a round touched nothing', async () => {
|
||||
const service = new ExpertiseIngestionService({} as never, 'user_1');
|
||||
|
||||
await expect(service['consolidateTouched']('domain_1', [])).resolves.toEqual([]);
|
||||
expect(dueForConsolidation).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -2,34 +2,62 @@ import { randomUUID } from 'node:crypto';
|
||||
|
||||
import { TRACING_SCENARIOS } from '@lobechat/const';
|
||||
import {
|
||||
acceptances,
|
||||
expertiseDomains,
|
||||
expertiseDomainSnapshots,
|
||||
expertiseHits,
|
||||
expertiseLessons,
|
||||
expertiseRuns,
|
||||
messages,
|
||||
projects,
|
||||
topics,
|
||||
verifyCheckResults,
|
||||
} from '@lobechat/database/schemas';
|
||||
import {
|
||||
chainExpertiseRejectionIngestion,
|
||||
chainExpertiseTopicIngestion,
|
||||
EXPERTISE_REJECTION_INGESTION_JSON_SCHEMA,
|
||||
EXPERTISE_REJECTION_INGESTION_PROMPT_VERSION,
|
||||
EXPERTISE_TOPIC_INGESTION_JSON_SCHEMA,
|
||||
EXPERTISE_TOPIC_INGESTION_PROMPT_VERSION,
|
||||
} from '@lobechat/prompts';
|
||||
import type { VerifyCheckDecisionDetail } from '@lobechat/types';
|
||||
import debug from 'debug';
|
||||
import { and, asc, count, desc, eq, gt, isNotNull, isNull, max, or, sql } from 'drizzle-orm';
|
||||
import pMap from 'p-map';
|
||||
import { z } from 'zod';
|
||||
|
||||
import { AgentSignalReviewContextModel } from '@/database/models/agentSignal/reviewContext';
|
||||
import { ExpertiseModel } from '@/database/models/expertise';
|
||||
import { FileModel } from '@/database/models/file';
|
||||
import { VerifyEvidenceModel } from '@/database/models/verifyEvidence';
|
||||
import type { LobeChatDatabase } from '@/database/type';
|
||||
import { notShareVisitorMessage, notShareVisitorTopic } from '@/database/utils/shareVisitor';
|
||||
import type { CompletionCallbackParams } from '@/server/services/agentSignal/policies/completionPolicy';
|
||||
import { AiGenerationService } from '@/server/services/aiGeneration';
|
||||
import { FileService } from '@/server/services/file';
|
||||
import { resolveModelReadableFrameUrl } from '@/server/services/verify/modelFrames';
|
||||
|
||||
import type { ConsolidationResult } from './consolidation';
|
||||
import { ExpertiseConsolidationService } from './consolidation';
|
||||
import { resolveExpertiseModelConfig } from './modelConfig';
|
||||
import { isProviderAccountError } from './providerAccountError';
|
||||
|
||||
const log = debug('lobe-server:expertise-ingestion');
|
||||
|
||||
const MAX_CONTEXT_MESSAGES = 24;
|
||||
const MAX_CONTEXT_CHARS = 24_000;
|
||||
/**
|
||||
* Frames attached to one round's distillation. A round is a batch, unlike the single-check review
|
||||
* that caps at 3 — but every frame is a full base64 body, so the cap is what keeps a 20-rejection
|
||||
* round from becoming a request no provider will accept.
|
||||
*/
|
||||
const MAX_REJECTION_FRAMES = 8;
|
||||
const VISUAL_REJECTION_EVIDENCE_TYPES = new Set(['screenshot', 'gif']);
|
||||
|
||||
/** Normalized 0-1 region coordinates read better to a model as percentages. */
|
||||
const pct = (value: number) => `${Math.round(value * 100)}%`;
|
||||
|
||||
const LESSON_CODE_PATTERN = /^P-\d+$/;
|
||||
const AnalysisSchema = z.object({
|
||||
domains: z.array(
|
||||
@@ -52,6 +80,37 @@ const AnalysisSchema = z.object({
|
||||
),
|
||||
});
|
||||
|
||||
/**
|
||||
* Same shape as {@link AnalysisSchema}, minus `outcome` (a rejection is always a violation) and
|
||||
* plus the rejections each observation was distilled from.
|
||||
*/
|
||||
const RejectionAnalysisSchema = z.object({
|
||||
domains: z.array(
|
||||
z.object({
|
||||
domainId: z.string(),
|
||||
matches: z.boolean(),
|
||||
observations: z
|
||||
.array(
|
||||
z.object({
|
||||
example: z.string(),
|
||||
// Empty string, not null: see the chain's schema comment — a nullable union comes
|
||||
// back from the pinned model as `{}` and takes the whole round down with it.
|
||||
existingLessonCode: z.string(),
|
||||
layer: z.string(),
|
||||
limits: z.string(),
|
||||
reasonKind: z.enum(['mechanism', 'taste']),
|
||||
reasoning: z.string(),
|
||||
reasonSource: z.enum(['reviewer', 'inferred']),
|
||||
sourceRefs: z.array(z.string()),
|
||||
subject: z.string(),
|
||||
title: z.string(),
|
||||
}),
|
||||
)
|
||||
.max(8),
|
||||
}),
|
||||
),
|
||||
});
|
||||
|
||||
/**
|
||||
* The identity a lesson is deduplicated by.
|
||||
*
|
||||
@@ -91,6 +150,55 @@ interface ExpertiseCompletionInput {
|
||||
topicId: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* One practice run, as the caller defines it. A topic completion and an acceptance round are both
|
||||
* "a complete judgement of one object", so they share the whole write path below and differ only
|
||||
* in these fields — notably `reflectionKey`, which is what makes a replayed ingestion a no-op.
|
||||
*/
|
||||
interface ExpertiseRunDescriptor {
|
||||
actorId: string;
|
||||
actorType: 'agent' | 'system' | 'user';
|
||||
hadHumanInLoop: boolean;
|
||||
operationId?: string;
|
||||
reflectionKey: string;
|
||||
subjectId: string;
|
||||
subjectType: 'document' | 'standalone' | 'task' | 'topic';
|
||||
}
|
||||
|
||||
/** An observation ready to persist, after whichever analysis produced it. */
|
||||
interface PersistableObservation {
|
||||
example: string;
|
||||
existingLessonCode: string | null;
|
||||
layer: string | null;
|
||||
/** When the lesson does not apply — kept out of the prompt-facing sections when absent. */
|
||||
limits?: string | null;
|
||||
outcome: 'pass' | 'violation';
|
||||
reasoning: string;
|
||||
/**
|
||||
* Whether the standard rests on something observable or on the owner's preference. A taste
|
||||
* standard is legitimate — a reviewer is allowed to just want things a certain way — but it has
|
||||
* no objective test, so the compile step must not turn it into a criterion that blocks a
|
||||
* delivery on its own.
|
||||
*/
|
||||
reasonKind?: 'mechanism' | 'taste';
|
||||
/**
|
||||
* Whether the reviewer gave the reason or the distillation supplied the mechanism. An inferred
|
||||
* reason is still worth keeping — it is what lets a standard transfer to a screen nobody has
|
||||
* built yet — but it must never read as something the reviewer said, because the compile step
|
||||
* downstream turns a lesson's reason into an enforced criterion.
|
||||
*/
|
||||
reasonSource?: 'inferred' | 'reviewer';
|
||||
/**
|
||||
* The rejections this observation was distilled from. One standard can be violated several
|
||||
* times in a single round, and each violation is its own hit — that is what makes "sourced from
|
||||
* N rejections" a real count rather than a count of analysis passes.
|
||||
*/
|
||||
sourceCheckResultIds?: string[];
|
||||
/** What the standard is really about, once the concrete names are replaced by what they exemplify. */
|
||||
subject?: null | string;
|
||||
title: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Turns one completed topic turn into bounded expertise evidence.
|
||||
*
|
||||
@@ -334,10 +442,19 @@ export class ExpertiseIngestionService {
|
||||
const domain = domains.find((item) => item.id === result.domainId);
|
||||
if (!domain || !result.matches) continue;
|
||||
await this.persistDomainRun({
|
||||
...input,
|
||||
domain,
|
||||
hadHumanInLoop: topicContext.hadHumanInLoop,
|
||||
observations: result.observations,
|
||||
run: {
|
||||
actorId: input.agentId,
|
||||
actorType: 'agent',
|
||||
hadHumanInLoop: topicContext.hadHumanInLoop,
|
||||
operationId: input.operationId,
|
||||
reflectionKey: input.ingestionKey
|
||||
? `topic:${input.topicId}:${input.ingestionKey}`
|
||||
: `topic:${input.topicId}:operation:${input.operationId}`,
|
||||
subjectId: input.topicId,
|
||||
subjectType: 'topic',
|
||||
},
|
||||
});
|
||||
ingested += 1;
|
||||
}
|
||||
@@ -345,6 +462,288 @@ export class ExpertiseIngestionService {
|
||||
return { ingested, reason: ingested > 0 ? 'matched' : 'no-match' } as const;
|
||||
};
|
||||
|
||||
/**
|
||||
* Distils one settled acceptance round's rejections into delivery standards.
|
||||
*
|
||||
* Called when the NEXT round lands (or the acceptance completes), because that is the first
|
||||
* moment the reviewer's judgement on this round is certainly final: the product's own reject
|
||||
* flow ends at a clipboard copy, so no server-side event marks "I finished reviewing".
|
||||
*/
|
||||
ingestAcceptanceRound = async (input: { acceptanceId: string; verifyRunId: string }) => {
|
||||
const [acceptance] = await this.db
|
||||
.select({
|
||||
id: acceptances.id,
|
||||
projectId: acceptances.projectId,
|
||||
subjectId: acceptances.subjectId,
|
||||
subjectType: acceptances.subjectType,
|
||||
})
|
||||
.from(acceptances)
|
||||
.where(and(eq(acceptances.id, input.acceptanceId), eq(acceptances.userId, this.userId)))
|
||||
.limit(1);
|
||||
if (!acceptance) return { ingested: 0, reason: 'no-acceptance' } as const;
|
||||
|
||||
const rejections = await this.db
|
||||
.select({
|
||||
detail: verifyCheckResults.userDecisionDetail,
|
||||
id: verifyCheckResults.id,
|
||||
title: verifyCheckResults.checkItemTitle,
|
||||
})
|
||||
.from(verifyCheckResults)
|
||||
.where(
|
||||
and(
|
||||
eq(verifyCheckResults.verifyRunId, input.verifyRunId),
|
||||
eq(verifyCheckResults.userDecision, 'rejected'),
|
||||
),
|
||||
)
|
||||
.orderBy(asc(verifyCheckResults.checkItemIndex), asc(verifyCheckResults.createdAt));
|
||||
if (rejections.length === 0) return { ingested: 0, reason: 'no-rejections' } as const;
|
||||
|
||||
const labelled = rejections.map((rejection, index) => ({
|
||||
...rejection,
|
||||
ref: `R${index + 1}`,
|
||||
}));
|
||||
const byRef = new Map(labelled.map((rejection) => [rejection.ref, rejection.id]));
|
||||
const { visuals, withheld } = await this.resolveRejectionFrames(labelled);
|
||||
const frameLabelByEvidence = new Map(
|
||||
visuals.map((visual, index) => [visual.evidenceId, `frame ${index + 1}`]),
|
||||
);
|
||||
const rendered = labelled
|
||||
.map((rejection) => {
|
||||
const regions = (rejection.detail?.annotations ?? []).map((annotation) => {
|
||||
const frame = frameLabelByEvidence.get(annotation.evidenceId);
|
||||
// Regions are normalized 0-1; percentages read better to a model than raw floats.
|
||||
const at = annotation.rect
|
||||
? ` at ${pct(annotation.rect.x)},${pct(annotation.rect.y)} sized ${pct(annotation.rect.width)}×${pct(annotation.rect.height)}`
|
||||
: '';
|
||||
return ` circled${frame ? ` on ${frame}` : ''}${at}: ${annotation.comment?.trim() || '(no note)'}`;
|
||||
});
|
||||
return [
|
||||
`[${rejection.ref}] promised: ${rejection.title ?? '(untitled check)'}`,
|
||||
rejection.detail?.comment?.trim() && ` said: ${rejection.detail.comment.trim()}`,
|
||||
...regions,
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join('\n');
|
||||
})
|
||||
.join('\n\n')
|
||||
.slice(0, MAX_CONTEXT_CHARS);
|
||||
|
||||
const expertiseModel = new ExpertiseModel(this.db, this.userId, this.workspaceId);
|
||||
const listDomains = () =>
|
||||
acceptance.projectId
|
||||
? expertiseModel.listDomainsForProject(acceptance.projectId)
|
||||
: expertiseModel.listDomainsForOwner();
|
||||
|
||||
let bound = await listDomains();
|
||||
if (bound.length === 0) {
|
||||
await this.createDeliveryStandardsDomain(acceptance.projectId);
|
||||
bound = await listDomains();
|
||||
if (bound.length === 0) return { ingested: 0, reason: 'no-domains' } as const;
|
||||
}
|
||||
|
||||
const modelConfig = await resolveExpertiseModelConfig(this.db, this.userId);
|
||||
const domains = await Promise.all(
|
||||
bound.map(async ({ domain }) => ({
|
||||
domainFilter: domain.domainFilter,
|
||||
id: domain.id,
|
||||
layers: domain.layers,
|
||||
lessons: (await expertiseModel.listLessons(domain.id)).map((lesson) => ({
|
||||
code: lesson.code,
|
||||
layer: lesson.layer,
|
||||
why: lesson.sections.find((section) => section.key === 'why')?.body ?? null,
|
||||
title: lesson.title,
|
||||
})),
|
||||
outOfScope: domain.outOfScope,
|
||||
title: domain.title,
|
||||
})),
|
||||
);
|
||||
|
||||
const ai = new AiGenerationService(this.db, this.userId, this.workspaceId);
|
||||
const raw = await ai.generateObject(
|
||||
{
|
||||
...chainExpertiseRejectionIngestion({
|
||||
domains,
|
||||
rejections: rendered,
|
||||
visuals,
|
||||
withheldEvidence: withheld,
|
||||
}),
|
||||
...modelConfig,
|
||||
schema: EXPERTISE_REJECTION_INGESTION_JSON_SCHEMA,
|
||||
},
|
||||
{
|
||||
metadata: { trigger: 'expertise_rejection_ingestion' },
|
||||
tracing: {
|
||||
promptVersion: EXPERTISE_REJECTION_INGESTION_PROMPT_VERSION,
|
||||
scenario: TRACING_SCENARIOS.ExpertiseRejectionIngestion,
|
||||
schemaName: EXPERTISE_REJECTION_INGESTION_JSON_SCHEMA.name,
|
||||
},
|
||||
},
|
||||
);
|
||||
const analysis = RejectionAnalysisSchema.parse(raw);
|
||||
|
||||
let ingested = 0;
|
||||
const consolidated: ConsolidationResult[] = [];
|
||||
for (const result of analysis.domains) {
|
||||
const domain = domains.find((item) => item.id === result.domainId);
|
||||
if (!domain || !result.matches || result.observations.length === 0) continue;
|
||||
const touched = await this.persistDomainRun({
|
||||
domain,
|
||||
observations: result.observations.map((observation) => ({
|
||||
...observation,
|
||||
existingLessonCode: observation.existingLessonCode.trim() || null,
|
||||
layer: observation.layer.trim() || null,
|
||||
// A rejection is a violation by construction — never let the model relabel it a pass.
|
||||
outcome: 'violation' as const,
|
||||
// Hallucinated refs are dropped rather than failing the round: losing one provenance
|
||||
// link is cheaper than losing the whole distillation.
|
||||
sourceCheckResultIds: observation.sourceRefs
|
||||
.map((ref) => byRef.get(ref))
|
||||
.filter((id): id is string => Boolean(id)),
|
||||
})),
|
||||
run: {
|
||||
actorId: this.userId,
|
||||
actorType: 'user',
|
||||
hadHumanInLoop: true,
|
||||
reflectionKey: `acceptance:${acceptance.id}:run:${input.verifyRunId}`,
|
||||
subjectId: acceptance.subjectId,
|
||||
subjectType: acceptance.subjectType,
|
||||
},
|
||||
});
|
||||
ingested += 1;
|
||||
consolidated.push(...(await this.consolidateTouched(domain.id, touched)));
|
||||
}
|
||||
|
||||
return {
|
||||
consolidated,
|
||||
ingested,
|
||||
reason: ingested > 0 ? 'matched' : 'no-match',
|
||||
} as const;
|
||||
};
|
||||
|
||||
/**
|
||||
* Restates the standards this round pushed past the instance threshold.
|
||||
*
|
||||
* Deliberately after the write, not inside it: consolidation reads frames and calls a model, and
|
||||
* a round of rejections that is safely recorded must not be rolled back because the rewrite of
|
||||
* an unrelated standard failed. For the same reason a failure here is logged, not thrown — the
|
||||
* standard keeps the wording it already had, which is exactly the state before this pass existed.
|
||||
*/
|
||||
private consolidateTouched = async (domainId: string, lessonIds: string[]) => {
|
||||
if (lessonIds.length === 0) return [];
|
||||
const service = new ExpertiseConsolidationService(this.db, this.userId, this.workspaceId);
|
||||
const due = await service.dueForConsolidation(domainId, lessonIds);
|
||||
|
||||
return (
|
||||
await pMap(
|
||||
due,
|
||||
async (lessonId) => {
|
||||
try {
|
||||
return await service.consolidate(lessonId);
|
||||
} catch (error) {
|
||||
log('consolidation failed for lesson %s: %O', lessonId, error);
|
||||
return null;
|
||||
}
|
||||
},
|
||||
{ concurrency: 2 },
|
||||
)
|
||||
).filter((result): result is ConsolidationResult => Boolean(result));
|
||||
};
|
||||
|
||||
/**
|
||||
* The frames behind one round's rejections, circled ones first.
|
||||
*
|
||||
* Most of these rejections are visual — 616 of this owner's 876 carry a circled region — so a
|
||||
* text-only request asks the model to distil "this isn't aligned" without ever seeing what was
|
||||
* not aligned. Frames are inlined as data URIs rather than linked, because a link makes the
|
||||
* provider fetch from our storage and a local or private bucket is simply unreachable.
|
||||
*
|
||||
* Priority is deliberate rather than first-come: a budget spent in insertion order drops exactly
|
||||
* the circled frames the reviewer pointed at. Whatever does not fit is named in the prompt, so
|
||||
* the model never reads a withheld frame as evidence of absence.
|
||||
*/
|
||||
private resolveRejectionFrames = async (
|
||||
rejections: { detail: VerifyCheckDecisionDetail | null; id: string; ref: string }[],
|
||||
) => {
|
||||
const fileModel = new FileModel(this.db, this.userId, this.workspaceId);
|
||||
const fileService = new FileService(this.db, this.userId, this.workspaceId);
|
||||
const evidenceModel = new VerifyEvidenceModel(this.db, this.userId, this.workspaceId);
|
||||
|
||||
const candidates: { circled: boolean; evidenceId: string; fileId: string; label: string }[] =
|
||||
[];
|
||||
for (const rejection of rejections) {
|
||||
const circled = new Set(
|
||||
(rejection.detail?.annotations ?? []).map((annotation) => annotation.evidenceId),
|
||||
);
|
||||
const rows = await evidenceModel.listByCheckResult(rejection.id);
|
||||
for (const row of rows) {
|
||||
if (!VISUAL_REJECTION_EVIDENCE_TYPES.has(row.type) || !row.fileId) continue;
|
||||
const isCircled = circled.has(row.id);
|
||||
candidates.push({
|
||||
circled: isCircled,
|
||||
evidenceId: row.id,
|
||||
fileId: row.fileId,
|
||||
label: `${rejection.ref}${isCircled ? ' (circled)' : ''} — ${row.description || 'evidence'}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const ordered = [
|
||||
...candidates.filter((candidate) => candidate.circled),
|
||||
...candidates.filter((candidate) => !candidate.circled),
|
||||
];
|
||||
const selected = ordered.slice(0, MAX_REJECTION_FRAMES);
|
||||
|
||||
const resolved = await pMap(
|
||||
selected,
|
||||
async (item) => {
|
||||
const file = await fileModel.findById(item.fileId);
|
||||
if (!file) return null;
|
||||
return {
|
||||
accessUrl: await resolveModelReadableFrameUrl(fileService, file),
|
||||
evidenceId: item.evidenceId,
|
||||
label: item.label,
|
||||
};
|
||||
},
|
||||
{ concurrency: 4 },
|
||||
);
|
||||
|
||||
const visuals = resolved.filter((item): item is NonNullable<typeof item> => Boolean(item));
|
||||
const dropped = ordered.length - selected.length;
|
||||
return {
|
||||
visuals,
|
||||
withheld:
|
||||
dropped > 0
|
||||
? `${dropped} further screenshot(s) from this round were not attached. Absence of a frame is not evidence that nothing was wrong there.`
|
||||
: undefined,
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* The domain a project's rejections land in before anyone has authored one.
|
||||
*
|
||||
* Owned by the user (never by the project — a project mounts standards, it does not own them),
|
||||
* so the same domain can later be mounted by a sibling project without being copied.
|
||||
*/
|
||||
private createDeliveryStandardsDomain = async (projectId: null | string) => {
|
||||
const [project] = projectId
|
||||
? await this.db
|
||||
.select({ name: projects.name })
|
||||
.from(projects)
|
||||
.where(eq(projects.id, projectId))
|
||||
.limit(1)
|
||||
: [];
|
||||
const scope = project?.name ?? 'my work';
|
||||
|
||||
return new ExpertiseModel(this.db, this.userId, this.workspaceId).createDomain({
|
||||
brief: `Delivery standards distilled from rejected acceptance checks on ${scope}.`,
|
||||
carrier: projectId ? { id: projectId, type: 'project' } : { type: 'user' },
|
||||
domainFilter: `Strip the screen names, component names and this task's name out of the requirement — does it still hold for any delivery on ${scope}? Only then is it mine.`,
|
||||
outOfScope:
|
||||
'One-off facts about a single screen, and anything that stops being true once the task changes.',
|
||||
title: `${scope} delivery standards`,
|
||||
});
|
||||
};
|
||||
|
||||
private readTopicContext = async (topicId: string) => {
|
||||
const rows = await this.db.query.messages.findMany({
|
||||
columns: { content: true, createdAt: true, role: true },
|
||||
@@ -369,16 +768,13 @@ export class ExpertiseIngestionService {
|
||||
};
|
||||
};
|
||||
|
||||
private persistDomainRun = async (
|
||||
input: ExpertiseCompletionInput & {
|
||||
domain: { id: string };
|
||||
observations: z.infer<typeof AnalysisSchema>['domains'][number]['observations'];
|
||||
},
|
||||
) => {
|
||||
const reflectionKey = input.ingestionKey
|
||||
? `topic:${input.topicId}:${input.ingestionKey}`
|
||||
: `topic:${input.topicId}:operation:${input.operationId}`;
|
||||
await this.db.transaction(async (tx) => {
|
||||
private persistDomainRun = async (input: {
|
||||
domain: { id: string };
|
||||
observations: PersistableObservation[];
|
||||
run: ExpertiseRunDescriptor;
|
||||
}): Promise<string[]> => {
|
||||
const { reflectionKey } = input.run;
|
||||
return this.db.transaction(async (tx) => {
|
||||
await tx
|
||||
.select({ id: expertiseDomains.id })
|
||||
.from(expertiseDomains)
|
||||
@@ -394,7 +790,7 @@ export class ExpertiseIngestionService {
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
if (existingRun) return;
|
||||
if (existingRun) return [];
|
||||
|
||||
const [prior] = await tx
|
||||
.select({ value: max(expertiseRuns.runIndex) })
|
||||
@@ -406,16 +802,16 @@ export class ExpertiseIngestionService {
|
||||
let instanceCount = 0;
|
||||
|
||||
await tx.insert(expertiseRuns).values({
|
||||
actorId: input.agentId,
|
||||
actorType: 'agent',
|
||||
actorId: input.run.actorId,
|
||||
actorType: input.run.actorType,
|
||||
completedAt: new Date(),
|
||||
domainId: input.domain.id,
|
||||
hadHumanInLoop: input.hadHumanInLoop ?? false,
|
||||
hadHumanInLoop: input.run.hadHumanInLoop,
|
||||
id: runId,
|
||||
reflectionKey,
|
||||
runIndex,
|
||||
subjectId: input.topicId,
|
||||
subjectType: 'topic',
|
||||
subjectId: input.run.subjectId,
|
||||
subjectType: input.run.subjectType,
|
||||
userId: this.userId,
|
||||
workspaceId: this.workspaceId,
|
||||
});
|
||||
@@ -444,6 +840,31 @@ export class ExpertiseIngestionService {
|
||||
if (!byTitle.has(key)) byTitle.set(key, lesson.id);
|
||||
}
|
||||
const countedLessonIds = new Set<string>();
|
||||
const touchedLessonIds = new Set<string>();
|
||||
|
||||
// One observation yields one hit per rejection it cites, so hitCount stays a count of real
|
||||
// violations; an observation with no cited source (the topic path) still yields one.
|
||||
//
|
||||
// Deduplicated here rather than at the caller: `hitCount` is what ranks the standards list
|
||||
// and decides core-versus-niche, so one rejection counted twice because the model repeated
|
||||
// its label ("R1", "R1") is a durable distortion, and this is the only place hits are made.
|
||||
const writeHits = async (lessonId: string, observation: PersistableObservation) => {
|
||||
const cited = [...new Set(observation.sourceCheckResultIds ?? [])];
|
||||
const sources = cited.length > 0 ? cited : [undefined];
|
||||
await tx.insert(expertiseHits).values(
|
||||
sources.map((sourceCheckResultId) => ({
|
||||
domainId: input.domain.id,
|
||||
example: observation.example,
|
||||
lessonId,
|
||||
note: observation.reasoning,
|
||||
operationId: input.run.operationId,
|
||||
outcome: observation.outcome,
|
||||
runId,
|
||||
sourceCheckResultId,
|
||||
})),
|
||||
);
|
||||
return sources.length;
|
||||
};
|
||||
|
||||
for (const observation of input.observations) {
|
||||
const matchedId = matchLesson(observation, { byCode, byTitle });
|
||||
@@ -458,50 +879,48 @@ export class ExpertiseIngestionService {
|
||||
domainId: input.domain.id,
|
||||
id: lessonId,
|
||||
exampleCount: 1,
|
||||
hitCount: 1,
|
||||
hitCount: new Set(observation.sourceCheckResultIds ?? []).size || 1,
|
||||
hitRunCount: 1,
|
||||
layer: observation.layer,
|
||||
lastHitAt: new Date(),
|
||||
lastHitRunId: runId,
|
||||
originRunId: runId,
|
||||
polarity: 'rule',
|
||||
// Columns, not a sentence appended to the body: the compile step has to be able to
|
||||
// refuse a taste standard, and the body is written in the reviewer's own language.
|
||||
reasonKind: observation.reasonKind,
|
||||
reasonSource: observation.reasonSource,
|
||||
sections: [
|
||||
{ body: observation.title, key: 'rule' },
|
||||
{ body: observation.reasoning, key: 'why' },
|
||||
{ body: observation.example, key: 'how' },
|
||||
{
|
||||
body: observation.subject?.trim()
|
||||
? `${observation.title}\n\n适用对象：${observation.subject.trim()}`
|
||||
: observation.title,
|
||||
key: 'rule' as const,
|
||||
},
|
||||
{ body: observation.reasoning, key: 'why' as const },
|
||||
{ body: observation.example, key: 'how' as const },
|
||||
...(observation.limits?.trim()
|
||||
? [{ body: observation.limits.trim(), key: 'limits' as const }]
|
||||
: []),
|
||||
],
|
||||
title: observation.title,
|
||||
});
|
||||
byCode.set(code, lessonId);
|
||||
byTitle.set(normalizeLessonTitle(observation.title), lessonId);
|
||||
countedLessonIds.add(lessonId);
|
||||
await tx.insert(expertiseHits).values({
|
||||
domainId: input.domain.id,
|
||||
example: observation.example,
|
||||
lessonId,
|
||||
note: observation.reasoning,
|
||||
operationId: input.operationId,
|
||||
outcome: observation.outcome,
|
||||
runId,
|
||||
});
|
||||
touchedLessonIds.add(lessonId);
|
||||
await writeHits(lessonId, observation);
|
||||
} else {
|
||||
instanceCount += 1;
|
||||
await tx.insert(expertiseHits).values({
|
||||
domainId: input.domain.id,
|
||||
example: observation.example,
|
||||
lessonId: matchedId,
|
||||
note: observation.reasoning,
|
||||
operationId: input.operationId,
|
||||
outcome: observation.outcome,
|
||||
runId,
|
||||
});
|
||||
const hits = await writeHits(matchedId, observation);
|
||||
const firstHitThisRun = !countedLessonIds.has(matchedId);
|
||||
countedLessonIds.add(matchedId);
|
||||
touchedLessonIds.add(matchedId);
|
||||
await tx
|
||||
.update(expertiseLessons)
|
||||
.set({
|
||||
exampleCount: sql`${expertiseLessons.exampleCount} + 1`,
|
||||
hitCount: sql`${expertiseLessons.hitCount} + 1`,
|
||||
hitCount: sql`${expertiseLessons.hitCount} + ${hits}`,
|
||||
hitRunCount: firstHitThisRun
|
||||
? sql`${expertiseLessons.hitRunCount} + 1`
|
||||
: expertiseLessons.hitRunCount,
|
||||
@@ -549,6 +968,8 @@ export class ExpertiseIngestionService {
|
||||
runId,
|
||||
runIndex,
|
||||
});
|
||||
|
||||
return [...touchedLessonIds];
|
||||
});
|
||||
};
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import { AcceptanceService } from '../acceptanceService';
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
attachToAcceptance: vi.fn(),
|
||||
distilRejections: vi.fn(),
|
||||
findById: vi.fn(),
|
||||
findOwnTopicById: vi.fn(),
|
||||
findPolicyById: vi.fn(),
|
||||
@@ -60,6 +61,9 @@ vi.mock('@/database/models/topic', () => ({
|
||||
}));
|
||||
vi.mock('@/database/models/document', () => ({ DocumentModel: vi.fn() }));
|
||||
vi.mock('@/server/services/task', () => ({ TaskService: vi.fn() }));
|
||||
vi.mock('@/server/workflows/expertiseRejection', () => ({
|
||||
ExpertiseRejectionWorkflow: { trigger: mocks.distilRejections },
|
||||
}));
|
||||
|
||||
const service = () => new AcceptanceService({} as any, 'user-1');
|
||||
|
||||
@@ -170,6 +174,44 @@ describe('AcceptanceService decision gating', () => {
|
||||
expect(mocks.attachToAcceptance).toHaveBeenCalledWith('run-2', 'acc-1', undefined);
|
||||
});
|
||||
|
||||
it('distils the previous round once a new one lands', async () => {
|
||||
mocks.findById.mockResolvedValue(acceptance('rejected'));
|
||||
mocks.findRunById.mockResolvedValue({ acceptanceId: null, id: 'run-2' });
|
||||
mocks.listByAcceptance.mockResolvedValue([
|
||||
{ id: 'run-1', planConfirmedAt: new Date(), roundIndex: 1, status: 'failed' },
|
||||
]);
|
||||
mocks.attachToAcceptance.mockResolvedValue({
|
||||
acceptanceId: 'acc-1',
|
||||
id: 'run-2',
|
||||
roundIndex: 2,
|
||||
});
|
||||
|
||||
await service().attachRun('run-2', 'acc-1');
|
||||
|
||||
// The settled round is the previous one, never the round that just landed.
|
||||
expect(mocks.distilRejections).toHaveBeenCalledWith({
|
||||
acceptanceId: 'acc-1',
|
||||
userId: 'user-1',
|
||||
verifyRunId: 'run-1',
|
||||
workspaceId: undefined,
|
||||
});
|
||||
});
|
||||
|
||||
it('does not distil when the incoming run only folds into a draft round', async () => {
|
||||
mocks.findById.mockResolvedValue(acceptance('planned'));
|
||||
mocks.findRunById.mockResolvedValue({ acceptanceId: null, id: 'run-2', plan: [] });
|
||||
mocks.listByAcceptance.mockResolvedValue([
|
||||
{ id: 'run-1', planConfirmedAt: null, roundIndex: 1, status: 'planned', userDecision: null },
|
||||
]);
|
||||
mocks.foldIntoRound.mockResolvedValue({ acceptanceId: 'acc-1', id: 'run-1', roundIndex: 1 });
|
||||
|
||||
await service().attachRun('run-2', 'acc-1');
|
||||
|
||||
// No new round opened, so nothing settled — distilling here would read a round the
|
||||
// reviewer is still working on.
|
||||
expect(mocks.distilRejections).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('folds a new run into the draft round instead of opening another', async () => {
|
||||
mocks.findById.mockResolvedValue(acceptance('planned'));
|
||||
mocks.findRunById.mockResolvedValue({ acceptanceId: null, id: 'run-2', plan: [] });
|
||||
@@ -227,4 +269,31 @@ describe('AcceptanceService decision gating', () => {
|
||||
);
|
||||
expect(mocks.updateStatus).toHaveBeenCalledWith('acc-1', 'rejected');
|
||||
});
|
||||
|
||||
it.each([
|
||||
['accept', (svc: ReturnType<typeof service>) => svc.accept('acc-1', 'looks good')],
|
||||
['reject', (svc: ReturnType<typeof service>) => svc.reject('acc-1', 'not yet')],
|
||||
])('distils the final round when the reviewer settles by %s', async (_verb, decide) => {
|
||||
mocks.findById.mockResolvedValue(acceptance('delivered'));
|
||||
|
||||
await decide(service());
|
||||
|
||||
// No later round will ever follow this one, so a terminal decision is the only thing that can
|
||||
// settle it. Without this the last round of every acceptance is silently never learned from.
|
||||
expect(mocks.distilRejections).toHaveBeenCalledWith({
|
||||
acceptanceId: 'acc-1',
|
||||
userId: 'user-1',
|
||||
verifyRunId: 'run-1',
|
||||
workspaceId: undefined,
|
||||
});
|
||||
});
|
||||
|
||||
it('refuses a terminal decision with no round, and distils nothing', async () => {
|
||||
mocks.findById.mockResolvedValue(acceptance('delivered'));
|
||||
mocks.listByAcceptance.mockResolvedValue([]);
|
||||
|
||||
await expect(service().accept('acc-1')).rejects.toThrow('no verification round');
|
||||
expect(mocks.distilRejections).not.toHaveBeenCalled();
|
||||
expect(mocks.updateStatus).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -36,6 +36,7 @@ import type {
|
||||
} from '@/database/schemas/verify';
|
||||
import type { LobeChatDatabase } from '@/database/type';
|
||||
import { TaskService } from '@/server/services/task';
|
||||
import { ExpertiseRejectionWorkflow } from '@/server/workflows/expertiseRejection';
|
||||
|
||||
import { type AcceptanceMergeSummary, mergeAcceptanceRounds } from './acceptanceMerge';
|
||||
import { computeFalseFlags } from './feedbackService';
|
||||
@@ -711,6 +712,12 @@ export class AcceptanceService {
|
||||
const run = await this.runModel.attachToAcceptance(runId, acceptanceId, acceptance.visibility);
|
||||
await this.recomputeStatus(acceptanceId);
|
||||
log('run %s attached to acceptance %s as round %d', runId, acceptanceId, run.roundIndex);
|
||||
|
||||
// A new round landing is the first server-side proof that the reviewer is done with the
|
||||
// previous one: rejecting a check ends at a clipboard copy, so nothing else marks "I finished
|
||||
// reviewing".
|
||||
if (latest) this.distilSettledRound(acceptanceId, latest.id);
|
||||
|
||||
return run;
|
||||
};
|
||||
|
||||
@@ -831,6 +838,23 @@ export class AcceptanceService {
|
||||
return this.acceptanceModel.setCheckGroups(acceptanceId, normalized, expectedVersion);
|
||||
};
|
||||
|
||||
/**
|
||||
* Hands one settled round to distillation, fire-and-forget.
|
||||
*
|
||||
* Never awaited and never allowed to throw: this rides on the reviewer's own paths, and losing a
|
||||
* distillation is a missed lesson, while failing the caller loses their decision. Triggering the
|
||||
* same round twice is harmless — a round is distilled under a reflection key, and the second pass
|
||||
* finds the run already recorded and returns.
|
||||
*/
|
||||
private distilSettledRound = (acceptanceId: string, verifyRunId: string) => {
|
||||
void ExpertiseRejectionWorkflow.trigger({
|
||||
acceptanceId,
|
||||
userId: this.userId,
|
||||
verifyRunId,
|
||||
workspaceId: this.workspaceId,
|
||||
});
|
||||
};
|
||||
|
||||
/**
|
||||
* The user accepts the delivery — the terminal business event (P-12). Stamps
|
||||
* the decision on the current round, closes the aggregate, and best-effort
|
||||
@@ -839,9 +863,16 @@ export class AcceptanceService {
|
||||
accept = async (acceptanceId: string, comment?: string): Promise<AcceptanceItem> => {
|
||||
const acceptance = await this.requireDecidableAcceptance(acceptanceId);
|
||||
|
||||
await this.stampDecision(acceptanceId, 'accept', comment);
|
||||
const settled = await this.stampDecision(acceptanceId, 'accept', comment);
|
||||
await this.acceptanceModel.updateStatus(acceptanceId, 'accepted');
|
||||
|
||||
// A terminal decision settles the current round as surely as a new round landing does — and it
|
||||
// is the ONLY thing that settles the last one, which no later round will ever follow. Without
|
||||
// this, every acceptance silently loses whatever its final round taught. Accepting the delivery
|
||||
// still settles it: a reviewer can accept overall while individual checks were rejected along
|
||||
// the way, and those rejections are exactly the material.
|
||||
this.distilSettledRound(acceptanceId, settled);
|
||||
|
||||
if (acceptance.subjectType === 'task') await this.completeTaskSubject(acceptance.subjectId);
|
||||
|
||||
return (await this.acceptanceModel.findById(acceptanceId))!;
|
||||
@@ -861,9 +892,11 @@ export class AcceptanceService {
|
||||
reject = async (acceptanceId: string, comment: string): Promise<AcceptanceItem> => {
|
||||
await this.requireDecidableAcceptance(acceptanceId);
|
||||
|
||||
await this.stampDecision(acceptanceId, 'reject', comment);
|
||||
const settled = await this.stampDecision(acceptanceId, 'reject', comment);
|
||||
await this.acceptanceModel.updateStatus(acceptanceId, 'rejected');
|
||||
|
||||
this.distilSettledRound(acceptanceId, settled);
|
||||
|
||||
return (await this.acceptanceModel.findById(acceptanceId))!;
|
||||
};
|
||||
|
||||
@@ -1020,11 +1053,12 @@ export class AcceptanceService {
|
||||
return acceptance;
|
||||
};
|
||||
|
||||
/** Stamps the decision on the current round and returns it — the round that decision settles. */
|
||||
private stampDecision = async (
|
||||
acceptanceId: string,
|
||||
decision: 'accept' | 'reject',
|
||||
comment?: string,
|
||||
): Promise<void> => {
|
||||
): Promise<string> => {
|
||||
const runs = await this.runModel.listByAcceptance(acceptanceId);
|
||||
const current = runs.at(-1);
|
||||
if (!current) throw new Error('This acceptance has no verification round to decide on');
|
||||
@@ -1035,6 +1069,8 @@ export class AcceptanceService {
|
||||
...(comment ? { comment } : {}),
|
||||
};
|
||||
await this.runModel.setDecision(current.id, decision, detail);
|
||||
|
||||
return current.id;
|
||||
};
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
// @vitest-environment node
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { ExpertiseRejectionWorkflow } from './index';
|
||||
|
||||
const { appEnv, ingestAcceptanceRound, workflowClient } = vi.hoisted(() => ({
|
||||
appEnv: { APP_URL: 'https://app.test', enableQueueAgentRuntime: false, INTERNAL_APP_URL: '' },
|
||||
ingestAcceptanceRound: vi.fn(),
|
||||
workflowClient: { trigger: vi.fn() },
|
||||
}));
|
||||
|
||||
vi.mock('@/database/server', () => ({ getServerDB: vi.fn().mockResolvedValue({}) }));
|
||||
vi.mock('@/envs/app', () => ({ appEnv }));
|
||||
vi.mock('@/libs/qstash', () => ({ workflowClient }));
|
||||
vi.mock('@/server/services/expertise/ingestion', () => ({
|
||||
ExpertiseIngestionService: class {
|
||||
ingestAcceptanceRound = ingestAcceptanceRound;
|
||||
},
|
||||
}));
|
||||
|
||||
const settle = (verifyRunId: string, userId = 'user-1', workspaceId?: string) =>
|
||||
ExpertiseRejectionWorkflow.trigger({ acceptanceId: 'acc-1', userId, verifyRunId, workspaceId });
|
||||
|
||||
/** Resolves once every microtask queued by the triggers has run. */
|
||||
const drain = async () => {
|
||||
for (let index = 0; index < 20; index += 1) await Promise.resolve();
|
||||
};
|
||||
|
||||
afterEach(() => {
|
||||
vi.clearAllMocks();
|
||||
appEnv.enableQueueAgentRuntime = false;
|
||||
});
|
||||
|
||||
describe('ExpertiseRejectionWorkflow.trigger without a queue', () => {
|
||||
it('runs one reviewer’s rounds one at a time', async () => {
|
||||
let running = 0;
|
||||
let overlapped = false;
|
||||
const release: (() => void)[] = [];
|
||||
ingestAcceptanceRound.mockImplementation(() => {
|
||||
running += 1;
|
||||
if (running > 1) overlapped = true;
|
||||
return new Promise((resolve) =>
|
||||
release.push(() => {
|
||||
running -= 1;
|
||||
resolve({});
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
await settle('run-1');
|
||||
await settle('run-2');
|
||||
await drain();
|
||||
|
||||
// Both rounds would otherwise read the lesson catalog before either writes, and fork a lesson
|
||||
// that should have attached.
|
||||
expect(ingestAcceptanceRound).toHaveBeenCalledTimes(1);
|
||||
release[0]();
|
||||
await drain();
|
||||
expect(ingestAcceptanceRound).toHaveBeenCalledTimes(2);
|
||||
expect(overlapped).toBe(false);
|
||||
release[1]();
|
||||
await drain();
|
||||
});
|
||||
|
||||
it('keeps the rounds queued behind a failed one', async () => {
|
||||
ingestAcceptanceRound.mockRejectedValueOnce(new Error('provider down')).mockResolvedValue({});
|
||||
|
||||
await settle('run-1');
|
||||
await drain();
|
||||
await settle('run-2');
|
||||
await drain();
|
||||
|
||||
expect(ingestAcceptanceRound).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it('serializes two workspace members, who share one lesson catalog', async () => {
|
||||
ingestAcceptanceRound.mockReturnValue(new Promise(() => {}));
|
||||
|
||||
await settle('run-1', 'user-1', 'ws-1');
|
||||
await settle('run-2', 'user-2', 'ws-1');
|
||||
await drain();
|
||||
|
||||
// Domains and their bindings are workspace-wide, so two members racing would each mint a
|
||||
// different default domain and split the catalog in two.
|
||||
expect(ingestAcceptanceRound).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('does not serialize across reviewers', async () => {
|
||||
ingestAcceptanceRound.mockReturnValue(new Promise(() => {}));
|
||||
|
||||
await settle('run-1', 'user-1');
|
||||
await settle('run-2', 'user-2');
|
||||
await drain();
|
||||
|
||||
expect(ingestAcceptanceRound).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it('keys the queue by the workspace when there is one', async () => {
|
||||
appEnv.enableQueueAgentRuntime = true;
|
||||
|
||||
await settle('run-1', 'user-1', 'ws-1');
|
||||
await drain();
|
||||
|
||||
expect(workflowClient.trigger).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
flowControl: { key: 'expertise-rejection.workspace.ws-1', parallelism: 1 },
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('hands off to the queue when one is configured, without running locally', async () => {
|
||||
appEnv.enableQueueAgentRuntime = true;
|
||||
|
||||
await settle('run-1');
|
||||
await drain();
|
||||
|
||||
expect(ingestAcceptanceRound).not.toHaveBeenCalled();
|
||||
expect(workflowClient.trigger).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
flowControl: { key: 'expertise-rejection.user.user-1', parallelism: 1 },
|
||||
}),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,87 @@
|
||||
import debug from 'debug';
|
||||
|
||||
import { getServerDB } from '@/database/server';
|
||||
import { appEnv } from '@/envs/app';
|
||||
import { workflowClient } from '@/libs/qstash';
|
||||
import { ExpertiseIngestionService } from '@/server/services/expertise/ingestion';
|
||||
|
||||
import { runExpertiseRejectionWorkflow } from './run';
|
||||
import type { ExpertiseRejectionWorkflowPayload } from './types';
|
||||
|
||||
const log = debug('lobe-server:workflows:expertise-rejection');
|
||||
|
||||
/**
|
||||
* In-flight local distillations, one chain per lesson catalog.
|
||||
*
|
||||
* The queue path buys serialization with `flowControl`; without a queue, two rounds settling at
|
||||
* once would each read the lesson catalog before the other writes. They would then both see no
|
||||
* bound domain and create a duplicate default, or both distil against the same stale catalog and
|
||||
* fork a lesson that should have attached — and the persistence lock, which only serializes the
|
||||
* writes, would happily record both.
|
||||
*/
|
||||
const localRuns = new Map<string, Promise<void>>();
|
||||
|
||||
/**
|
||||
* What two rounds have to share before they can collide: the catalog they read and write.
|
||||
*
|
||||
* That is the workspace when there is one — domains and their bindings are workspace-wide, so two
|
||||
* members reviewing at the same time race exactly as one member's two rounds would, and a key per
|
||||
* user would let each of them mint a different default domain. Personal work falls back to the
|
||||
* user. Both modes use this key so they serialize the same things.
|
||||
*/
|
||||
const catalogKey = (payload: ExpertiseRejectionWorkflowPayload) =>
|
||||
payload.workspaceId ? `workspace.${payload.workspaceId}` : `user.${payload.userId}`;
|
||||
|
||||
export class ExpertiseRejectionWorkflow {
|
||||
/**
|
||||
* Distils one settled acceptance round in the background.
|
||||
*
|
||||
* Never throws: this rides on the round-attach path, and losing a distillation is a missed
|
||||
* lesson, while failing the attach loses the reviewer's round.
|
||||
*/
|
||||
static async trigger(payload: ExpertiseRejectionWorkflowPayload) {
|
||||
try {
|
||||
if (!appEnv.enableQueueAgentRuntime) {
|
||||
const key = catalogKey(payload);
|
||||
const previous = localRuns.get(key) ?? Promise.resolve();
|
||||
// `catch` before chaining: one failed distillation must not cancel the rounds queued
|
||||
// behind it, and every branch already logs its own error.
|
||||
const current = previous
|
||||
.catch(() => undefined)
|
||||
.then(async () => {
|
||||
const db = await getServerDB();
|
||||
const result = await new ExpertiseIngestionService(
|
||||
db,
|
||||
payload.userId,
|
||||
payload.workspaceId,
|
||||
).ingestAcceptanceRound({
|
||||
acceptanceId: payload.acceptanceId,
|
||||
verifyRunId: payload.verifyRunId,
|
||||
});
|
||||
log('local ingestion for run %s: %O', payload.verifyRunId, result);
|
||||
});
|
||||
|
||||
localRuns.set(key, current);
|
||||
void current
|
||||
.catch((error) => log('failed to distil run %s: %O', payload.verifyRunId, error))
|
||||
.finally(() => localRuns.get(key) === current && localRuns.delete(key));
|
||||
return;
|
||||
}
|
||||
|
||||
const baseUrl = appEnv.INTERNAL_APP_URL || appEnv.APP_URL;
|
||||
if (!baseUrl) throw new Error('INTERNAL_APP_URL or APP_URL is required');
|
||||
await workflowClient.trigger({
|
||||
body: payload,
|
||||
// One distillation at a time per lesson catalog: two rounds settling together would race
|
||||
// on the same domain's `runIndex`, and the row lock would just serialize them anyway.
|
||||
flowControl: { key: `expertise-rejection.${catalogKey(payload)}`, parallelism: 1 },
|
||||
url: new URL('/api/workflows/expertise-rejection/run', baseUrl).toString(),
|
||||
});
|
||||
} catch (error) {
|
||||
log('failed to distil run %s: %O', payload.verifyRunId, error);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export { runExpertiseRejectionWorkflow };
|
||||
export type { ExpertiseRejectionWorkflowPayload };
|
||||
@@ -0,0 +1,24 @@
|
||||
import type { WorkflowContext } from '@upstash/workflow';
|
||||
|
||||
import { getServerDB } from '@/database/server';
|
||||
import { ExpertiseIngestionService } from '@/server/services/expertise/ingestion';
|
||||
import { runStep } from '@/server/workflows/step';
|
||||
|
||||
import type { ExpertiseRejectionWorkflowPayload } from './types';
|
||||
|
||||
export const runExpertiseRejectionWorkflow = async (
|
||||
context: WorkflowContext<ExpertiseRejectionWorkflowPayload>,
|
||||
) => {
|
||||
const payload = context.requestPayload;
|
||||
return runStep(context, `expertise-rejection:run:${payload.verifyRunId}`, async () => {
|
||||
const db = await getServerDB();
|
||||
return new ExpertiseIngestionService(
|
||||
db,
|
||||
payload.userId,
|
||||
payload.workspaceId,
|
||||
).ingestAcceptanceRound({
|
||||
acceptanceId: payload.acceptanceId,
|
||||
verifyRunId: payload.verifyRunId,
|
||||
});
|
||||
});
|
||||
};
|
||||
@@ -0,0 +1,7 @@
|
||||
export interface ExpertiseRejectionWorkflowPayload {
|
||||
acceptanceId: string;
|
||||
userId: string;
|
||||
/** The settled round whose rejections are being distilled — not the round that just landed. */
|
||||
verifyRunId: string;
|
||||
workspaceId?: string;
|
||||
}
|
||||
@@ -11,7 +11,9 @@ export const TRACING_SCENARIOS = {
|
||||
AgentWelcome: 'agent_welcome',
|
||||
BuilderSuggestion: 'builder_suggestion',
|
||||
DocumentToSkillMeta: 'document_to_skill_meta',
|
||||
ExpertiseConsolidation: 'expertise_consolidation',
|
||||
ExpertiseDomainDraft: 'expertise_domain_draft',
|
||||
ExpertiseRejectionIngestion: 'expertise_rejection_ingestion',
|
||||
ExpertiseTopicIngestion: 'expertise_topic_ingestion',
|
||||
FollowUp: 'follow_up',
|
||||
GoalCriteriaGen: 'goal_criteria_gen',
|
||||
|
||||
@@ -161,8 +161,8 @@ describe('AgentModel.transferAgent', () => {
|
||||
const agent = await agentModel.create({ title: 'Learning Agent' });
|
||||
const expertiseModel = new ExpertiseModel(serverDB, userId, wsId1);
|
||||
const domainId = await expertiseModel.createDomain({
|
||||
agentId: agent.id,
|
||||
brief: 'Improve incident response',
|
||||
carrier: { id: agent.id, type: 'agent' },
|
||||
domainFilter: 'I practice when I investigate production incidents.',
|
||||
title: 'Incident response',
|
||||
});
|
||||
|
||||
@@ -133,8 +133,8 @@ describe('ExpertiseModel', () => {
|
||||
const model = new ExpertiseModel(serverDB, userId);
|
||||
|
||||
const domainId = await model.createDomain({
|
||||
agentId: 'owned-agent',
|
||||
brief: 'Improve production incident diagnosis, excluding general design discussions.',
|
||||
carrier: { id: 'owned-agent', type: 'agent' },
|
||||
domainFilter: 'Include production incident diagnosis and remediation.',
|
||||
outOfScope: 'Exclude general design discussions without an incident.',
|
||||
title: 'Production incident response',
|
||||
|
||||
@@ -28,6 +28,30 @@ const CORE_CUT_MIN = 2;
|
||||
|
||||
export type ExpertiseTier = 'core' | 'niche' | 'unused';
|
||||
|
||||
/**
|
||||
* Where a domain is mounted. Ownership (`expertise_domains.user_id` / `workspace_id`) is a
|
||||
* separate axis: a project never owns a domain, it only mounts one, so the same standard can be
|
||||
* mounted by several projects at once.
|
||||
*/
|
||||
export type ExpertiseCarrier =
|
||||
{ id: string; type: 'agent' } | { id: string; type: 'project' } | { type: 'user' };
|
||||
|
||||
/**
|
||||
* The single carrier column a binding sets. `user` resolves to the workspace when one is in
|
||||
* scope, so a workspace member's always-on standards reach their teammates rather than only
|
||||
* themselves — the same arm `listDomainsBoundTo` reads back.
|
||||
*/
|
||||
const carrierColumns = (
|
||||
carrier: ExpertiseCarrier,
|
||||
owner: { userId: string; workspaceId?: string },
|
||||
) => {
|
||||
if (carrier.type === 'agent') return { agentId: carrier.id };
|
||||
if (carrier.type === 'project') return { projectId: carrier.id };
|
||||
return owner.workspaceId
|
||||
? { boundWorkspaceId: owner.workspaceId }
|
||||
: { boundUserId: owner.userId };
|
||||
};
|
||||
|
||||
export class ExpertiseModel {
|
||||
private db: LobeChatDatabase;
|
||||
private userId: string;
|
||||
@@ -61,6 +85,26 @@ export class ExpertiseModel {
|
||||
.limit(1);
|
||||
if (!agent) return [];
|
||||
|
||||
return this.listDomainsBoundTo(eq(expertiseBindings.agentId, agentId));
|
||||
};
|
||||
|
||||
/**
|
||||
* Lists the expertise mounted on one project, plus the caller's own always-on domains.
|
||||
*
|
||||
* Deliberately does not verify the project row: the caller reaches this through an acceptance
|
||||
* it already owns, and `scopeWhere` still confines the result to domains in scope.
|
||||
*/
|
||||
listDomainsForProject = async (projectId: string) =>
|
||||
this.listDomainsBoundTo(eq(expertiseBindings.projectId, projectId));
|
||||
|
||||
/** The caller's always-on domains only — the scope an acceptance without a project falls back to. */
|
||||
listDomainsForOwner = async () => this.listDomainsBoundTo();
|
||||
|
||||
private listDomainsBoundTo = async (carrierWhere?: ReturnType<typeof eq>) => {
|
||||
const ownerWhere = this.workspaceId
|
||||
? eq(expertiseBindings.boundWorkspaceId, this.workspaceId)
|
||||
: eq(expertiseBindings.boundUserId, this.userId);
|
||||
|
||||
const rows = await this.db
|
||||
.select({
|
||||
binding: {
|
||||
@@ -78,12 +122,7 @@ export class ExpertiseModel {
|
||||
eq(expertiseBindings.enabled, true),
|
||||
isNotNull(expertiseDomains.anchorChosenAt),
|
||||
this.scopeWhere(),
|
||||
or(
|
||||
eq(expertiseBindings.agentId, agentId),
|
||||
this.workspaceId
|
||||
? eq(expertiseBindings.boundWorkspaceId, this.workspaceId)
|
||||
: eq(expertiseBindings.boundUserId, this.userId),
|
||||
),
|
||||
carrierWhere ? or(carrierWhere, ownerWhere) : ownerWhere,
|
||||
),
|
||||
)
|
||||
.orderBy(asc(expertiseBindings.sortOrder));
|
||||
@@ -404,13 +443,13 @@ export class ExpertiseModel {
|
||||
// Writes
|
||||
|
||||
/**
|
||||
* Persists a reviewed anchor and binds it to the selected agent.
|
||||
* Persists a reviewed anchor and mounts it on the selected carrier.
|
||||
* The chosen candidate is also kept in anchorCandidates so the alternative can be revisited.
|
||||
*/
|
||||
createDomain = async (params: {
|
||||
agentId: string;
|
||||
brief: string;
|
||||
canonEntries?: ExpertiseCanonEntry[];
|
||||
carrier: ExpertiseCarrier;
|
||||
domainFilter: string;
|
||||
layerCanonRef?: string;
|
||||
layerSource?: 'canonical' | 'invented';
|
||||
@@ -459,9 +498,9 @@ export class ExpertiseModel {
|
||||
});
|
||||
await tx.insert(expertiseBindings).values({
|
||||
addedByUserId: this.userId,
|
||||
agentId: params.agentId,
|
||||
domainId: id,
|
||||
workspaceId: this.workspaceId,
|
||||
...carrierColumns(params.carrier, { userId: this.userId, workspaceId: this.workspaceId }),
|
||||
});
|
||||
});
|
||||
return id;
|
||||
|
||||
@@ -369,13 +369,14 @@ export const expertiseLessons = pgTable(
|
||||
.notNull()
|
||||
.default('compilable'),
|
||||
/**
|
||||
* Pre-compile check-up: run the lesson against deliveries the owner judged **in the past** and
|
||||
* see whether the owner actually rejected when it fired.
|
||||
* How this standard scored against deliveries the reviewer already judged.
|
||||
*
|
||||
* Only this can answer "would compiling it block things they would have passed", and that is the
|
||||
* one failure mode that makes someone turn the whole feature off. jsonb rather than a few named
|
||||
* columns: the metrics will still change (precision/fired first, possibly per-layer breakdowns
|
||||
* later), and a migration per new metric is not worth it. null = not measured yet.
|
||||
* Nothing writes it yet, deliberately. Measured on 898 circled rejections, a score against
|
||||
* those labels cannot gate compilation: a reviewer circles the worst thing in a delivery, so a
|
||||
* standard that correctly spots a defect they did not circle that time reads as a false alarm.
|
||||
* See `ExpertiseBacktestResult` for the numbers. The column stays because the measurement
|
||||
* belongs on the lesson once one with unbiased labels exists; jsonb so its shape can change
|
||||
* without a migration each time.
|
||||
*/
|
||||
backtest: jsonb('backtest').$type<ExpertiseBacktestResult>(),
|
||||
compiledCriterionId: uuid('compiled_criterion_id').references(() => verifyCriteria.id, {
|
||||
|
||||
@@ -2,9 +2,11 @@ import { describe, expect, it } from 'vitest';
|
||||
|
||||
import {
|
||||
chainExpertiseDomainDraft,
|
||||
chainExpertiseRejectionIngestion,
|
||||
chainExpertiseTopicIngestion,
|
||||
EXPERTISE_DOMAIN_DRAFT_JSON_SCHEMA,
|
||||
EXPERTISE_DOMAIN_DRAFT_PROMPT_VERSION,
|
||||
EXPERTISE_REJECTION_INGESTION_JSON_SCHEMA,
|
||||
EXPERTISE_TOPIC_INGESTION_JSON_SCHEMA,
|
||||
EXPERTISE_TOPIC_INGESTION_PROMPT_VERSION,
|
||||
} from './expertise';
|
||||
@@ -68,3 +70,92 @@ describe('chainExpertiseTopicIngestion', () => {
|
||||
expect(result.messages[1].content).toContain('[user] 帮我修改这段比喻');
|
||||
});
|
||||
});
|
||||
|
||||
describe('chainExpertiseRejectionIngestion', () => {
|
||||
it('declares no nullable unions, which the pinned model answers with `{}`', () => {
|
||||
// A live replay against gemini-3.6-flash returned `"existingLessonCode": {}` for every
|
||||
// `['string', 'null']` field under a strict json_schema, and the parse took the whole round
|
||||
// down with it. Empty string is the contract now; this test is the tripwire.
|
||||
const properties =
|
||||
EXPERTISE_REJECTION_INGESTION_JSON_SCHEMA.schema.properties.domains.items.properties
|
||||
.observations.items.properties;
|
||||
|
||||
for (const [name, definition] of Object.entries(properties)) {
|
||||
expect(`${name}:${JSON.stringify((definition as { type: unknown }).type)}`).not.toContain(
|
||||
'null',
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it('attaches the circled frames and names what it withheld', () => {
|
||||
const result = chainExpertiseRejectionIngestion({
|
||||
domains: [{ domainFilter: '交付标准', id: 'domain-1' }],
|
||||
rejections: '[R1] promised: 讨论区\n circled on frame 1 at 17%,35%: 这个顺序反了',
|
||||
visuals: [{ accessUrl: 'data:image/png;base64,AAAA', label: 'R1 (circled) — 讨论区' }],
|
||||
withheldEvidence: '2 further screenshot(s) from this round were not attached.',
|
||||
});
|
||||
|
||||
const [system, user] = result.messages;
|
||||
expect(system.content).toContain('Most of these rejections are visual');
|
||||
expect(Array.isArray(user.content)).toBe(true);
|
||||
const blocks = user.content as { image_url?: { url: string }; text?: string; type: string }[];
|
||||
expect(blocks.filter((block) => block.type === 'image_url')).toHaveLength(1);
|
||||
expect(blocks[0].text).toContain('[frame 1] R1 (circled)');
|
||||
// Without this line the model reads a missing frame as proof that nothing was wrong there.
|
||||
expect(blocks[0].text).toContain('WITHHELD FROM THIS REQUEST');
|
||||
});
|
||||
|
||||
it('falls back to a plain text message when no frame could be resolved', () => {
|
||||
const result = chainExpertiseRejectionIngestion({
|
||||
domains: [],
|
||||
rejections: '[R1] promised: 后端接口\n said: 这个字段不对',
|
||||
});
|
||||
|
||||
expect(typeof result.messages[1].content).toBe('string');
|
||||
expect(result.messages[1].content).toContain('(none)');
|
||||
});
|
||||
});
|
||||
|
||||
describe('chainExpertiseRejectionIngestion reason provenance', () => {
|
||||
it('lets a standard admit it is taste instead of fabricating a mechanism', () => {
|
||||
// Not every standard has a mechanism under it. Forced to find one, the model returns a synonym
|
||||
// of the objection — "visual noise" for "untidy" — which reads as objective and would be
|
||||
// enforced as if it were. Turning that failure into the detector split the same replayed round
|
||||
// into 4 mechanism / 3 taste, and the 3 are exactly the ones that used to fake a mechanism.
|
||||
const system = chainExpertiseRejectionIngestion({ domains: [], rejections: '' }).messages[0]
|
||||
.content as string;
|
||||
|
||||
expect(system).toContain('Use your own failure as the detector');
|
||||
expect(system).toContain('A fabricated mechanism is worse than an admitted preference');
|
||||
// A taste reason still has to tell the next delivery what to do instead.
|
||||
expect(system).toContain(
|
||||
'State the preference plainly and in a form the next delivery can act on',
|
||||
);
|
||||
});
|
||||
|
||||
it('judges a reason by whether it can settle a case the round never described', () => {
|
||||
// "Harms visual tidiness" is the objection in nicer words — tidiness is the judgement under
|
||||
// review, so it settles nothing. Naming the transfer test moved 5 of 7 reasons from verdict
|
||||
// to mechanism on replay ("must click to enlarge", "retries a network error that never was").
|
||||
const system = chainExpertiseRejectionIngestion({ domains: [], rejections: '' }).messages[0]
|
||||
.content as string;
|
||||
|
||||
expect(system).toContain('could someone facing a situation this round never described settle');
|
||||
expect(system).toContain('verdicts wearing a reason');
|
||||
expect(system).toContain('which property of the delivery causes what concrete consequence');
|
||||
});
|
||||
|
||||
it('gives reasonSource a test the model can run, not a judgement call', () => {
|
||||
// Asking for "did the reviewer say why" produced 3 of 7 marked `reviewer` on rejections that
|
||||
// were pure instructions ("put it in one row", "there's an extra line here"). Naming the
|
||||
// observable — a stated consequence or cause — fixed all three, so the wording is the fix.
|
||||
const system = chainExpertiseRejectionIngestion({ domains: [], rejections: '' }).messages[0]
|
||||
.content as string;
|
||||
|
||||
expect(system).toContain("does the reviewer's own text state a consequence or a cause");
|
||||
expect(system).toContain('When in doubt answer "inferred"');
|
||||
// The mechanism must still be written even when the reviewer only pointed — a reason-free
|
||||
// lesson cannot transfer to a screen nobody has built yet.
|
||||
expect(system).toContain('Write the mechanism even when the reviewer only pointed');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -187,3 +187,293 @@ export const chainExpertiseTopicIngestion = (input: {
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
export const EXPERTISE_REJECTION_INGESTION_PROMPT_VERSION = 'v3';
|
||||
|
||||
export const EXPERTISE_REJECTION_INGESTION_JSON_SCHEMA = {
|
||||
name: 'expertise_rejection_ingestion',
|
||||
schema: {
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
domains: {
|
||||
items: {
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
domainId: { type: 'string' },
|
||||
matches: { type: 'boolean' },
|
||||
observations: {
|
||||
items: {
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
example: { type: 'string' },
|
||||
// Plain strings, not `['string', 'null']` unions: under a strict json_schema the
|
||||
// pinned model answers a nullable union with `{}`, which fails the parse and
|
||||
// loses the whole round. An empty string is the "none" the service reads.
|
||||
existingLessonCode: { type: 'string' },
|
||||
layer: { type: 'string' },
|
||||
limits: { type: 'string' },
|
||||
// Whether the standard rests on something observable or on the owner's
|
||||
// preference. Orthogonal to `reasonSource` (who supplied the reason) and load
|
||||
// bearing downstream: a taste standard has no objective test, so it must not be
|
||||
// compiled into a criterion that blocks a delivery on its own.
|
||||
reasonKind: { enum: ['mechanism', 'taste'], type: 'string' },
|
||||
reasoning: { type: 'string' },
|
||||
// Self-classified provenance. The model may explain the mechanism — that is what
|
||||
// makes a standard transferable — but an explanation it invented must never read
|
||||
// as something the reviewer stated.
|
||||
reasonSource: { enum: ['reviewer', 'inferred'], type: 'string' },
|
||||
sourceRefs: { items: { type: 'string' }, minItems: 1, type: 'array' },
|
||||
// Naming the abstracted subject is what forces the climb: a model that cannot
|
||||
// say what the concrete thing is an example of has not generalized at all.
|
||||
subject: { type: 'string' },
|
||||
title: { type: 'string' },
|
||||
},
|
||||
required: [
|
||||
'example',
|
||||
'existingLessonCode',
|
||||
'layer',
|
||||
'limits',
|
||||
'reasonKind',
|
||||
'reasonSource',
|
||||
'reasoning',
|
||||
'sourceRefs',
|
||||
'subject',
|
||||
'title',
|
||||
],
|
||||
type: 'object',
|
||||
},
|
||||
maxItems: 8,
|
||||
type: 'array',
|
||||
},
|
||||
},
|
||||
required: ['domainId', 'matches', 'observations'],
|
||||
type: 'object',
|
||||
},
|
||||
type: 'array',
|
||||
},
|
||||
},
|
||||
required: ['domains'],
|
||||
type: 'object',
|
||||
},
|
||||
} as const satisfies ExpertiseGenerateObjectSchema;
|
||||
|
||||
const EXPERTISE_REJECTION_INGESTION_SYSTEM_PROMPT = `You maintain a reviewer's delivery standards from the checks they rejected.
|
||||
|
||||
Every entry below is one rejected acceptance check: what was promised, what the reviewer said was wrong with it, and — when they circled a region on a screenshot — the screenshot itself. The reviewer's own words are the evidence; never soften, reinterpret, or argue with them.
|
||||
|
||||
Most of these rejections are visual. "This isn't aligned", "this is too big", "the colour is too heavy" cannot be understood from text alone: read the attached frame, find the circled region, and describe what is actually wrong there. A standard written without looking at the frame is a paraphrase of a complaint, not a standard.
|
||||
|
||||
First apply each domainFilter and outOfScope literally. If none of the rejections fall inside a domain, return matches=false and no observations for it. One rejection may legitimately land in several domains.
|
||||
|
||||
Then generalize, and check how far you got. A rejection is an instance; a lesson is the standard behind it. Restating one rejection verbatim produces a rule that never fires again, which is the failure mode this whole pipeline exists to avoid — but stripping the screen name out of a sentence is not generalizing either.
|
||||
|
||||
Apply this test to every title before you return it: could a delivery on a completely unrelated screen violate this standard? If the answer is no, you are still describing the instance. Climb one level: replace the concrete things with what they are an example of, and say it in \`subject\` — "the reaction buttons under a comment" is really "a set of results plus a control that adds to it"; "the annotation border on a screenshot" is really "a thin line drawn over user content".
|
||||
|
||||
Do not climb past what the reviewer actually objected to. If they rejected one decorative divider, the standard is about decoration that was not asked for — not about dividers in text inputs, and not about every visual element on the page.
|
||||
|
||||
Attaching to an existing lesson is the default; a new lesson is the exception:
|
||||
|
||||
- Attach whenever a listed lesson already carries the same standard, even when this rejection words it differently or hits another screen. Put that lesson's code in existingLessonCode, copied character for character from its \`code\` field (for example "P-07").
|
||||
- existingLessonCode holds a lesson code and nothing else. Never put a check title, a file path, or any identifier taken from the rejections there — those all read as "no existing lesson" and silently fork a duplicate.
|
||||
- Only when no listed lesson carries the standard, set existingLessonCode to "" and propose one. Before doing so, state to yourself what it adds that every listed lesson misses; if you cannot, attach instead.
|
||||
- Prefer one lesson supported by several rejections over several near-identical lessons. Cite every rejection that supports it in sourceRefs.
|
||||
|
||||
For each observation return:
|
||||
- title — the standard as one imperative sentence, stated about \`subject\` rather than about the screen it happened on;
|
||||
- subject — what the standard is really about, once the concrete names are replaced by what they exemplify;
|
||||
- reasoning — the MECHANISM: which property of the delivery causes what concrete consequence, and for whom. Name both halves. "A pale line has too little luminance difference from a white background, so the reader cannot tell which region was circled" names both; "the reviewer said the cyan is too light" is the evidence, not a reason, and it already lives elsewhere.
|
||||
|
||||
The mechanism is what lets a standard transfer to a screen nobody has built yet, so judge your own sentence by exactly that: could someone facing a situation this round never described settle it using your reason alone? If not, you have not written a mechanism. Restating the objection in more flattering words always fails this test — "harms visual tidiness", "feels inelegant", "hurts consistency", "adds cognitive load" are verdicts wearing a reason's clothes, because tidiness and elegance are the very judgement in question. Replace every such phrase with the observable consequence underneath it: what does a person fail to see, misread, mis-click, or have to do twice?
|
||||
|
||||
Write the mechanism even when the reviewer only pointed at something — that is what reasonSource is for;
|
||||
- reasonKind — "mechanism" when the reason above survives that transfer test, "taste" when it does not. Not every standard has a mechanism underneath it, and that is legitimate: some of what a reviewer requires is simply what they prefer. Use your own failure as the detector — if the only reason you can produce is a synonym of the objection ("untidy", "visual noise", "inelegant", "not clean"), stop trying and answer "taste".
|
||||
|
||||
For "taste", do not dress the verdict up. State the preference plainly and in a form the next delivery can act on: "the owner does not accept dividers that were not asked for; regions are separated by spacing and container edges alone" is a complete and honest reason. A fabricated mechanism is worse than an admitted preference, because it reads as objective and gets enforced as if it were;
|
||||
- reasonSource — where that reason came from, decided by one test you can actually run: does the reviewer's own text state a consequence or a cause, not just an instruction? "the cyan is too light, I can't see it" states a consequence → "reviewer". "put it in one row, annotations left, actions right" and "there's an extra line here" are instructions with no cause → "inferred", however obvious the cause seems. So is "this is ugly" / "this is wrong" / "this doesn't work". When in doubt answer "inferred": over-claiming the reviewer said something is the one failure this field exists to prevent, and under-claiming costs nothing;
|
||||
- example — how it showed up this time, concretely enough to recognise again;
|
||||
- limits — the boundary THE REVIEWER drew. Fill it only when they said where the standard stops, or when another rejection in this same round contradicts it. Otherwise answer exactly "边界未由评审者说明" (or the same sentence in their language). An invented exemption is worse than an empty one: it silently narrows a standard the reviewer stated without limit;
|
||||
- sourceRefs — the reference labels (for example "R2") of every rejection supporting it. Never invent a label that is not listed.
|
||||
|
||||
Leave existingLessonCode and layer as an empty string rather than null when they do not apply — never as an object. Use only declared layer keys. Write human-facing text in the language the reviewer used.`;
|
||||
|
||||
export const chainExpertiseRejectionIngestion = (input: {
|
||||
domains: readonly unknown[];
|
||||
rejections: string;
|
||||
/** Frames the reviewer circled, labelled so the text can point at them. */
|
||||
visuals?: { accessUrl: string; label: string }[];
|
||||
/** Frames left out of this request, stated so the model does not read their absence as proof. */
|
||||
withheldEvidence?: string;
|
||||
}): { messages: OpenAIChatMessage[] } => {
|
||||
const visuals = input.visuals ?? [];
|
||||
const frameList = visuals.length
|
||||
? visuals.map((visual, index) => ` [frame ${index + 1}] ${visual.label}`).join('\n')
|
||||
: ' (none)';
|
||||
|
||||
const text = [
|
||||
`DOMAINS\n${JSON.stringify(input.domains)}`,
|
||||
`\nREJECTED CHECKS (one acceptance round)\n${input.rejections}`,
|
||||
`\nATTACHED FRAMES (in order)\n${frameList}`,
|
||||
input.withheldEvidence ? `\nWITHHELD FROM THIS REQUEST\n${input.withheldEvidence}` : '',
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join('\n');
|
||||
|
||||
return {
|
||||
messages: [
|
||||
{ content: EXPERTISE_REJECTION_INGESTION_SYSTEM_PROMPT, role: 'system' },
|
||||
{
|
||||
content: visuals.length
|
||||
? [
|
||||
{ text, type: 'text' as const },
|
||||
...visuals.map((visual) => ({
|
||||
image_url: { detail: 'high' as const, url: visual.accessUrl },
|
||||
type: 'image_url' as const,
|
||||
})),
|
||||
]
|
||||
: text,
|
||||
role: 'user',
|
||||
},
|
||||
],
|
||||
};
|
||||
};
|
||||
|
||||
export const EXPERTISE_CONSOLIDATION_PROMPT_VERSION = 'v2';
|
||||
|
||||
export const EXPERTISE_CONSOLIDATION_JSON_SCHEMA = {
|
||||
name: 'expertise_consolidation',
|
||||
schema: {
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
// Whether the standard's current `limits` is ingestion's "the reviewer stated no boundary"
|
||||
// placeholder rather than a boundary. The service keeps existing limits verbatim, so this is
|
||||
// the one thing it cannot decide for itself: only a reader of that text knows which it is.
|
||||
currentLimitsArePlaceholder: { type: 'boolean' },
|
||||
// `false` is a real answer, not a failure: instances that only share a category do not
|
||||
// share a standard, and rewriting them into one invents a rule nobody stated.
|
||||
generalized: { type: 'boolean' },
|
||||
// Structured rather than one `limits` sentence, because a boundary is only legitimate if it
|
||||
// can name what it rests on. The service resolves every ref to a check result id and drops
|
||||
// any entry that resolves to nothing — which turns "never invent an exemption" from a request
|
||||
// into something enforced.
|
||||
// Additions only. The limits the standard already carries are never echoed back and never
|
||||
// re-derived from the answer — the service keeps them verbatim, because a model that forgets
|
||||
// to repeat one would silently widen a standard the reviewer had bounded.
|
||||
limits: {
|
||||
items: {
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
shippedRefs: { items: { type: 'string' }, type: 'array' },
|
||||
text: { type: 'string' },
|
||||
},
|
||||
required: ['shippedRefs', 'text'],
|
||||
type: 'object',
|
||||
},
|
||||
type: 'array',
|
||||
},
|
||||
// Plain strings rather than nullable unions — the pinned model answers a nullable under a
|
||||
// strict schema with `{}`, which fails the parse and loses the pass.
|
||||
note: { type: 'string' },
|
||||
reasonKind: { enum: ['mechanism', 'taste'], type: 'string' },
|
||||
reasoning: { type: 'string' },
|
||||
subject: { type: 'string' },
|
||||
title: { type: 'string' },
|
||||
},
|
||||
required: [
|
||||
'currentLimitsArePlaceholder',
|
||||
'generalized',
|
||||
'limits',
|
||||
'note',
|
||||
'reasonKind',
|
||||
'reasoning',
|
||||
'subject',
|
||||
'title',
|
||||
],
|
||||
type: 'object',
|
||||
},
|
||||
} as const satisfies ExpertiseGenerateObjectSchema;
|
||||
|
||||
const EXPERTISE_CONSOLIDATION_SYSTEM_PROMPT = `You are rewriting one delivery standard now that the reviewer has rejected several deliveries for it.
|
||||
|
||||
The standard in front of you was written from a single rejection, so it is worded at the level of the screen that rejection happened on. Every instance since then attached to it because it said the same thing. Your job is to restate it at the level all of those instances actually share — and to give it the boundary it has never had.
|
||||
|
||||
INSTANCES are the deliveries the reviewer rejected under this standard: what was promised, what they said was wrong, and the frame they circled. SHIPPED are deliveries the same reviewer accepted, with their frames. Read the frames; a standard rewritten from the text alone is a summary of complaints, not a standard.
|
||||
|
||||
## Restating it
|
||||
|
||||
State the standard so that every instance is an example of it, and nothing is smuggled in that only one instance supports. Two failures to avoid, in both directions:
|
||||
|
||||
- Too low: the sentence still names a screen, a component or a feature from one instance. A standard that only fires again on that screen is the instance wearing a rule's clothes.
|
||||
- Too high: the sentence would also condemn deliveries the reviewer never objected to. Each climb has to be paid for by an instance — if only one instance supports the wider wording, the wider wording is yours, not theirs.
|
||||
|
||||
If the instances do not share a standard, answer \`generalized\`: false and return the current wording unchanged. This is the common case for placement and ordering complaints: "put the status next to the title" and "move topics above the profile" are both about position and are not the same rule. Say so in \`note\` and stop. A rewrite that unites unrelated instances is worse than no rewrite, because it starts firing on everything.
|
||||
|
||||
## The boundary
|
||||
|
||||
\`limits\` is the one thing you could not write from a single rejection, and SHIPPED is what makes it writable now.
|
||||
|
||||
Look for a shipped delivery whose frame plainly shows this standard's subject, in the state the standard objects to — and which the reviewer accepted anyway. That is a boundary the reviewer drew with their own hands: the standard stops somewhere before that delivery. Write it as where the standard stops, pointing at what is different about the case they let through.
|
||||
|
||||
Return \`limits\` as the exemptions you are ADDING. Each entry:
|
||||
- \`text\` — where the standard stops, in the reviewer's language.
|
||||
- \`shippedRefs\` — the labels of the SHIPPED deliveries it was read from (for example ["S2"]). Every entry must name at least one. An entry that names none, or names an instance, is discarded — and so is any label that is not listed.
|
||||
|
||||
Do not repeat the limits the standard already carries; they are kept for you, word for word, and nothing you write can remove one. Set \`currentLimitsArePlaceholder\` to true only when the standard's current \`limits\` is ingestion's "边界未由评审者说明" placeholder (in any language) rather than a boundary the reviewer drew — that is the one case where the existing text is dropped, and only if you supply a real exemption to replace it.
|
||||
|
||||
Rules on this, in order:
|
||||
- Only a shipped delivery whose frame you have actually read can justify a new exemption. Never infer one from the instances, from the standard's own wording, or from what a reasonable person "would obviously" exempt — an invented exemption silently narrows a standard the reviewer stated without limit.
|
||||
- A limit the reviewer already stated stays. Dropping it widens the standard past what they said, which is the same failure in the other direction.
|
||||
- If the shipped frames do not show the subject at all, you have learned nothing about the boundary: return only the limits you are carrying over (often none), and do not apologise for it in \`note\`.
|
||||
- If a shipped delivery shows the subject in the objectionable state and you cannot tell what makes it different, say that in \`note\` and add no entry for it. "I cannot see the distinction" is information; a guessed distinction is not.
|
||||
|
||||
## The rest
|
||||
|
||||
- \`subject\` — what the standard is about once the concrete names are replaced by what they exemplify. It has to cover every instance.
|
||||
- \`reasoning\` — the MECHANISM: which property of the delivery causes what concrete consequence, and for whom. Judge your own sentence by whether someone facing a case none of these instances describe could settle it using your reason alone. "Harms tidiness", "adds visual noise", "feels inconsistent" are verdicts wearing a reason's clothes — the tidiness is the very judgement in question. Replace them with what a person fails to see, misreads, mis-clicks, or has to do twice.
|
||||
- \`reasonKind\` — "mechanism" if that reason survives the transfer test, "taste" if the only reason you can produce is a synonym of the objection. Several instances make this easier to settle honestly, not harder: if the reviewer has rejected the same thing four times and you still cannot name a consequence, it is taste, and saying so plainly is the useful answer.
|
||||
- \`note\` — one sentence for the reviewer on what changed and what carried it, naming the instances. This is read by a person deciding whether to trust the rewrite.
|
||||
|
||||
Write human-facing text in the language the reviewer used.`;
|
||||
|
||||
export const chainExpertiseConsolidation = (input: {
|
||||
/** The deliveries rejected under this standard, each labelled with the frame it circled. */
|
||||
instances: string;
|
||||
lesson: string;
|
||||
/** Deliveries the reviewer accepted — the only evidence that can justify a limit. */
|
||||
shipped: string;
|
||||
visuals?: { accessUrl: string; label: string }[];
|
||||
/** Frames left out, stated so their absence does not read as proof the subject is not there. */
|
||||
withheldEvidence?: string;
|
||||
}): { messages: OpenAIChatMessage[] } => {
|
||||
const visuals = input.visuals ?? [];
|
||||
const frameList = visuals.length
|
||||
? visuals.map((visual, index) => ` [frame ${index + 1}] ${visual.label}`).join('\n')
|
||||
: ' (none)';
|
||||
|
||||
const text = [
|
||||
`STANDARD AS IT STANDS\n${input.lesson}`,
|
||||
`\nINSTANCES (rejected under it)\n${input.instances}`,
|
||||
`\nSHIPPED (accepted by the same reviewer)\n${input.shipped || ' (none available)'}`,
|
||||
`\nATTACHED FRAMES (in order)\n${frameList}`,
|
||||
input.withheldEvidence ? `\nWITHHELD FROM THIS REQUEST\n${input.withheldEvidence}` : '',
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join('\n');
|
||||
|
||||
return {
|
||||
messages: [
|
||||
{ content: EXPERTISE_CONSOLIDATION_SYSTEM_PROMPT, role: 'system' },
|
||||
{
|
||||
content: visuals.length
|
||||
? [
|
||||
{ text, type: 'text' as const },
|
||||
...visuals.map((visual) => ({
|
||||
image_url: { detail: 'high' as const, url: visual.accessUrl },
|
||||
type: 'image_url' as const,
|
||||
})),
|
||||
]
|
||||
: text,
|
||||
role: 'user',
|
||||
},
|
||||
],
|
||||
};
|
||||
};
|
||||
|
||||
@@ -150,17 +150,20 @@ export type ExpertiseReasonSource = 'inferred' | 'reviewer';
|
||||
export type ExpertiseBacktestVerdict = 'ready' | 'too-broad' | 'insufficient-evidence';
|
||||
|
||||
/**
|
||||
* How a distilled standard scored against the reviewer's own past decisions, before anything
|
||||
* compiles it into a criterion that can block a delivery.
|
||||
* How a distilled standard scored against the reviewer's own past decisions.
|
||||
*
|
||||
* The metric is precision on the units it fired on, deliberately not recall. Every standard covers
|
||||
* a narrow slice, so recall over the whole corpus is near zero for all of them and answers
|
||||
* nothing. What decides whether compiling is safe is the opposite question: when it fires, was the
|
||||
* reviewer actually going to reject? A standard that fires on deliveries they approved blocks work
|
||||
* they would have shipped — the failure that makes someone switch the feature off.
|
||||
* Read the numbers, not the name: measuring this on 898 of one reviewer's circled rejections showed
|
||||
* the score cannot be used as a gate, because the labels it scores against are incomplete. A
|
||||
* reviewer circles the one or two worst things in a delivery, so a standard that correctly spots a
|
||||
* real defect they did not circle that time is counted a false alarm. Concretely: standards fired on
|
||||
* 25 of 44 rejections from the same category, but matched what the reviewer had actually written on
|
||||
* 4 — while firing on 39% of deliveries they accepted. Neither number separates a good standard
|
||||
* from a bad one.
|
||||
*
|
||||
* Read `precision` against the corpus base rate, not against 1.0: roughly 45% of this owner's
|
||||
* judged units are rejections, so a standard firing at random already scores ~0.45.
|
||||
* So nothing writes this column yet, and nothing should gate on `verdict` until a measurement with
|
||||
* unbiased labels exists — the candidate being forward measurement, where a standard is injected
|
||||
* without blocking and what is counted is how often the reviewer overrides it once it has fired.
|
||||
* Kept as a column because that measurement still belongs on the lesson.
|
||||
*/
|
||||
export interface ExpertiseBacktestResult {
|
||||
/** ISO 8601. */
|
||||
|
||||
Reference in New Issue
Block a user