From 44631fc328d79095f26df8b31ec541b2ceb99dc9 Mon Sep 17 00:00:00 2001 From: Benjamin Taylor Date: Thu, 10 Sep 2026 15:09:32 -0500 Subject: [PATCH] test(skills): guard the repository facts the procedure skills name MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two entry-point skills point at documentation, so the existing guard checks that every docs path they name resolves. The Inspector and Intelligence skills point at the repository instead — an Nx target, a dev-server port, a lab scenario id, a landing-page source file, a Callout snippet — and nothing checked those. They had already drifted. `465abb0239` (OSS-948) deleted `open-inspector-step-angular.mdx`, saying so plainly in its message: the Angular snippet "is deleted in favour of the shared one every other web frontend already uses", because `@copilotkit/angular@0.4.0` auto-mounts a pinned web-inspector and there is no install step left to link. That commit never touched `pane-map.md`, so the map kept pointing at the deleted file and `inspector-docs` kept telling agents Angular uses its own snippet. Both are corrected here. Four assertions, all with unambiguous ground truth: - every `open-inspector-*.mdx` the pane map names exists - every `docs/…mdx` page the pane map names exists - every `showcase/shell-docs/src/…` file `intelligence-docs` says to edit in the same change exists - the Nx target, port, scenario ids and query keys `inspector-workbench` tells an agent to run resolve against `project.json` and the lab Each was mutation-checked: breaking the intelligence path, the port, the scenario, the target, and a docs page reference each fails exactly one test, and the restored tree passes. Deliberately not asserted: which panes the Inspector ships. Panes are not enumerated as data in `packages/web-inspector`, whose entry point is a single fourteen-thousand-line module, so matching a pane label against source proves nothing in either direction. A first draft tried it and passed while the map was in fact wrong — "Pop-out window" is listed unshipped although `src/lib/pop-out.ts` is imported by the package entry and has its own suite — which is worse than no test, because green reads as confirmation. That direction needs a pane registry in the package, not a cleverer regex here; the comment in the suite says so, and the Pop-out row is left for the Inspector owner to rule on rather than guessed at. This is the cheap half of testing a skill: whether it is still true. Whether it measurably helps an agent is the other half, and that needs the lift eval in #5689. Co-Authored-By: Claude Opus 5 --- scripts/__tests__/public-skill-drift.test.ts | 114 +++++++++++++++++++ skills/inspector-docs/SKILL.md | 2 +- skills/inspector-docs/references/pane-map.md | 28 ++--- 3 files changed, 129 insertions(+), 15 deletions(-) diff --git a/scripts/__tests__/public-skill-drift.test.ts b/scripts/__tests__/public-skill-drift.test.ts index 2ad446ad51..0252255390 100644 --- a/scripts/__tests__/public-skill-drift.test.ts +++ b/scripts/__tests__/public-skill-drift.test.ts @@ -107,3 +107,117 @@ describe("packaged skills point at pages that exist", () => { expect(inventories).toEqual(["skills/copilotkit-channels/sources.md"]); }); }); + +/** + * The procedure skills rot differently from the two entry points. + * + * `copilotkit` and `copilotkit-cli` point at documentation, so their failure + * mode is a dead docs path — guarded above. The Inspector and Intelligence + * skills instead name concrete repository facts: an Nx target, a dev-server + * port, a lab scenario id, a landing-page source file, a Callout snippet. + * Every one of those is a claim that can quietly stop being true, and when it + * does the skill sends an agent somewhere that no longer exists. + * + * Only claims with an unambiguous ground truth are asserted. Which panes the + * Inspector *ships* is deliberately not asserted: panes are not enumerated as + * data anywhere in `packages/web-inspector`, whose entry point is one + * fourteen-thousand-line module, so matching a pane label against source + * proves nothing in either direction. A first draft of this suite tried it and + * passed while the map really was wrong — "Pop-out window" is listed unshipped + * although `src/lib/pop-out.ts` is imported by the package entry — which is + * worse than not testing it, because a green run reads as confirmation. + * Making that direction testable needs a pane registry in the package, not a + * cleverer regex here. + */ +describe("procedure skills name repository facts that still hold", () => { + /** Reads a skill file, or fails loudly rather than silently passing. */ + function skill(path: string): string { + const full = resolve(skillsDir, path); + if (!existsSync(full)) throw new Error(`missing skill file: ${path}`); + return readFileSync(full, "utf8"); + } + + it("maps every Inspector pane to a Callout snippet that exists", () => { + const map = skill("inspector-docs/references/pane-map.md"); + // Snippet cells name bare `open-inspector-*.mdx` files, all in one + // directory. Anchored on that prefix so the `docs/…mdx` page references in + // the surfaces table below are not mistaken for snippets. + const named = [ + ...new Set( + [...map.matchAll(/(? m[1], + ), + ), + ]; + expect(named.length).toBeGreaterThan(0); + const dir = resolve(contentDir, "snippets/shared/inspector"); + expect(named.filter((f) => !existsSync(join(dir, f)))).toEqual([]); + }); + + it("maps every Inspector pane to docs pages that exist", () => { + const map = skill("inspector-docs/references/pane-map.md"); + const pages = [ + ...new Set( + [...map.matchAll(/`(docs\/[A-Za-z0-9_./-]+\.mdx)`/g)].map((m) => m[1]), + ), + ]; + expect(pages.length).toBeGreaterThan(0); + expect(pages.filter((f) => !existsSync(resolve(contentDir, f)))).toEqual( + [], + ); + }); + + it("keeps the Intelligence landing sources it tells you to edit", () => { + const md = skill("intelligence-docs/SKILL.md"); + const paths = [ + ...new Set( + [...md.matchAll(/`(showcase\/shell-docs\/src\/[A-Za-z0-9_./-]+)`/g)] + .map((m) => m[1]) + // Trailing-slash entries are directory prose, not editable files. + .filter((f) => !f.endsWith("/")), + ), + ]; + expect(paths.length).toBeGreaterThan(0); + expect(paths.filter((f) => !existsSync(resolve(root, f)))).toEqual([]); + }); + + it("names an Inspector workbench target, port and scenario that exist", () => { + const md = skill("inspector-workbench/SKILL.md"); + const project = JSON.parse( + readFileSync( + resolve(root, "packages/web-inspector/project.json"), + "utf8", + ), + ) as { targets?: Record }; + + // The Nx target the skill tells the agent to run. + const target = md.match(/nx run @copilotkit\/web-inspector:([\w:-]+)/)?.[1]; + expect(target, "skill names an nx target").toBeDefined(); + expect(Object.keys(project.targets ?? {})).toContain(target); + + // The port it tells the agent to open, which that target has to pin. + const port = md.match(/127\.0\.0\.1:(\d+)/)?.[1]; + expect(port, "skill names a port").toBeDefined(); + expect(JSON.stringify(project.targets?.[target as string])).toContain( + `--port ${port}`, + ); + + // The lab scenarios and query keys it tells the agent to load. + const lab = readFileSync( + resolve(root, "packages/web-inspector/dev/threads-state-lab.ts"), + "utf8", + ); + const scenarios = [ + ...new Set( + [...md.matchAll(/[?&]scenario=([a-z0-9-]+)/g)].map((m) => m[1]), + ), + ]; + expect(scenarios.length).toBeGreaterThan(0); + expect(scenarios.filter((s) => !lab.includes(s))).toEqual([]); + for (const key of [ + ...new Set([...md.matchAll(/[?&]([a-z-]+)=1\b/g)].map((m) => m[1])), + ]) { + expect(lab, `lab supports ?${key}=`).toContain(key); + } + }); +}); diff --git a/skills/inspector-docs/SKILL.md b/skills/inspector-docs/SKILL.md index 234d3427f6..4d1e94b8e5 100644 --- a/skills/inspector-docs/SKILL.md +++ b/skills/inspector-docs/SKILL.md @@ -55,7 +55,7 @@ The default web quickstart includes a numbered step after the first chat: 2. Send a chat message. **Agents** then **AG-UI Events**: events are moving. 3. **Threads**: unlocked, or locked with Enable Intelligence. -Angular uses the Angular step snippet, which links the Angular Inspector install page first. React Native and Channels do not get this step. +Angular uses the shared step snippet, like every other web frontend: `@copilotkit/angular` auto-mounts a pinned `@copilotkit/web-inspector`, so there is no install step to link (OSS-948). React Native and Channels do not get this step at all. ## Decision Tree diff --git a/skills/inspector-docs/references/pane-map.md b/skills/inspector-docs/references/pane-map.md index 17d7ea8065..247dc1035c 100644 --- a/skills/inspector-docs/references/pane-map.md +++ b/skills/inspector-docs/references/pane-map.md @@ -3,20 +3,20 @@ Source of truth for which shipped Inspector pane has a docs Callout. Update this file in the same change that adds or removes a pane. -| Shipped pane | Docs page | Callout snippet | Notes | -| ----------------- | --------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------- | -| Agent | Default web quickstart step | `snippets/shared/inspector/open-inspector-step.mdx` | First check in the quickstart step | -| AG-UI Events | Default web quickstart step | `snippets/shared/inspector/open-inspector-step.mdx` | Second check, after a chat message | -| Threads | Default web quickstart step, Threads overview | `open-inspector-step.mdx`, `open-inspector-pane-threads.mdx` | Unlocked or Enable Intelligence both count. Callout also names **Try from here**. | -| Playground | no page yet | | Workbench leaf. Threads **Try from here** copies into this scratch session. | -| Try from here | Threads overview | `open-inspector-pane-threads.mdx` | Overlay action on a real stored thread detail header. | -| Frontend Tools | Frontend tools, human-in-the-loop overview | `open-inspector-pane-frontend-tools.mdx` | HITL tools appear here when registered | -| State | Shared state | `open-inspector-pane-state.mdx` | Thread detail tab | -| Context | `useAgentContext` / agent-readonly | `open-inspector-pane-context.mdx` | Agents group | -| Learning | CopilotKit Intelligence overview | `open-inspector-pane-learning.mdx` | Primary nav; published Skills, supporting Insights, and Thread evidence | -| Capabilities | no page yet | | Client tool and catalog toggles. No dedicated docs page in this slice | -| Messages | no page yet | | Thread detail tab. Covered by Threads Callout | -| Angular Inspector | Angular frontend getting-started | `open-inspector-step-angular.mdx` | Install page first | +| Shipped pane | Docs page | Callout snippet | Notes | +| ----------------- | --------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | +| Agent | Default web quickstart step | `snippets/shared/inspector/open-inspector-step.mdx` | First check in the quickstart step | +| AG-UI Events | Default web quickstart step | `snippets/shared/inspector/open-inspector-step.mdx` | Second check, after a chat message | +| Threads | Default web quickstart step, Threads overview | `open-inspector-step.mdx`, `open-inspector-pane-threads.mdx` | Unlocked or Enable Intelligence both count. Callout also names **Try from here**. | +| Playground | no page yet | | Workbench leaf. Threads **Try from here** copies into this scratch session. | +| Try from here | Threads overview | `open-inspector-pane-threads.mdx` | Overlay action on a real stored thread detail header. | +| Frontend Tools | Frontend tools, human-in-the-loop overview | `open-inspector-pane-frontend-tools.mdx` | HITL tools appear here when registered | +| State | Shared state | `open-inspector-pane-state.mdx` | Thread detail tab | +| Context | `useAgentContext` / agent-readonly | `open-inspector-pane-context.mdx` | Agents group | +| Learning | CopilotKit Intelligence overview | `open-inspector-pane-learning.mdx` | Primary nav; published Skills, supporting Insights, and Thread evidence | +| Capabilities | no page yet | | Client tool and catalog toggles. No dedicated docs page in this slice | +| Messages | no page yet | | Thread detail tab. Covered by Threads Callout | +| Angular Inspector | Angular frontend getting-started | `open-inspector-step.mdx` | Uses the shared step. `@copilotkit/angular` auto-mounts a pinned web-inspector, so there is nothing to install (OSS-948) | ## Unshipped (no Callout)