From fae400eddffb18a0eef20c8cb667d40a39983f5a Mon Sep 17 00:00:00 2001 From: Lukas Moschitz Date: Tue, 8 Sep 2026 10:28:22 +0200 Subject: [PATCH] chore(showcase/claude-sdk-python): carry the byoc-to-declarative QA rename further Work in progress on top of the OSS-578 parity branch, committed to preserve it outside the worktree rather than to ship it. Retires the byoc-* QA docs and e2e specs and adds declarative-* replacements, plus new QA docs and specs for a2ui-recovery, a2ui-fixed-schema, beautiful-chat, gen-ui-interrupt, interrupt-headless, mcp-apps, reasoning-default, reasoning-custom, threadid-frontend-tool-roundtrip and tool-rendering-reasoning-chain. declarative-gen-ui.spec.ts is largely rewritten. Also touches agent_server.py, readonly_state_agent_context.py, the copilotkit route and PARITY_NOTES.md. Unreviewed and unverified. PR #6235 is untouched: this goes to its own branch because the local branch and the PR branch have diverged. --- .../claude-sdk-python/PARITY_NOTES.md | 152 +++++++++-- .../claude-sdk-python/qa/a2ui-fixed-schema.md | 57 ++++ .../claude-sdk-python/qa/a2ui-recovery.md | 47 ++++ .../claude-sdk-python/qa/beautiful-chat.md | 93 +++++++ .../claude-sdk-python/qa/byoc-hashbrown.md | 49 ---- .../claude-sdk-python/qa/byoc-json-render.md | 59 ---- .../qa/declarative-gen-ui.md | 76 ++++++ .../qa/declarative-hashbrown.md | 87 ++++++ .../qa/declarative-json-render.md | 68 +++++ .../claude-sdk-python/qa/gen-ui-interrupt.md | 76 ++++++ .../qa/interrupt-headless.md | 28 ++ .../claude-sdk-python/qa/mcp-apps.md | 64 +++++ .../claude-sdk-python/qa/reasoning-default.md | 24 ++ .../qa/shared-state-write.md | 41 --- .../qa/threadid-frontend-tool-roundtrip.md | 43 +++ .../qa/tool-rendering-reasoning-chain.md | 25 ++ .../claude-sdk-python/src/agent_server.py | 41 +++ .../agents/readonly_state_agent_context.py | 12 +- .../src/app/api/copilotkit/route.ts | 8 + .../tests/e2e/a2ui-recovery.spec.ts | 115 ++++++++ .../tests/e2e/beautiful-chat.spec.ts | 19 +- .../tests/e2e/byoc-hashbrown.spec.ts | 17 -- .../tests/e2e/byoc-json-render.spec.ts | 21 -- .../tests/e2e/declarative-gen-ui.spec.ts | 255 ++++++++++++------ .../tests/e2e/declarative-hashbrown.spec.ts | 54 ++++ .../tests/e2e/declarative-json-render.spec.ts | 76 ++++++ .../tests/e2e/reasoning-custom.spec.ts | 119 ++++++++ .../tests/e2e/reasoning-default.spec.ts | 41 +++ .../tests/e2e/shared-state-read-write.spec.ts | 2 +- .../tests/e2e/shared-state-write.spec.ts | 113 -------- .../threadid-frontend-tool-roundtrip.spec.ts | 27 ++ 31 files changed, 1486 insertions(+), 423 deletions(-) create mode 100644 showcase/integrations/claude-sdk-python/qa/a2ui-fixed-schema.md create mode 100644 showcase/integrations/claude-sdk-python/qa/a2ui-recovery.md create mode 100644 showcase/integrations/claude-sdk-python/qa/beautiful-chat.md delete mode 100644 showcase/integrations/claude-sdk-python/qa/byoc-hashbrown.md delete mode 100644 showcase/integrations/claude-sdk-python/qa/byoc-json-render.md create mode 100644 showcase/integrations/claude-sdk-python/qa/declarative-gen-ui.md create mode 100644 showcase/integrations/claude-sdk-python/qa/declarative-hashbrown.md create mode 100644 showcase/integrations/claude-sdk-python/qa/declarative-json-render.md create mode 100644 showcase/integrations/claude-sdk-python/qa/gen-ui-interrupt.md create mode 100644 showcase/integrations/claude-sdk-python/qa/interrupt-headless.md create mode 100644 showcase/integrations/claude-sdk-python/qa/mcp-apps.md create mode 100644 showcase/integrations/claude-sdk-python/qa/reasoning-default.md delete mode 100644 showcase/integrations/claude-sdk-python/qa/shared-state-write.md create mode 100644 showcase/integrations/claude-sdk-python/qa/threadid-frontend-tool-roundtrip.md create mode 100644 showcase/integrations/claude-sdk-python/qa/tool-rendering-reasoning-chain.md create mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/a2ui-recovery.spec.ts delete mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/byoc-hashbrown.spec.ts delete mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/byoc-json-render.spec.ts create mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/declarative-hashbrown.spec.ts create mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/declarative-json-render.spec.ts create mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/reasoning-custom.spec.ts create mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/reasoning-default.spec.ts delete mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/shared-state-write.spec.ts create mode 100644 showcase/integrations/claude-sdk-python/tests/e2e/threadid-frontend-tool-roundtrip.spec.ts diff --git a/showcase/integrations/claude-sdk-python/PARITY_NOTES.md b/showcase/integrations/claude-sdk-python/PARITY_NOTES.md index 5a722c8ee2..4d918fb62c 100644 --- a/showcase/integrations/claude-sdk-python/PARITY_NOTES.md +++ b/showcase/integrations/claude-sdk-python/PARITY_NOTES.md @@ -46,7 +46,8 @@ frontend-aligned with the reference: - **Shared state**: `shared-state-read`, `shared-state-read-write` (dedicated `/shared-state-read-write`, emits `StateSnapshot`), `shared-state-streaming` (per-token deltas), - `readonly-state-agent-context`. + `readonly-state-agent-context` (dedicated `/readonly-state-agent-context`, + `tools=[]` + read-only-context prompt — see "Masking fixes landed"). - **Multi-agent**: `subagents` (dedicated `/subagents`, delegations via `STATE_SNAPSHOT`). - **Declarative UI**: `declarative-gen-ui` (A2UI dynamic), @@ -73,9 +74,40 @@ resume-path defect that also affects the reference: `useHeadlessInterrupt`. Both demos remain wired (frontend byte-aligned; backend on the shared -`/interrupt-adapted` scheduling agent in `interrupt_agent.py`). The identical -QUARANTINE comment lives in both this manifest and the langgraph-python -manifest. +`/interrupt-adapted` scheduling agent in `interrupt_agent.py`). + +### Correction: the upstream bug is NOT the only blocker here + +The shared react-core defect above is real and blocks the reference too, but for +THIS integration it is not the whole story, and the earlier wording (which cited +only the upstream bug, and framed these as "not regressions in this +integration") was incomplete. The two halves of these cells expect DIFFERENT +mechanisms: + +- The **backend** (`src/agents/interrupt_agent.py`) implements "Strategy B", + mirroring `ms-agent-python`: the Claude Agent SDK has no `interrupt()` + primitive, so the agent calls a frontend tool named `schedule_meeting`. Its + docstring states "The frontend registers the tool via `useFrontendTool`", and + it forwards frontend-defined tools. +- The **frontend** is the byte-identical langgraph-python copy and uses + `useInterrupt` — the low-level primitive for LangGraph `interrupt(...)` + events. Nothing in `gen-ui-interrupt/` or `interrupt-headless/` registers + `schedule_meeting` via `useFrontendTool` (grep returns nothing), so the + backend's tool call has no client implementation, and no interrupt payload is + ever produced for `useInterrupt` to consume. + +The reference is internally consistent (real `interrupt()` in +`langgraph-python/src/agents/interrupt_agent.py`, paired with `useInterrupt`). +Note that `_shared/interrupt-fallback-slots.ts` does NOT rescue this: it only +fills in missing slots WITHIN an interrupt payload, and no such payload arrives. + +Closing this locally would mean either diverging the frontend (violating the +near-identical-frontend rule) or teaching the backend to emit real interrupt +events — net-new feature work this ticket explicitly excludes. Per the ticket's +own rule ("if a declared cell can't go green without new-feature/upstream work, +reclassify to honest NSF — never force"), both cells stay NSF, but the reason +recorded here is now the accurate one. Even if the react-core fix ships, these +two cells will still need local work before they can go green. Both are honestly marked skipped-incapable (not green, not red). @@ -87,11 +119,35 @@ stack they loop (openGenerativeUI `generateSandboxedUi` re-emits on the follow-up run), but the loop is a **local-repro artifact**: it was ruled out as (a) a fixture issue — the pre-blitz/staging fixture also loops locally, and (b) a `@copilotkit/*` version drift — the container versions match the -green langgraph-python container, which passes ogui locally. The exact local -delta vs staging was not pinned (the backend's stdout does not surface to -`docker logs`, blocking deeper capture). CI (staging-equivalent x86 build) -is the arbiter for these two cells. The fixtures here are the canonical -staging-green versions (`hasToolResult`-gated follow-up leg). +langgraph-python container. The exact local delta vs staging was not pinned +(the backend's stdout does not surface to `docker logs`, blocking deeper +capture). CI (staging-equivalent x86 build) is the arbiter for these two +cells. The fixtures here are the canonical staging-green versions +(`hasToolResult`-gated follow-up leg). + +**Correction (measured 2026-08-31):** an earlier version of this note claimed +langgraph-python "passes ogui locally", making the failure look +claude-sdk-python-specific. That is not true on the current local stack. A +control run of the REFERENCE's own cell — +`bin/showcase test langgraph-python:open-gen-ui --d6 --direct --isolate …` — +also goes RED (`gen-ui-open: feature exceeded 300000ms wall-clock`, with +`ERR_NETWORK_IO_SUSPENDED`, `agent_run_failed`, and a streamed-JSON parse +error). So the local docker stack cannot adjudicate `open-gen-ui` for ANY +integration, the reference included. Treat local ogui results as +uninformative and rely on CI/staging, until someone pins the local delta. + +## Note on local D6 vs staging (multimodal) + +`multimodal` also cannot be verified on a local checkout that has not resolved +Git LFS. `public/demo-files/sample.png` and `sample.pdf` are LFS pointers; the +demo detects this and refuses to send, so the probe fails with +`settle-dom-missing` and the page prints "Sample \"sample.png\" is a Git LFS +pointer, not the real asset." This is NOT a cell defect — nothing in this +integration's `public/` was changed. CI resolves the assets explicitly +(`.github/workflows/showcase_validate.yml`: +`git lfs pull --include="showcase/integrations/*/public/demo-files/*"`, which +hard-errors if the pull no-ops), so the cell is verifiable there. To verify +locally you need `git-lfs` installed plus that pull. ## a2ui-recovery — native recovery loop @@ -107,7 +163,7 @@ aimock `sequenceIndex`. ## Masking fixes landed -Three cells were aimock-green but were routing to the WRONG agent live (a +Four cells were aimock-green but were routing to the WRONG agent live (a generic-fallback prompt masked by the fixture). Each now has a dedicated, correctly-prompted backend so the live behavior matches the fixture: @@ -118,16 +174,72 @@ correctly-prompted backend so the live behavior matches the fixture: tailored prompt (was masked generic fallback). 3. `hitl-in-app` — dedicated `/hitl-in-app` endpoint applying the HITL prompt via `system_prompt_override` (was masked generic fallback). +4. `readonly-state-agent-context` — dedicated `/readonly-state-agent-context` + endpoint applying the read-only-context prompt with `tools_override=[]` + (mirrors the reference's `tools=[]` graph). This one was the starkest + masking case: `src/agents/readonly_state_agent_context.py` was imported + NOWHERE, yet `manifest.yaml` lists it under this demo's `highlight:` block + and the integration is `docs_mode: generated` — so the published docs + presented a module as this cell's backend that the runtime never executed. + Verified via aimock's request recorder (`GET /v1/_requests`), which captures + the system prompt the fixture matcher ignores: + - BEFORE: generic sales-assistant `SYSTEM_PROMPT`, 9 backend tools, and + `POST /readonly-state-agent-context` → HTTP 404 (the path did not exist). + - AFTER: the read-only-context prompt carrying the probe's context sentinel, + 0 tools, and the path serving HTTP 200. + D6 was green BEFORE and AFTER — which is the point: the probe cannot see + this defect (GOTCHAS #8). + +## Verification-surface parity (specs + QA docs) + +`scripts/validate-parity.ts` now reports this integration at **39 demos / 39 +e2e specs / 40 QA docs with 3 warnings — identical to langgraph-python**, up +from 39 / 36 / 31 with 21 warnings. The 3 remaining warnings are byte-identical +to the reference's own (`interrupt-headless` has no spec, `reasoning-custom` has +no QA doc, and the stale demo-count baseline); closing those only here would +break the identical-tests rule. + +Work that got it there: + +- Ported the 4 missing specs (`a2ui-recovery`, `reasoning-custom`, + `reasoning-default`, `threadid-frontend-tool-roundtrip`). +- Renamed `byoc-hashbrown` / `byoc-json-render` spec+QA files to + `declarative-*` to match the demo ids (`validate-parity` keys filenames to + demo ids). Python module names KEEP the `byoc_` prefix — the reference does + too, and `manifest.yaml` cites them under `highlight:`. +- Deleted the orphan `shared-state-write` spec+QA pair (no such demo dir, no + manifest entry). +- Authored the 10 missing QA docs in the reference's own cut: full checklists + where it has them, `> Stub — authored for column completeness` where it + deliberately stubs (`interrupt-headless`, `reasoning-default`, + `tool-rendering-reasoning-chain`). Every file path, env var, endpoint, + `data-testid` and quoted prompt in them was verified to exist. + +### Two specs were not merely drifted — they were dead + +- `declarative-gen-ui.spec.ts` asserted suggestion pills that no longer exist + ("Show a KPI dashboard", "Pie chart — sales by region", …) against the + current Vantage Threads set ("Show my sales dashboard", "Team performance", + "Anything at risk?", "Top account details"), and its header comment claimed + the demo has no `data-testid` — false: the same seven ids the reference uses + are present. It could only ever have failed; nobody noticed because `--d6` + never invokes this surface (GOTCHAS #7). Rewritten onto the stable testids. +- `beautiful-chat.spec.ts` had the `Catalog not found` regression guard + (#4733 / #4734 / #5425) nested inside a conditional, so a regression could + slip past. Hoisted to the reference's unconditional placement. ## Flags / caveats -- **`readonly-state-agent-context` live-prompt gap**: the reference uses a - dedicated tailored-prompt graph; here the cell routes to the generic shared - agent (`readonly_state_agent_context.py` is docs-only). D6 and the frontend - are correct — context is injected via `useAgentContext` — but the live - system prompt is not tailored the way the reference is. Adjudicated at live - smoke. -- **`declarative-json-render` zod build caveat**: `catalog.ts` was reverted - from `zod4` back to `zod` for byte-parity with the reference. If - `@json-render` 0.18 turns out to require `zod4` here, the build may break — - re-introduce `zod4` and re-flag if so. +- **`beautiful-chat` is weaker by FIXTURE, not by test.** After the fix above + the test logic is byte-identical to the reference; the one remaining + divergence is a comment, and it is deliberate. The reference's fixture omits + `catalogId` so the run exercises route-level `defaultCatalogId` resolution — + the exact path that regressed in #4733 / #4734 / #5425. This integration's + `aimock/d4/claude-sdk-python/chat.json` hardcodes + `catalogId: "copilotkit://app-dashboard-catalog"`, so that path is never + exercised here. Copying the reference's comment verbatim would have stated + something false about our fixture. Our route already configures the same + `defaultCatalogId`, so aligning the fixture (dropping the explicit id) is a + small, viable follow-up in the fixture lane. +- **`declarative-json-render` zod caveat — RESOLVED.** `catalog.ts` imports + plain `zod`, matching the reference, which now does the same. No action. diff --git a/showcase/integrations/claude-sdk-python/qa/a2ui-fixed-schema.md b/showcase/integrations/claude-sdk-python/qa/a2ui-fixed-schema.md new file mode 100644 index 0000000000..560d0712fe --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/a2ui-fixed-schema.md @@ -0,0 +1,57 @@ +# QA: Declarative Generative UI (A2UI — Fixed Schema) — Claude Agent SDK (Python) + +## Prerequisites + +- Demo is deployed and accessible at `/demos/a2ui-fixed-schema` on the dashboard host +- Next.js host is healthy (`GET /api/health` → `{"status":"ok","integration":"claude-sdk-python"}`) and the Python backend is reachable (`GET /api/copilotkit` reports `agent_status: "reachable"`, which it derives by polling `${AGENT_URL}/health`) +- `ANTHROPIC_API_KEY` is set — this cell's model calls go to Anthropic, not OpenAI (`OPENAI_API_KEY` is in `.env.example` for the shared stack, but no code path in this demo reads it). `ANTHROPIC_MODEL` is optional; `.env.example` sets `claude-opus-4-8`, and `src/agents/a2ui_fixed.py` falls back to that same id +- `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI agent server `src/agent_server.py`, which exposes `@app.post("/a2ui-fixed-schema")` → `run_a2ui_fixed_agent`. There is **no** LangGraph deployment and no graph registration in this integration +- Note: unlike the langgraph-python reference, the outer card here **does** carry a stable `data-testid="a2ui-fixed-card"` (see `src/app/demos/a2ui-fixed-schema/a2ui/renderers.tsx`). Everything else below relies on verbatim visible text, DOM structure, and the JSON schema at `src/agents/a2ui_schemas/flight_schema.json` + +## Test Steps + +### 1. Basic Functionality + +- [ ] Navigate to `/demos/a2ui-fixed-schema`; verify the page renders within 3s: a `max-w-4xl` column with `border-x border-neutral-200 bg-white` on a `bg-neutral-50` page, filling full viewport height, with the `CopilotChat` itself `rounded-2xl` +- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-a2ui-fixed-schema"` and `agent="a2ui-fixed-schema"` (DevTools → Network: sending a message hits that endpoint, not `/api/copilotkit`) +- [ ] Verify the single suggestion pill is visible with verbatim title "Find SFO → JFK" (message body: "Find me a flight from SFO to JFK on United for $289.") +- [ ] Send "Hello" and verify an assistant text response appears within 10s (no flight card for plain text) + +### 2. Feature-Specific Checks + +#### Schema Wiring (fixed catalog + `includeBasicCatalog`) + +- [ ] DevTools → Network: after the first successful `display_flight` call, verify the tool result contains an `a2ui_operations` container whose `createSurface` carries `surfaceId: "flight-fixed-schema"` and `catalogId: "copilotkit://flight-fixed-catalog"` (matches `SURFACE_ID` / `CATALOG_ID` in `src/agents/a2ui_fixed.py` and `CATALOG_ID` in `src/app/demos/a2ui-fixed-schema/a2ui/catalog.ts`) +- [ ] Verify the same container's `updateComponents` carries the full `FLIGHT_SCHEMA` tree (12 nodes from `src/agents/a2ui_schemas/flight_schema.json`: `root`, `content`, `title`, `route`, `from`, `arrow`, `to`, `meta`, `airline`, `price`, `bookButton`, `bookButtonLabel`) and that `updateDataModel` writes `{origin, destination, airline, price}` at path `/` + +#### Search-Flights Prompt (`display_flight` tool → `flight_schema.json`) + +- [ ] Click the "Find SFO → JFK" suggestion; within 20s verify a single flight card renders in-transcript with `data-testid="a2ui-fixed-card"`, assembled per `flight_schema.json`: + - outer `Card` (`max-w-md`, 20px padding) wrapping a `Column` of children in this order: title row, route row, meta row, book button + - `Title` node renders the eyebrow "Itinerary" above the literal schema text "Flight Details", with an outline mono badge "1-stop · economy" on the right + - `route` row shows `Airport` "SFO" → `Arrow` (an SVG chevron flanked by two hairline separators) → `Airport` "JFK" (both monospaced, `text-2xl`, semibold, wide tracking) + - `meta` row shows `AirlineBadge` "UNITED" (secondary pill, uppercase, `0.08em` tracking) on the left and `PriceTag` — the eyebrow "Total" followed by monospaced "$289" — on the right + - `Button` renders full-width with label "Book flight" +- [ ] Verify all four data-model fields resolved correctly (origin=`SFO`, destination=`JFK`, airline=`United`, price=`$289`) — each is a `{ path: "/..." }` binding in the schema and must reach the DOM as a plain string via the binder (no literal `{path}` leak and no React error #31) + +#### Book-Flight Button (inert — pure presentation) + +- [ ] Verify the "Book flight" button renders the schema-declared label and is clickable, but the click is a no-op: the agent is not invoked, no schema swap occurs, and the button does not transition to a "Booked" state. The schema declares an `action` (`book_flight`) purely for fidelity; the `Button` renderer in `a2ui/renderers.tsx` deliberately drops it — schema-swap-on-action waits on the Python SDK exposing `action_handlers=` on `a2ui.render` (see the `Button` comment in `a2ui/renderers.tsx`). `src/agents/a2ui_schemas/booked_schema.json` ships alongside but is not loaded by any code path + +#### Follow-up Prompt (data-model refresh) + +- [ ] Send "Find me a flight from LAX to ORD on Delta for $412."; within 20s verify the card updates in place with origin=`LAX`, destination=`ORD`, airline=`DELTA`, price=`$412` (same schema, new data model — proves the fixed-schema pattern: schema once, data streams) + +### 3. Error Handling + +- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response) +- [ ] Send "What is the capital of France?"; verify the agent replies in plain text without invoking `display_flight` (no flight card rendered, no `a2ui_operations` in the response) +- [ ] DevTools → Console: walk through all flows above; verify no uncaught errors and specifically no React error #31 ("objects are not valid as a React child, found: object with keys {path}") — the `DynString` union in `a2ui/definitions.ts` is what prevents this, so a single occurrence is a regression + +## Expected Results + +- Chat loads within 3s; plain-text response within 10s; flight card renders within 20s of the search prompt +- `display_flight` is called exactly once per search prompt; result contains an `a2ui_operations` container with `catalogId: "copilotkit://flight-fixed-catalog"` and the full 12-node flight schema +- All custom renderers in `a2ui/renderers.tsx` (`Card`, `Title`, `Airport`, `Arrow`, `AirlineBadge`, `PriceTag`, `Button`) render at least once per search-flights run +- Clicking "Book flight" is a no-op (inert presentation button) +- No UI layout breaks, no `{path}` leak into the DOM, no uncaught console errors diff --git a/showcase/integrations/claude-sdk-python/qa/a2ui-recovery.md b/showcase/integrations/claude-sdk-python/qa/a2ui-recovery.md new file mode 100644 index 0000000000..d194d89d43 --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/a2ui-recovery.md @@ -0,0 +1,47 @@ +# QA: A2UI Error Recovery — Claude Agent SDK (Python) + +## Prerequisites + +- Demo is deployed and accessible at `/demos/a2ui-recovery` on the dashboard host +- Next.js host is healthy (`GET /api/health`); the Python backend is reachable (`GET /api/copilotkit` reports `agent_status: "reachable"`); `ANTHROPIC_API_KEY` is set. `AGENT_URL` (default `http://localhost:8000`) points at `src/agent_server.py`, which exposes `@app.post("/a2ui-recovery")` → `run_a2ui_recovery_agent`. There is **no** LangGraph deployment in this integration +- **The recovery loop is NATIVE here.** The langgraph-python reference owns `generate_a2ui` via `ag_ui_langgraph.get_a2ui_tools` and runs the validate→retry loop inside the toolkit. claude-sdk-python uses its own adapter (`ag-ui-claude-sdk` + `claude-agent-sdk`) and does not depend on `ag_ui_langgraph` / `ag_ui_a2ui_toolkit`, so `src/agents/recovery_agent.py` re-implements the loop: `_validate_a2ui_components` (structural checks — `empty_components`, `missing_id`, `missing_component_type`, `unresolved_child`, `no_root`) driven by `_run_render_with_recovery` with `MAX_A2UI_ATTEMPTS = 3`, one inner `render_a2ui` Claude call **per attempt**, and `_wrap_recovery_exhausted_envelope` returning `{"error": …, "code": "a2ui_recovery_exhausted", "attempts": [...]}` on cap +- Backend-owned wiring: `src/app/api/copilotkit-a2ui-recovery/route.ts` sets `injectA2UITool: false` (load-bearing — the backend owns `generate_a2ui`, whose only argument is `intent`) plus `defaultCatalogId: "declarative-gen-ui-catalog"` +- Reuses the **declarative-gen-ui** catalog (`myCatalog`, `catalogId: "declarative-gen-ui-catalog"`) and the Vantage Threads sales context (`useSalesAnalystContext`) — no new components +- The `building` / `retrying` / `failed` lifecycle chrome comes from `@copilotkit/react-core/v2` (`A2UIRecoveryStates.tsx`, mounted by `A2UIMessageRenderer`) + +## Test Steps + +### 1. Basic Functionality + +- [ ] Navigate to `/demos/a2ui-recovery`; verify the page renders within 3s and a single `CopilotChat` pane is centered (`max-w-4xl`, `rounded-2xl`, full viewport height) +- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-a2ui-recovery"` and `agent="a2ui-recovery"` (DevTools → Network: sending a message hits that endpoint, not `/api/copilotkit`) +- [ ] Verify both suggestion pills are visible with verbatim titles: + - "Recover a bad render" + - "Show an unrecoverable failure" + +### 2. Healing path + +- [ ] Click "Recover a bad render" ("Build my Q2 revenue summary and self-correct a malformed first attempt.") +- [ ] Attempt 1 of the inner `render_a2ui` returns a **structurally invalid** surface (`root` lists a child id that no component defines → `unresolved_child`). Verify the native loop rejects it, retries, and attempt 2 paints — no broken surface, no error banner. A "Building interface" skeleton is expected; the "Retrying generation… (N/M attempts)" sub-label is threshold-gated (2 attempts / 2000ms) so it may or may not become visible — do not fail the run on its absence +- [ ] Verify the **painted** surface is valid: at least two `declarative-metric` tiles — "QUARTERLY REVENUE / $4.2M / ↑ +12% QoQ" and "WIN RATE / 31% / ↓ -2 pts" +- [ ] DevTools → Network: verify the `generate_a2ui` tool result carries an `a2ui_operations` container (and **no** `a2ui_recovery_exhausted`) +- [ ] Verify the chat reply is one short sentence noting the heal +- [ ] If backend stdout is reachable, verify two `[a2ui recovery] attempt N: …` log lines — attempt 1 `invalid` with an `unresolved_child` error, attempt 2 `valid` (emitted by `_log_attempt` on the `agents.recovery_agent` logger) + +### 3. Hard-fail (recovery exhausted) path + +- [ ] Click "Show an unrecoverable failure" ("Build a report that fails every validation pass so I can preview the fallback.") +- [ ] Verify the lifecycle ends in the tasteful `failed` card — amber panel reading "Couldn't generate the UI" over "Something went wrong rendering this. You can keep chatting and try again." — and NOT a broken/half-rendered surface, and NOT a silent drop. No new `declarative-metric` tile may appear for this pill +- [ ] DevTools → Network: verify `render_a2ui` was attempted up to the cap (3 attempts, all invalid) and the `generate_a2ui` result is an `a2ui_recovery_exhausted` envelope with a 3-entry `attempts` array (no `a2ui_operations` painted) +- [ ] Verify the chat reply gracefully explains the fallback (one short sentence) + +### 4. Regression / isolation + +- [ ] Verify the recovery demo does not affect the `declarative-gen-ui` or `beautiful-chat` demos (separate routes and agents, even though the catalog is shared) +- [ ] Re-run each pill a second time and verify the same lifecycle + +## Notes + +- The malformed renders are forced by aimock fixtures (`showcase/aimock/d6/claude-sdk-python/a2ui-recovery.json`): the inner `render_a2ui` calls are matched by `userMessage` + `toolName=render_a2ui` (+ `sequenceIndex` 0/1 for the heal pill's two attempts), and the outer narration by the emit's unique `toolCallId`. The retry DECISION is made live by the native loop in `recovery_agent.py` — the fixture only supplies the render args. +- The pill prompts are unique per integration on purpose: the inner `render_a2ui` calls carry no `x-aimock-context`, so identical prompts across integrations would collide in the shared aimock matcher. Keep `src/app/demos/a2ui-recovery/suggestions.ts` in sync with `showcase/harness/src/probes/scripts/d5-a2ui-recovery.ts`. +- Heads-up on a stale comment: `suggestions.ts` still describes the heal as `parse_and_fix` healing sloppy JSON-string args in one pass. That is the toolkit's mechanism, not this integration's. Here the heal is a genuine **invalid → retry → valid** two-attempt loop; the fixture `_comment` fields are authoritative. diff --git a/showcase/integrations/claude-sdk-python/qa/beautiful-chat.md b/showcase/integrations/claude-sdk-python/qa/beautiful-chat.md new file mode 100644 index 0000000000..50bdc73864 --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/beautiful-chat.md @@ -0,0 +1,93 @@ +# QA: Beautiful Chat — Claude Agent SDK (Python) + +## Prerequisites + +- Demo is deployed and accessible at `/demos/beautiful-chat` on the dashboard host +- Agent backend is healthy: `GET /api/health` returns `{"status":"ok","integration":"claude-sdk-python"}`, and `GET /api/copilotkit` reports `agent_status: "reachable"` (it polls `${AGENT_URL}/health` on the FastAPI backend) +- `ANTHROPIC_API_KEY` is set on the deployment; `ANTHROPIC_MODEL` defaults to `claude-opus-4-8`; `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI server exposing `POST /beautiful-chat` (`src/agent_server.py`) +- Note: the demo source contains no `data-testid` attributes. Checks below rely on verbatim visible text and DOM structure. + +## Test Steps + +### 1. Basic Functionality + +- [ ] Navigate to `/demos/beautiful-chat`; verify the page renders within 3s with the "CopilotKit" wordmark plus the logo mark (`img[alt="CopilotKit"]`, src `/copilotkit-logo-mark.svg`) top-left of the chat pane +- [ ] Verify the `Chat` / `App` mode pill is fixed top-right, `Chat` active (highlighted) by default, and the right-side canvas region is collapsed (width 0) +- [ ] Verify the `CopilotChat` input is rendered with no disclaimer text below it, and that attachments are enabled +- [ ] Verify all 9 suggestion pills are visible with verbatim titles (`showcase.json` is `"default"`, so no pill carries a highlight class): + - "Pie Chart (Controlled Generative UI)" + - "Bar Chart (Controlled Generative UI)" + - "Schedule Meeting (Human In The Loop)" + - "Search Flights (A2UI Fixed Schema)" + - "Sales Dashboard (A2UI Dynamic)" + - "Excalidraw Diagram (MCP App)" + - "Calculator App (Open Generative UI)" + - "Toggle Theme (Frontend Tools)" + - "Task Manager (Shared State)" +- [ ] Send "Hello" and verify an assistant text response appears within 10s +- [ ] DevTools → Network: verify the send POSTs to `/api/copilotkit-beautiful-chat` (the dedicated runtime, not the shared `/api/copilotkit`) + +### 2. Feature-Specific Checks + +#### Mode Toggle (frontend tools `enableAppMode` / `enableChatMode`) + +- [ ] Click `App`; verify the canvas expands to ~2/3 width showing the TodoList empty state: pencil emoji, heading "No todos yet", subtext "Create your first task to get started", enabled "Add a task" button +- [ ] Click `Chat`; verify the canvas collapses back to width 0 + +#### Shared State — Task Manager (agent tools `manage_todos`, `get_todos`) + +- [ ] Click the "Task Manager (Shared State)" pill; verify the mode auto-switches to App (the system prompt in `src/agents/agent.py` routes todos through `enableAppMode` first) and within 15s the "To Do" column renders exactly 3 todo cards (each with emoji, title, description) +- [ ] Verify the "Done" column is empty (shows "No completed todos yet") +- [ ] Click a todo's checkbox; verify the card moves from "To Do" to "Done" + +#### Controlled Generative UI — Pie Chart (agent tool `query_data` + frontend component `pieChart`) + +- [ ] Click "Pie Chart (Controlled Generative UI)"; within 15s verify a pie-chart card renders in-transcript with a non-empty `CardTitle` and `CardDescription` +- [ ] Verify the donut SVG renders at least 2 `` slice elements inside the card +- [ ] Verify the legend renders one row per slice: colored dot, label, comma-formatted value, and a percentage ending in "%"; percentages sum to 100% + +#### Controlled Generative UI — Bar Chart (agent tool `query_data` + frontend component `barChart`) + +- [ ] Click "Bar Chart (Controlled Generative UI)"; within 15s verify a bar-chart card renders with `CardTitle`, `CardDescription`, and a bar-chart icon in the header +- [ ] Verify the recharts `ResponsiveContainer` (height 280px) renders at least 2 bar rectangles with X-axis labels matching the `label` field values; bars animate in via the `barSlideIn` keyframe on first render + +#### Human-in-the-Loop — Schedule Meeting (frontend tool `scheduleTime`) + +- [ ] Click "Schedule Meeting (Human In The Loop)"; within 15s verify a MeetingTimePicker card renders with a clock icon, a heading (agent-supplied `reasonForScheduling` or default "Schedule a Meeting"), 3 time-slot buttons each with date + time + a "30 min" duration badge, and a "None of these work" ghost button +- [ ] Click a time slot; verify the card switches to the confirmed state with heading "Meeting Scheduled", the chosen date/time, and a green check icon +- [ ] Re-trigger, click "None of these work"; verify the card shows heading "No Time Selected" and subtext "Looking for a better time that works for you" + +#### A2UI Fixed Schema — Search Flights (agent tool `search_flights`) + +- [ ] Click "Search Flights (A2UI Fixed Schema)"; within 20s verify exactly 2 flight cards render in-transcript, each with airline name, airline logo image, flight number, origin/destination, date, departure/arrival times, duration, a colored status dot, a status label (e.g. "On Time"), and a price +- [ ] Note: the beautiful-chat runtime sets `injectA2UITool: false` with `defaultCatalogId: "copilotkit://app-dashboard-catalog"` — the same id the `demonstrationCatalog` registers, so cards must resolve against the local catalog rather than an injected tool + +#### A2UI Dynamic — Sales Dashboard (agent tool `generate_a2ui`) + +- [ ] Click "Sales Dashboard (A2UI Dynamic)"; within 30s verify a dynamic dashboard surface renders containing total-revenue metric, new-customers metric, conversion-rate metric, a pie chart (revenue by category), and a bar chart (monthly sales) + +#### MCP App — Excalidraw Diagram + +- [ ] Click "Excalidraw Diagram (MCP App)"; within 30s verify an Excalidraw embed renders a diagram with a router, 2 switches, and 4 computers (no console errors referencing `MCP_SERVER_URL`, default `https://mcp.excalidraw.com`, pinned `serverId: "excalidraw"`) + +#### Open Generative UI — Calculator App (`generateSandboxedUi`) + +- [ ] Click "Calculator App (Open Generative UI)"; within 30s verify a sandboxed calculator UI renders with digit/operator buttons plus labeled metric shortcut buttons +- [ ] Click a metric shortcut button; verify its value is inserted into the calculator display + +#### Frontend Tool — Toggle Theme (`toggleTheme`) + +- [ ] Click "Toggle Theme (Frontend Tools)"; verify the `html` element's `class` attribute flips between containing `dark` and containing `light` (the `ThemeProvider` removes both classes then adds the active one) + +### 3. Error Handling + +- [ ] Attempt to send an empty message; verify it is a no-op (no user bubble, no assistant response) +- [ ] Send a ~500-character message; verify it wraps in-transcript without horizontal scroll or layout break +- [ ] With the FastAPI backend stopped, send a message; verify the UI surfaces a visible error path rather than hanging silently, and DevTools → Console shows no uncaught errors during any flow above + +## Expected Results + +- Chat loads within 3 seconds; plain-text response within 10 seconds +- Controlled charts (pie/bar) render within 15 seconds of prompt; A2UI surfaces within 20–30 seconds +- No UI layout breaks, no flash of unstyled content, no uncaught console errors +- All 5 agent tools reachable via `BEAUTIFUL_CHAT_TOOLS` in `src/agents/agent.py` (`query_data`, `search_flights`, `generate_a2ui`, `manage_todos`, `get_todos`) are exercised by at least one check above diff --git a/showcase/integrations/claude-sdk-python/qa/byoc-hashbrown.md b/showcase/integrations/claude-sdk-python/qa/byoc-hashbrown.md deleted file mode 100644 index 4292cfe455..0000000000 --- a/showcase/integrations/claude-sdk-python/qa/byoc-hashbrown.md +++ /dev/null @@ -1,49 +0,0 @@ -# QA: BYOC hashbrown — Claude Agent SDK (Python) - -## Prerequisites - -- Demo is deployed and accessible -- Agent backend is healthy (check /api/health) -- `ANTHROPIC_API_KEY` is set on the deployment - -## Test Steps - -### 1. Basic Functionality - -- [ ] Navigate to `/demos/byoc-hashbrown` -- [ ] Verify the header "BYOC: Hashbrown" renders -- [ ] Verify the description paragraph mentions `@hashbrownai/react` - -### 2. Feature-Specific Checks - -#### Q4 Sales Summary (mixed catalog) - -- [ ] Send "Show me a Q4 sales summary" (or click a suggestion) -- [ ] Verify a `data-testid="metric-card"` renders with a formatted value -- [ ] Verify a `data-testid="pie-chart"` renders with at least three - legend rows -- [ ] Verify a `data-testid="bar-chart"` renders with at least three - columns -- [ ] Verify at least one Markdown heading renders inline - -#### Deal card - -- [ ] Ask "Show me a sample deal in the negotiation stage" -- [ ] Verify a `data-testid="hashbrown-deal-card"` renders with a stage - badge - -### 3. Streaming behaviour - -- [ ] Observe components progressively appear as Claude streams the - JSON envelope — no full-refresh flash at the end of streaming - -### 4. Error Handling - -- [ ] No console errors during normal usage. -- [ ] No hashbrown schema-validation errors logged. - -## Expected Results - -- Chat loads within 3 seconds -- Agent responds within 15 seconds -- Backend emits the JSON envelope (`{ui: [...]}`), NEVER XML diff --git a/showcase/integrations/claude-sdk-python/qa/byoc-json-render.md b/showcase/integrations/claude-sdk-python/qa/byoc-json-render.md deleted file mode 100644 index 82da7f3562..0000000000 --- a/showcase/integrations/claude-sdk-python/qa/byoc-json-render.md +++ /dev/null @@ -1,59 +0,0 @@ -# QA: BYOC json-render — Claude Agent SDK (Python) - -## Prerequisites - -- Demo is deployed and accessible -- Agent backend is healthy (check /api/health) -- `ANTHROPIC_API_KEY` is set on the deployment - -## Test Steps - -### 1. Basic Functionality - -- [ ] Navigate to `/demos/byoc-json-render` -- [ ] Verify the chat surface loads inside the centered 4xl container -- [ ] Verify the three suggestion pills are visible: - "Sales dashboard", "Revenue by category", "Expense trend" - -### 2. Feature-Specific Checks - -#### Sales dashboard (MetricCard + BarChart) - -- [ ] Click the "Sales dashboard" suggestion pill -- [ ] Verify a `data-testid="metric-card"` element renders with a label - and a dollar-formatted value -- [ ] Verify a `data-testid="bar-chart"` element renders inside the same - `data-testid="json-render-root"` wrapper - -#### Revenue by category (PieChart) - -- [ ] Click "Revenue by category" -- [ ] Verify a `data-testid="pie-chart"` element renders with at least - three legend rows - -#### Expense trend (BarChart) - -- [ ] Click "Expense trend" -- [ ] Verify `data-testid="bar-chart"` renders with three months of data - -### 3. Streaming behaviour - -- [ ] Observe the raw JSON streaming into the chat bubble briefly while - the model emits the spec -- [ ] Verify the catalog components swap in cleanly once the JSON - becomes valid — no flicker, no duplicate render - -### 4. Error Handling - -- [ ] Ask a free-form question that has nothing to do with dashboards - (e.g. "What is 2+2?"). The agent should still reply with a JSON - spec — it may emit a single MetricCard — and the page must NOT - white-screen. -- [ ] No console errors during normal usage. - -## Expected Results - -- Chat loads within 3 seconds -- Agent responds within 15 seconds (Claude opus) -- Components render from the json-render catalog wrapped in a single - `` (no missing-provider crashes) diff --git a/showcase/integrations/claude-sdk-python/qa/declarative-gen-ui.md b/showcase/integrations/claude-sdk-python/qa/declarative-gen-ui.md new file mode 100644 index 0000000000..5d448d5ae2 --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/declarative-gen-ui.md @@ -0,0 +1,76 @@ +# QA: Declarative Generative UI (A2UI — Dynamic Schema) — Claude Agent SDK (Python) + +## Prerequisites + +- Demo is deployed and accessible at `/demos/declarative-gen-ui` on the dashboard host +- Next.js host is healthy (`GET /api/health`) and the Python backend is reachable (`GET /api/copilotkit` reports `agent_status: "reachable"`, derived from `${AGENT_URL}/health`) +- `ANTHROPIC_API_KEY` is set — this cell's model calls go to Anthropic, not OpenAI. `ANTHROPIC_MODEL` is optional; `.env.example` sets `claude-opus-4-8` and `src/agents/a2ui_dynamic.py` falls back to the same id for BOTH the outer agent call and the inner design call +- `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI agent server `src/agent_server.py`, which exposes `@app.post("/declarative-gen-ui")` → `run_a2ui_dynamic_agent`. There is **no** LangGraph deployment and no graph registration in this integration +- Backend-owned wiring: `src/app/api/copilotkit-declarative-gen-ui/route.ts` sets `injectA2UITool: false` (the backend owns `generate_a2ui`) plus `defaultCatalogId: "declarative-gen-ui-catalog"`. `generate_a2ui(context)` in `src/agents/a2ui_dynamic.py` runs a SECOND Claude call forced onto the `render_a2ui` schema via `tool_choice`, then converts the args with `build_a2ui_operations_from_tool_call` from `tools/generate_a2ui.py` (a symlink to `showcase/shared/python/tools/generate_a2ui.py`) +- The demo plays a sales analyst for the fictional **Vantage Threads** company. The dataset and per-question composition rules are registered as agent context in `src/app/demos/declarative-gen-ui/sales-context.ts` — surfaces should reflect those numbers ($4.2M Q2 revenue, 4 regions, 5 reps, 3 at-risk accounts, Meridian Apparel Group as top account) +- Each custom renderer carries a stable `data-testid`: `declarative-card`, `declarative-metric`, `declarative-pie-chart`, `declarative-bar-chart`, `declarative-status-badge`, `declarative-data-table`, `declarative-info-row` (see `src/app/demos/declarative-gen-ui/a2ui/renderers.tsx`) + +## Test Steps + +### 1. Basic Functionality + +- [ ] Navigate to `/demos/declarative-gen-ui`; verify the page renders within 3s and a single `CopilotChat` pane is centered (max-width ~896px / `max-w-4xl`, `rounded-2xl`, full viewport height) +- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-declarative-gen-ui"` and `agent="declarative-gen-ui"` (DevTools → Network: sending a message hits that endpoint, not `/api/copilotkit`) +- [ ] Verify all 4 suggestion pills are visible with verbatim titles: + - "Show my sales dashboard" + - "Team performance" + - "Anything at risk?" + - "Top account details" +- [ ] Verify no pill mentions a chart type — chart steering lives in `SYSTEM_PROMPT` (`src/agents/a2ui_dynamic.py`) and in the `COMPOSITION_RULES` context entry, not in the user prompt (OSS-136) +- [ ] Send "Hello" and verify an assistant text response appears within 10s (no A2UI surface rendered for plain text) + +### 2. Feature-Specific Checks + +#### Catalog Wiring (provider `a2ui={{ catalog: myCatalog }}`) + +- [ ] DevTools → Network: on the first tool-driven response, verify the `generate_a2ui` tool result contains an `a2ui_operations` container with `catalogId: "declarative-gen-ui-catalog"` (matches `createCatalog(..., { catalogId: "declarative-gen-ui-catalog" })` in `a2ui/catalog.ts` and the route's `defaultCatalogId`) +- [ ] Verify only ONE `generate_a2ui` tool call is emitted per surface-producing prompt — the runtime must not inject a second A2UI tool on top of the backend's (that is what `injectA2UITool: false` prevents) + +#### Hero Pill — Composed Sales Dashboard + +- [ ] Click "Show my sales dashboard" ("Show me my sales dashboard for this quarter."); within 60s verify ONE composed surface renders containing ALL of (no surrounding `declarative-card` — the charts carry their own card chrome): + - a bare row of 4 `declarative-metric` KPI tiles (uppercase label, 1.5rem value, trend arrow with delta — green `↑` `#059669` for up, red `↓` `#dc2626` for down, e.g. "↑ +12% QoQ") + - a `declarative-pie-chart` (recharts donut, `innerRadius` 40 / `outerRadius` 80, `paddingAngle` 2, one `.recharts-pie-sector` per slice, tooltip on hover, no legend) showing revenue by region + - a `declarative-bar-chart` (recharts, 200px tall, single blue `#3b82f6` bars with rounded tops `[4,4,0,0]`, dashed `3 3` grid) showing monthly revenue for all six months Jan–Jun +- [ ] Verify the surface is a single composed dashboard, NOT a lonely single widget — this is the regression OSS-136 was filed about +- [ ] Verify the pie slices cycle through `CHART_COLORS` (`#3b82f6`, `#8b5cf6`, `#ec4899`, `#f59e0b`, `#10b981`, `#6366f1`) and bars are uniform blue `#3b82f6`; every chart sits in the shared `CardShell` chrome (12px radius, 20px padding, soft shadow) +- [ ] Verify the chat reply text beneath the surface is one short sentence (per `SYSTEM_PROMPT`: "Keep chat replies to one short sentence; let the UI do the talking.") +- [ ] Verify metric numbers match the Vantage Threads dataset (revenue $4.2M, 186 new customers, 31% win rate, $22.6k avg deal) + +#### Team Performance — DataTable + +- [ ] Click "Team performance" ("How are our sales reps performing against quota?"); within 60s verify a `declarative-data-table` renders inside a `declarative-card`: uppercase column headers (rep / attainment / pipeline), one body row per rep (5 reps, Dana Whitfield 124% through Elena Vasquez 71%), tabular numerals +- [ ] Verify a quota-attainment `declarative-bar-chart` renders alongside the table (dashboardy, not a bare table); no `declarative-status-badge` or `declarative-info-row` + +#### At Risk — StatusBadge Cards + +- [ ] Click "Anything at risk?" ("Are any accounts or pipeline deals at risk this quarter?"); within 60s verify a risk panel: a strip of 3 `declarative-metric` tiles (ARR at risk $615k, accounts at risk 3, biggest exposure Northwind $340k) above three side-by-side `declarative-card`s (Northwind Retail, Cascadia Outfitters, Atlas Goods), each with a content-sized `declarative-status-badge` (`error` for high severity, `warning` for medium) and a one-line reason + recommended next action +- [ ] Verify the badges are content-sized pills (not full-width banners) and that no chart or table renders for this pill + +#### Top Account — InfoRow Facts + +- [ ] Click "Top account details" ("Pull up the details on our biggest account."); within 60s verify a `declarative-card` for Meridian Apparel Group with at least 3 `declarative-info-row` label/value rows (owner, region, ARR $612k, renewal Sep 30, last contact), each separated by a 1px bottom border with no trailing border on the last row +- [ ] Verify a product-line `declarative-pie-chart` renders next to the fact card (grounded in Meridian's product mix: Outerwear $260k, Footwear $180k, Accessories $112k, Custom $60k); no data table or status badge + +#### Cross-Pill Differentiation (mirrors the D5 probe) + +- [ ] Run all 4 pills in one conversation; verify each pill mounts its distinguishing component fresh (the D5 probe `showcase/harness/src/probes/scripts/d5-gen-ui-declarative.ts` asserts a newly-mounted testid per pill — leftovers from earlier pills must not be the only match) + +### 3. Error Handling + +- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response) +- [ ] Send "What is 2+2?"; verify the agent replies in plain text without invoking `generate_a2ui` (no `a2ui_operations` in the response stream, no surface rendered) +- [ ] DevTools → Console: walk through all flows above; verify no uncaught errors, no React error #31, no A2UI render-error banners ("Cannot create component root without a type", "Catalog not found"), and no `Invalid chart value` warnings (the chart renderers log that when the model emits a non-numeric `value`) + +## Expected Results + +- Chat loads within 3s; plain-text response within 10s; A2UI surfaces render within 60s of prompt (the inner `render_a2ui` Claude call can be slow on cold start) +- `generate_a2ui` is called exactly once per surface-producing prompt; result contains a valid `a2ui_operations` container with `catalogId: "declarative-gen-ui-catalog"` +- The hero pill produces a composed dashboard (4 KPI tile metrics + 1 PieChart + 1 BarChart in one surface, with NO surrounding Card per OSS-136); pills 2–4 produce their distinguishing component (data-table / status-badge / info-row) +- Numbers are consistent with the Vantage Threads dataset across all four pills +- No UI layout breaks, no flash of unstyled content, no uncaught console errors diff --git a/showcase/integrations/claude-sdk-python/qa/declarative-hashbrown.md b/showcase/integrations/claude-sdk-python/qa/declarative-hashbrown.md new file mode 100644 index 0000000000..81ba4c7963 --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/declarative-hashbrown.md @@ -0,0 +1,87 @@ +# QA: Declarative UI — Hashbrown — Claude Agent SDK (Python) + +## Prerequisites + +- Demo deployed at `/demos/declarative-hashbrown` +- Agent backend healthy (`GET /api/health` returns + `{"status":"ok","integration":"claude-sdk-python"}`) +- `ANTHROPIC_API_KEY` set in the deployment environment (`ANTHROPIC_MODEL` + defaults to `claude-opus-4-8`) +- `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI server + exposing `POST /declarative-hashbrown` (`src/agent_server.py`, prompt in + `src/agents/byoc_hashbrown_agent.py`) +- `@hashbrownai/core` + `@hashbrownai/react` installed in the package + (pinned to `0.5.0-beta.4`) +- Frontend runtime: `/api/copilotkit-declarative-hashbrown`, agent name + `declarative-hashbrown-demo` + +## Test Steps + +### 1. Page load + +- [ ] Navigate to `/demos/declarative-hashbrown` +- [ ] Header "Declarative UI: Hashbrown" visible +- [ ] Short description mentioning `@hashbrownai/react` visible +- [ ] Chat composer visible at the bottom of the chat area +- [ ] 3 suggestion pills visible inside the composer with labels: + "Sales dashboard", "Revenue by category", "Expense trend" +- [ ] No red console errors (amber hydration warnings tolerated) + +### 2. Sales dashboard suggestion + +- [ ] Click the "Sales dashboard" pill +- [ ] The prompt is dispatched automatically (useConfigureSuggestions sends + the message on pill click) +- [ ] Within 45 seconds, at least one MetricCard (`data-testid="metric-card"`) + renders in the transcript +- [ ] Within 45 seconds, at least one chart + (`data-testid="bar-chart"` or `data-testid="pie-chart"`) renders +- [ ] At least one Markdown heading renders inline (the prompt's worked + example leads with `## Q4 Sales Summary`) +- [ ] Rendered content streams progressively — partial UI appears before the + full response completes (optional visual check) + +### 3. Revenue by category + +- [ ] Click "Revenue by category" +- [ ] Within 45s, a pie chart (`data-testid="pie-chart"`) renders +- [ ] Legend shows at least 4 segments with readable labels and values + +### 4. Expense trend + +- [ ] Click "Expense trend" +- [ ] Within 45s, a bar chart (`data-testid="bar-chart"`) renders +- [ ] Chart has at least 3 bars with month-like labels + +### 5. Free-form prompt + +- [ ] Type "Show me revenue trends for the last six months" and press Enter +- [ ] Verify at least one catalog component renders (metric, chart, or deal) + +### 6. Deal card + +- [ ] Ask "Show me a sample deal in the negotiation stage" +- [ ] Verify a `data-testid="hashbrown-deal-card"` renders with a stage badge + reading `negotiation` and a `$`-formatted value + +### 7. Multi-turn + +- [ ] After a first render completes, send a follow-up prompt + (e.g. "Now break it down by region") +- [ ] A new render appears alongside prior renders in the transcript + +### 8. Error handling + +- [ ] Empty send is a no-op (button stays disabled) +- [ ] Console remains clean during successful flows + +## Expected Results + +- Suggestion pills produce a hashbrown render within 45 seconds +- The backend emits the JSON envelope (`{"ui": [...]}`), NEVER the `` XML + form — the XML in `useUiKit({ examples })` is hashbrown's prompt DSL, not + the wire format this demo consumes +- Streaming renders assemble progressively as JSON chunks arrive +- No uncaught errors; no `HashBrownRenderMessage must be used within +HashBrownDashboard` errors +- Multi-turn works without clearing prior renders diff --git a/showcase/integrations/claude-sdk-python/qa/declarative-json-render.md b/showcase/integrations/claude-sdk-python/qa/declarative-json-render.md new file mode 100644 index 0000000000..7c52d30910 --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/declarative-json-render.md @@ -0,0 +1,68 @@ +# QA: Declarative UI — json-render — Claude Agent SDK (Python) + +## Prerequisites + +- Demo deployed at `/demos/declarative-json-render` +- Agent backend healthy (`GET /api/health` returns + `{"status":"ok","integration":"claude-sdk-python"}`) +- `ANTHROPIC_API_KEY` set in the deployment environment (`ANTHROPIC_MODEL` + defaults to `claude-opus-4-8`) +- `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI server + exposing `POST /declarative-json-render` (`src/agent_server.py`, prompt in + `src/agents/byoc_json_render_agent.py`) +- `@json-render/core` + `@json-render/react` present in `package.json` + (pinned to `0.18.0`) +- Frontend runtime: `/api/copilotkit-declarative-json-render`, agent name + `byoc_json_render` + +## Test Steps + +### 1. Page load + +- [ ] Navigate to `/demos/declarative-json-render`. +- [ ] The chat surface loads inside the centered `max-w-4xl` container. +- [ ] Chat composer is visible. +- [ ] Three suggestion pills appear with titles: "Sales dashboard", "Revenue by category", "Expense trend". +- [ ] No console errors. + +### 2. Sales dashboard suggestion + +- [ ] Click the "Sales dashboard" suggestion. +- [ ] Within 60 seconds, a `data-testid="json-render-root"` wrapper appears in the assistant bubble. +- [ ] A `data-testid="metric-card"` renders inside the wrapper. +- [ ] A chart (`data-testid="bar-chart"` or `data-testid="pie-chart"`) renders inside the wrapper. +- [ ] No raw JSON text is shown once rendering finishes — the streaming JSON is replaced by components. + +### 3. Revenue by category + +- [ ] Click the "Revenue by category" suggestion. +- [ ] Within 60 seconds, a `data-testid="pie-chart"` renders with multiple category slices + legend. + +### 4. Expense trend + +- [ ] Click the "Expense trend" suggestion. +- [ ] Within 60 seconds, a `data-testid="bar-chart"` renders with month labels. + +### 5. Free-form prompt + +- [ ] Type "Show me a metric for quarterly revenue" and send. +- [ ] Verify at least one `metric-card` renders; no console errors. + +### 6. Multi-turn + +- [ ] After a previous render is visible, send a follow-up prompt ("Now break that down by region"). +- [ ] A new assistant message appears with a new json-render rendering — prior renders stay in the transcript. + +### 7. Malformed output handling + +- [ ] Force non-spec output by asking "tell me a joke". The renderer's + `parseSpec` returns null for anything that is not a `{ root, elements }` + object whose element `type`s are all in `MetricCard` / `BarChart` / + `PieChart`, so the chat falls back to the default + `CopilotChatAssistantMessage` bubble. No crash, no stuck spinner. + +## Expected Results + +- Suggestion renders land within 60 seconds. Budget is slightly higher than the hashbrown demo because a JSON `{ root, elements }` spec is more verbose than hashbrown's token stream. +- No uncaught errors in the console. +- Streaming falls back to plain text until the JSON parses, then swaps to rendered components wrapped in a single `` (no missing-provider crashes). diff --git a/showcase/integrations/claude-sdk-python/qa/gen-ui-interrupt.md b/showcase/integrations/claude-sdk-python/qa/gen-ui-interrupt.md new file mode 100644 index 0000000000..d932992cc1 --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/gen-ui-interrupt.md @@ -0,0 +1,76 @@ +# QA: In-Chat HITL via useInterrupt — Claude Agent SDK (Python) + +> **STATUS: QUARANTINED.** `gen-ui-interrupt` is listed under +> `not_supported_features` in `manifest.yaml` (alongside `interrupt-headless`), +> with this reason: turn-2 fails on a `useInterrupt` / `useHeadlessInterrupt` +> **resume-path** bug in `@copilotkit/react-core/v2` — the backend resumes and +> streams (HTTP 200) but the frontend never appends the confirmation assistant +> bubble, so the harness DOM settle-check times out. The fix is a +> published-package change. The reference integration (langgraph-python) +> quarantines the same cell for the same reason. The demo stays fully wired. +> +> Sections 1–2 below are verifiable today. Section 3 is the **re-qualification +> checklist**: do not tick it, and do not report the cell green off the back of +> it — run it only once the upstream react-core fix lands, and report the result +> to whoever owns the manifest. + +## Prerequisites + +- Demo is deployed and accessible at `/demos/gen-ui-interrupt` on the dashboard host +- Next.js host is healthy (`GET /api/health`) and the Python backend is reachable (`GET /api/copilotkit` reports `agent_status: "reachable"`); `ANTHROPIC_API_KEY` is set. `ANTHROPIC_MODEL` is optional (`.env.example` sets `claude-opus-4-8`, which is also the in-code fallback in `src/agents/interrupt_agent.py`) +- `AGENT_URL` (default `http://localhost:8000`) points at `src/agent_server.py`. This demo goes through the SHARED runtime `/api/copilotkit`: `dedicatedAgentPaths` in `src/app/api/copilotkit/route.ts` maps agent name `gen-ui-interrupt` → `${AGENT_URL}/interrupt-adapted` → `run_interrupt_agent`. There is **no** LangGraph deployment and no `interrupt()` primitive in this integration +- **How this integration adapts the demo.** The Claude Agent SDK has no LangGraph checkpoint/resume `interrupt()`. `src/agents/interrupt_agent.py` instead forwards the frontend tool definitions it receives in `input_data.tools` straight to Claude and, per its `SYSTEM_PROMPT` ("you MUST call the `schedule_meeting` tool" with a `topic` and optional `attendee`), emits a `schedule_meeting` tool call. Two facts to hold onto while testing, both verified in this integration's source: + - the Python backend emits **no** AG-UI interrupt signal — no `on_interrupt` custom event and no `RUN_FINISHED` `outcome: "interrupt"` — anywhere in `src/agents/` + - `src/app/demos/gen-ui-interrupt/page.tsx` registers **no** `useFrontendTool` / `useRenderTool`; it wires only `useInterrupt({ agentId: "gen-ui-interrupt", renderInChat: true, render })` + So `useInterrupt`'s `render` callback has no event source here, and the picker card is expected NOT to mount. If it DOES mount, that is new information — record it and escalate, because it changes the quarantine rationale +- The picker component is `src/app/demos/gen-ui-interrupt/_components/time-picker-card.tsx` (testids `time-picker-card`, `time-picker-slot`, `time-picker-cancel`, `time-picker-picked`, `time-picker-cancelled`); fallback slot labels come from `src/app/demos/_shared/interrupt-fallback-slots.ts` + +## Test Steps + +### 1. Basic Functionality (verifiable today) + +- [ ] Navigate to `/demos/gen-ui-interrupt`; verify the page renders within 3s with the `CopilotChat` centered in a `max-w-4xl` container filling full viewport height, `rounded-2xl` +- [ ] Verify the `CopilotChat` input placeholder is visible and the transcript is empty on first load +- [ ] Send "Hello" and verify the agent responds with a text-only reply (no picker — the prompt only instructs `schedule_meeting` for booking/scheduling requests) + +### 2. Feature-Specific Checks (verifiable today) + +#### Suggestions + +- [ ] Verify both suggestion pills are visible with verbatim titles: + - "Book a call with sales" (message: "Book an intro call with the sales team to discuss pricing.") + - "Schedule a 1:1 with Alice" (message: "Schedule a 1:1 with Alice next week to review Q2 goals.") + +#### Turn 1 — Backend tool call reaches the client + +- [ ] Click "Book a call with sales" +- [ ] DevTools → Network: verify the request goes to `/api/copilotkit` and the SSE stream carries `TOOL_CALL_START` / `TOOL_CALL_ARGS` / `TOOL_CALL_END` for `schedule_meeting` with a `topic` (and `attendee` where the prompt names one) — this is the backend half of the adaptation and it should work +- [ ] Record whether a `data-testid="time-picker-card"` element mounts inside the chat transcript. Per the Prerequisites, the expected answer on this integration is **no** — the tool call arrives with no interrupt signal and no frontend handler, so the transcript shows the assistant text and however `CopilotChat` renders an unhandled tool call. Note down exactly what you see — that observation is the useful output of this step +- [ ] Verify the page does not crash: no uncaught console errors, no blank pane, chat input still accepts a second message + +#### Contract Check — Interrupt Is Low-Level + +- [ ] Confirm a plain conversational message ("What's the weather?") does not render a picker and does not call `schedule_meeting` +- [ ] Confirm no approval-dialog-style modal appears at any point (this demo is inline, not modal — contrast `hitl-in-app`, which portals a modal) + +### 3. Re-qualification checklist — BLOCKED, do not tick + +Run only after the `@copilotkit/react-core/v2` resume-path fix ships AND an interrupt signal exists on this backend. These are the reference behaviors this cell owes; each one is currently unreachable. + +- Picker renders INLINE in the transcript (`time-picker-card`), a descendant of the chat container, NOT portaled to `` +- Card header shows the outline badge "Book a call", the agent-supplied topic as the title, "With " when present, and the description "Pick a time that works for you." +- A 2-column grid of `time-picker-slot` buttons; with no backend-supplied slots the fallback labels are "Tomorrow 10:00 AM", "Tomorrow 2:00 PM", "Monday 9:00 AM", "Monday 3:30 PM" +- A ghost `time-picker-cancel` button labeled "None of these work" below the grid +- Pick path: card switches to `time-picker-picked` (green-tinted, "Booked" badge + bold slot label), all buttons disable, and `resolve({chosen_time, chosen_label})` fires after the deliberate 500ms commit delay in `page.tsx` +- Cancel path: card switches to `time-picker-cancelled` ("Cancelled" badge + "No time picked.") and `resolve({cancelled: true})` fires +- **Turn 2 (the quarantined step):** the agent resumes and appends a confirmation assistant bubble naming the chosen slot, or noting the cancellation. This is exactly what the react-core bug drops +- Multi-turn: a fresh independent picker renders for a follow-up booking prompt while the earlier card stays in its resolved state +- Double-click a slot button rapidly: only one selection commits + +## Expected Results + +- Chat loads within 3 seconds; plain-text response within 10 seconds +- A booking prompt produces a `schedule_meeting` tool call on the wire within 20 seconds +- No inline picker card, and therefore no pick/cancel/resume flow, while the cell is quarantined — this is the accepted outcome, honestly marked skipped-incapable in `manifest.yaml` (not green, not red) +- No UI layout breaks and no uncaught console errors at any point +- Anything that contradicts the two verified facts in the Prerequisites (a picker that mounts, an `on_interrupt` event on the wire) is a finding worth reporting, not a pass diff --git a/showcase/integrations/claude-sdk-python/qa/interrupt-headless.md b/showcase/integrations/claude-sdk-python/qa/interrupt-headless.md new file mode 100644 index 0000000000..4f05ebfc53 --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/interrupt-headless.md @@ -0,0 +1,28 @@ +# QA: Interrupt (Headless) — Claude Agent SDK (Python) + +> Stub — authored for column completeness. This is a testing-kind demo +> (see `kind: "testing"` in `showcase/shared/feature-registry.json`) and +> does not warrant a full manual checklist. The cell is additionally +> quarantined under `not_supported_features` in `manifest.yaml` (shared +> upstream `@copilotkit/react-core/v2` resume-path defect). + +## Prerequisites + +- Demo is deployed and accessible +- Agent backend is healthy; `ANTHROPIC_API_KEY` set. The cell uses the shared + `/api/copilotkit` runtime with agent `interrupt-headless`, mapped to the + FastAPI `POST /interrupt-adapted` endpoint (`src/agents/interrupt_agent.py`) + +## Test Steps + +- [ ] Navigate to /demos/interrupt-headless and verify the left app surface + (`data-testid="interrupt-headless-app-surface"`) shows the empty state + (`data-testid="interrupt-headless-empty"`, "Nothing scheduled yet") +- [ ] Send a scheduling prompt (e.g. the "Book a call with sales" pill) and verify a time-slot picker popup (`data-testid="interrupt-headless-popup"`) appears in the left app surface, not in the chat +- [ ] Click one of the time-slot buttons and verify the popup disappears and the agent confirms the booking back in the chat + +## Expected Results + +- Page loads without errors +- Interrupt resolves via the plain button grid (no `useInterrupt` render prop, no in-chat picker) and the agent continues the run with the picked slot +- Known deviation: the turn-2 confirmation bubble may never append — that is the quarantined react-core resume-path bug, not an integration regression diff --git a/showcase/integrations/claude-sdk-python/qa/mcp-apps.md b/showcase/integrations/claude-sdk-python/qa/mcp-apps.md new file mode 100644 index 0000000000..e21346c861 --- /dev/null +++ b/showcase/integrations/claude-sdk-python/qa/mcp-apps.md @@ -0,0 +1,64 @@ +# QA: MCP Apps — Claude Agent SDK (Python) + +## Prerequisites + +- Demo is deployed and accessible at `/demos/mcp-apps` on the dashboard host +- Agent backend is healthy; `ANTHROPIC_API_KEY` is set on the deployment (`ANTHROPIC_MODEL` defaults to `claude-opus-4-8`); `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI server exposing `POST /mcp-apps` (`src/agent_server.py`, handler in `src/agents/mcp_apps_agent.py`), registered as agent name `mcp-apps` — see `src/app/api/copilotkit-mcp-apps/route.ts` +- MCP server target: the public Excalidraw MCP app at `https://mcp.excalidraw.com` (override via `MCP_SERVER_URL`). Pinned `serverId: "excalidraw"` so URL changes don't silently break persisted activities +- Note: the demo source contains no `data-testid` attributes and registers no custom activity renderer — CopilotKit's built-in `MCPAppsActivityRenderer` handles the sandboxed iframe automatically. Checks below rely on verbatim visible text, network traffic, and the iframe DOM + +## Test Steps + +### 1. Basic Functionality + +- [ ] Navigate to `/demos/mcp-apps`; verify the page renders within 3s and a single `CopilotChat` pane is centered (max-width ~896px via `max-w-4xl`, `rounded-2xl`, full-height) +- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-mcp-apps"` and `agent="mcp-apps"` (DevTools → Network: sending a message hits that endpoint) +- [ ] Verify both suggestion pills are visible with verbatim titles: + - "Draw a flowchart" + - "Sketch a system diagram" +- [ ] Send "Hello" and verify an assistant text response appears within 10s (no MCP activity iframe for plain text) + +### 2. Feature-Specific Checks + +#### MCP Server Connection (runtime `mcpApps.servers`) + +- [ ] Send the first flow-chart prompt; in DevTools → Network, verify the POST to `/api/copilotkit-mcp-apps` succeeds (status 200) and the server-side runtime resolves tools from `https://mcp.excalidraw.com` (the MCP Apps middleware appends them to the AG-UI request's `tools` array, which `_build_anthropic_tools` in `src/agents/mcp_apps_agent.py` forwards verbatim to Claude — notably `create_view`) +- [ ] Verify no console errors mentioning the MCP server URL, auth, or tool-schema parse failures + +#### MCP Tool Invocation (`create_view`) + +- [ ] Click "Draw a flowchart"; within 60s verify the agent calls the `create_view` MCP tool exactly ONCE (per `SYSTEM_PROMPT` in `src/agents/mcp_apps_agent.py`: "Call `create_view` ONCE with 3-5 elements total") — confirm via DevTools → Network stream or backend logs +- [ ] Verify the tool payload contains 3-5 Excalidraw elements (shapes + arrows + optional title text), each with a unique string `id`, and ends with ONE `cameraUpdate` sized `600x450` or `800x600` + +#### Activity Renderer (built-in `MCPAppsActivityRenderer`) + +- [ ] Within 60s of the tool call, verify a sandboxed `