mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
chore(showcase/claude-sdk-python): carry the byoc-to-declarative QA rename further
Work in progress on top of the OSS-578 parity branch, committed to preserve it outside the worktree rather than to ship it. Retires the byoc-* QA docs and e2e specs and adds declarative-* replacements, plus new QA docs and specs for a2ui-recovery, a2ui-fixed-schema, beautiful-chat, gen-ui-interrupt, interrupt-headless, mcp-apps, reasoning-default, reasoning-custom, threadid-frontend-tool-roundtrip and tool-rendering-reasoning-chain. declarative-gen-ui.spec.ts is largely rewritten. Also touches agent_server.py, readonly_state_agent_context.py, the copilotkit route and PARITY_NOTES.md. Unreviewed and unverified. PR #6235 is untouched: this goes to its own branch because the local branch and the PR branch have diverged.
This commit is contained in:
Lukas Moschitz
@@ -46,7 +46,8 @@ frontend-aligned with the reference:
|
||||
- **Shared state**: `shared-state-read`, `shared-state-read-write`
|
||||
(dedicated `/shared-state-read-write`, emits `StateSnapshot`),
|
||||
`shared-state-streaming` (per-token deltas),
|
||||
`readonly-state-agent-context`.
|
||||
`readonly-state-agent-context` (dedicated `/readonly-state-agent-context`,
|
||||
`tools=[]` + read-only-context prompt — see "Masking fixes landed").
|
||||
- **Multi-agent**: `subagents` (dedicated `/subagents`, delegations via
|
||||
`STATE_SNAPSHOT`).
|
||||
- **Declarative UI**: `declarative-gen-ui` (A2UI dynamic),
|
||||
@@ -73,9 +74,40 @@ resume-path defect that also affects the reference:
|
||||
`useHeadlessInterrupt`.
|
||||
|
||||
Both demos remain wired (frontend byte-aligned; backend on the shared
|
||||
`/interrupt-adapted` scheduling agent in `interrupt_agent.py`). The identical
|
||||
QUARANTINE comment lives in both this manifest and the langgraph-python
|
||||
manifest.
|
||||
`/interrupt-adapted` scheduling agent in `interrupt_agent.py`).
|
||||
|
||||
### Correction: the upstream bug is NOT the only blocker here
|
||||
|
||||
The shared react-core defect above is real and blocks the reference too, but for
|
||||
THIS integration it is not the whole story, and the earlier wording (which cited
|
||||
only the upstream bug, and framed these as "not regressions in this
|
||||
integration") was incomplete. The two halves of these cells expect DIFFERENT
|
||||
mechanisms:
|
||||
|
||||
- The **backend** (`src/agents/interrupt_agent.py`) implements "Strategy B",
|
||||
mirroring `ms-agent-python`: the Claude Agent SDK has no `interrupt()`
|
||||
primitive, so the agent calls a frontend tool named `schedule_meeting`. Its
|
||||
docstring states "The frontend registers the tool via `useFrontendTool`", and
|
||||
it forwards frontend-defined tools.
|
||||
- The **frontend** is the byte-identical langgraph-python copy and uses
|
||||
`useInterrupt` — the low-level primitive for LangGraph `interrupt(...)`
|
||||
events. Nothing in `gen-ui-interrupt/` or `interrupt-headless/` registers
|
||||
`schedule_meeting` via `useFrontendTool` (grep returns nothing), so the
|
||||
backend's tool call has no client implementation, and no interrupt payload is
|
||||
ever produced for `useInterrupt` to consume.
|
||||
|
||||
The reference is internally consistent (real `interrupt()` in
|
||||
`langgraph-python/src/agents/interrupt_agent.py`, paired with `useInterrupt`).
|
||||
Note that `_shared/interrupt-fallback-slots.ts` does NOT rescue this: it only
|
||||
fills in missing slots WITHIN an interrupt payload, and no such payload arrives.
|
||||
|
||||
Closing this locally would mean either diverging the frontend (violating the
|
||||
near-identical-frontend rule) or teaching the backend to emit real interrupt
|
||||
events — net-new feature work this ticket explicitly excludes. Per the ticket's
|
||||
own rule ("if a declared cell can't go green without new-feature/upstream work,
|
||||
reclassify to honest NSF — never force"), both cells stay NSF, but the reason
|
||||
recorded here is now the accurate one. Even if the react-core fix ships, these
|
||||
two cells will still need local work before they can go green.
|
||||
|
||||
Both are honestly marked skipped-incapable (not green, not red).
|
||||
|
||||
@@ -87,11 +119,35 @@ stack they loop (openGenerativeUI `generateSandboxedUi` re-emits on the
|
||||
follow-up run), but the loop is a **local-repro artifact**: it was ruled out
|
||||
as (a) a fixture issue — the pre-blitz/staging fixture also loops locally, and
|
||||
(b) a `@copilotkit/*` version drift — the container versions match the
|
||||
green langgraph-python container, which passes ogui locally. The exact local
|
||||
delta vs staging was not pinned (the backend's stdout does not surface to
|
||||
`docker logs`, blocking deeper capture). CI (staging-equivalent x86 build)
|
||||
is the arbiter for these two cells. The fixtures here are the canonical
|
||||
staging-green versions (`hasToolResult`-gated follow-up leg).
|
||||
langgraph-python container. The exact local delta vs staging was not pinned
|
||||
(the backend's stdout does not surface to `docker logs`, blocking deeper
|
||||
capture). CI (staging-equivalent x86 build) is the arbiter for these two
|
||||
cells. The fixtures here are the canonical staging-green versions
|
||||
(`hasToolResult`-gated follow-up leg).
|
||||
|
||||
**Correction (measured 2026-08-31):** an earlier version of this note claimed
|
||||
langgraph-python "passes ogui locally", making the failure look
|
||||
claude-sdk-python-specific. That is not true on the current local stack. A
|
||||
control run of the REFERENCE's own cell —
|
||||
`bin/showcase test langgraph-python:open-gen-ui --d6 --direct --isolate …` —
|
||||
also goes RED (`gen-ui-open: feature exceeded 300000ms wall-clock`, with
|
||||
`ERR_NETWORK_IO_SUSPENDED`, `agent_run_failed`, and a streamed-JSON parse
|
||||
error). So the local docker stack cannot adjudicate `open-gen-ui` for ANY
|
||||
integration, the reference included. Treat local ogui results as
|
||||
uninformative and rely on CI/staging, until someone pins the local delta.
|
||||
|
||||
## Note on local D6 vs staging (multimodal)
|
||||
|
||||
`multimodal` also cannot be verified on a local checkout that has not resolved
|
||||
Git LFS. `public/demo-files/sample.png` and `sample.pdf` are LFS pointers; the
|
||||
demo detects this and refuses to send, so the probe fails with
|
||||
`settle-dom-missing` and the page prints "Sample \"sample.png\" is a Git LFS
|
||||
pointer, not the real asset." This is NOT a cell defect — nothing in this
|
||||
integration's `public/` was changed. CI resolves the assets explicitly
|
||||
(`.github/workflows/showcase_validate.yml`:
|
||||
`git lfs pull --include="showcase/integrations/*/public/demo-files/*"`, which
|
||||
hard-errors if the pull no-ops), so the cell is verifiable there. To verify
|
||||
locally you need `git-lfs` installed plus that pull.
|
||||
|
||||
## a2ui-recovery — native recovery loop
|
||||
|
||||
@@ -107,7 +163,7 @@ aimock `sequenceIndex`.
|
||||
|
||||
## Masking fixes landed
|
||||
|
||||
Three cells were aimock-green but were routing to the WRONG agent live (a
|
||||
Four cells were aimock-green but were routing to the WRONG agent live (a
|
||||
generic-fallback prompt masked by the fixture). Each now has a dedicated,
|
||||
correctly-prompted backend so the live behavior matches the fixture:
|
||||
|
||||
@@ -118,16 +174,72 @@ correctly-prompted backend so the live behavior matches the fixture:
|
||||
tailored prompt (was masked generic fallback).
|
||||
3. `hitl-in-app` — dedicated `/hitl-in-app` endpoint applying the HITL prompt
|
||||
via `system_prompt_override` (was masked generic fallback).
|
||||
4. `readonly-state-agent-context` — dedicated `/readonly-state-agent-context`
|
||||
endpoint applying the read-only-context prompt with `tools_override=[]`
|
||||
(mirrors the reference's `tools=[]` graph). This one was the starkest
|
||||
masking case: `src/agents/readonly_state_agent_context.py` was imported
|
||||
NOWHERE, yet `manifest.yaml` lists it under this demo's `highlight:` block
|
||||
and the integration is `docs_mode: generated` — so the published docs
|
||||
presented a module as this cell's backend that the runtime never executed.
|
||||
Verified via aimock's request recorder (`GET /v1/_requests`), which captures
|
||||
the system prompt the fixture matcher ignores:
|
||||
- BEFORE: generic sales-assistant `SYSTEM_PROMPT`, 9 backend tools, and
|
||||
`POST /readonly-state-agent-context` → HTTP 404 (the path did not exist).
|
||||
- AFTER: the read-only-context prompt carrying the probe's context sentinel,
|
||||
0 tools, and the path serving HTTP 200.
|
||||
D6 was green BEFORE and AFTER — which is the point: the probe cannot see
|
||||
this defect (GOTCHAS #8).
|
||||
|
||||
## Verification-surface parity (specs + QA docs)
|
||||
|
||||
`scripts/validate-parity.ts` now reports this integration at **39 demos / 39
|
||||
e2e specs / 40 QA docs with 3 warnings — identical to langgraph-python**, up
|
||||
from 39 / 36 / 31 with 21 warnings. The 3 remaining warnings are byte-identical
|
||||
to the reference's own (`interrupt-headless` has no spec, `reasoning-custom` has
|
||||
no QA doc, and the stale demo-count baseline); closing those only here would
|
||||
break the identical-tests rule.
|
||||
|
||||
Work that got it there:
|
||||
|
||||
- Ported the 4 missing specs (`a2ui-recovery`, `reasoning-custom`,
|
||||
`reasoning-default`, `threadid-frontend-tool-roundtrip`).
|
||||
- Renamed `byoc-hashbrown` / `byoc-json-render` spec+QA files to
|
||||
`declarative-*` to match the demo ids (`validate-parity` keys filenames to
|
||||
demo ids). Python module names KEEP the `byoc_` prefix — the reference does
|
||||
too, and `manifest.yaml` cites them under `highlight:`.
|
||||
- Deleted the orphan `shared-state-write` spec+QA pair (no such demo dir, no
|
||||
manifest entry).
|
||||
- Authored the 10 missing QA docs in the reference's own cut: full checklists
|
||||
where it has them, `> Stub — authored for column completeness` where it
|
||||
deliberately stubs (`interrupt-headless`, `reasoning-default`,
|
||||
`tool-rendering-reasoning-chain`). Every file path, env var, endpoint,
|
||||
`data-testid` and quoted prompt in them was verified to exist.
|
||||
|
||||
### Two specs were not merely drifted — they were dead
|
||||
|
||||
- `declarative-gen-ui.spec.ts` asserted suggestion pills that no longer exist
|
||||
("Show a KPI dashboard", "Pie chart — sales by region", …) against the
|
||||
current Vantage Threads set ("Show my sales dashboard", "Team performance",
|
||||
"Anything at risk?", "Top account details"), and its header comment claimed
|
||||
the demo has no `data-testid` — false: the same seven ids the reference uses
|
||||
are present. It could only ever have failed; nobody noticed because `--d6`
|
||||
never invokes this surface (GOTCHAS #7). Rewritten onto the stable testids.
|
||||
- `beautiful-chat.spec.ts` had the `Catalog not found` regression guard
|
||||
(#4733 / #4734 / #5425) nested inside a conditional, so a regression could
|
||||
slip past. Hoisted to the reference's unconditional placement.
|
||||
|
||||
## Flags / caveats
|
||||
|
||||
- **`readonly-state-agent-context` live-prompt gap**: the reference uses a
|
||||
dedicated tailored-prompt graph; here the cell routes to the generic shared
|
||||
agent (`readonly_state_agent_context.py` is docs-only). D6 and the frontend
|
||||
are correct — context is injected via `useAgentContext` — but the live
|
||||
system prompt is not tailored the way the reference is. Adjudicated at live
|
||||
smoke.
|
||||
- **`declarative-json-render` zod build caveat**: `catalog.ts` was reverted
|
||||
from `zod4` back to `zod` for byte-parity with the reference. If
|
||||
`@json-render` 0.18 turns out to require `zod4` here, the build may break —
|
||||
re-introduce `zod4` and re-flag if so.
|
||||
- **`beautiful-chat` is weaker by FIXTURE, not by test.** After the fix above
|
||||
the test logic is byte-identical to the reference; the one remaining
|
||||
divergence is a comment, and it is deliberate. The reference's fixture omits
|
||||
`catalogId` so the run exercises route-level `defaultCatalogId` resolution —
|
||||
the exact path that regressed in #4733 / #4734 / #5425. This integration's
|
||||
`aimock/d4/claude-sdk-python/chat.json` hardcodes
|
||||
`catalogId: "copilotkit://app-dashboard-catalog"`, so that path is never
|
||||
exercised here. Copying the reference's comment verbatim would have stated
|
||||
something false about our fixture. Our route already configures the same
|
||||
`defaultCatalogId`, so aligning the fixture (dropping the explicit id) is a
|
||||
small, viable follow-up in the fixture lane.
|
||||
- **`declarative-json-render` zod caveat — RESOLVED.** `catalog.ts` imports
|
||||
plain `zod`, matching the reference, which now does the same. No action.
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# QA: Declarative Generative UI (A2UI — Fixed Schema) — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/a2ui-fixed-schema` on the dashboard host
|
||||
- Next.js host is healthy (`GET /api/health` → `{"status":"ok","integration":"claude-sdk-python"}`) and the Python backend is reachable (`GET /api/copilotkit` reports `agent_status: "reachable"`, which it derives by polling `${AGENT_URL}/health`)
|
||||
- `ANTHROPIC_API_KEY` is set — this cell's model calls go to Anthropic, not OpenAI (`OPENAI_API_KEY` is in `.env.example` for the shared stack, but no code path in this demo reads it). `ANTHROPIC_MODEL` is optional; `.env.example` sets `claude-opus-4-8`, and `src/agents/a2ui_fixed.py` falls back to that same id
|
||||
- `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI agent server `src/agent_server.py`, which exposes `@app.post("/a2ui-fixed-schema")` → `run_a2ui_fixed_agent`. There is **no** LangGraph deployment and no graph registration in this integration
|
||||
- Note: unlike the langgraph-python reference, the outer card here **does** carry a stable `data-testid="a2ui-fixed-card"` (see `src/app/demos/a2ui-fixed-schema/a2ui/renderers.tsx`). Everything else below relies on verbatim visible text, DOM structure, and the JSON schema at `src/agents/a2ui_schemas/flight_schema.json`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/a2ui-fixed-schema`; verify the page renders within 3s: a `max-w-4xl` column with `border-x border-neutral-200 bg-white` on a `bg-neutral-50` page, filling full viewport height, with the `CopilotChat` itself `rounded-2xl`
|
||||
- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-a2ui-fixed-schema"` and `agent="a2ui-fixed-schema"` (DevTools → Network: sending a message hits that endpoint, not `/api/copilotkit`)
|
||||
- [ ] Verify the single suggestion pill is visible with verbatim title "Find SFO → JFK" (message body: "Find me a flight from SFO to JFK on United for $289.")
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s (no flight card for plain text)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Schema Wiring (fixed catalog + `includeBasicCatalog`)
|
||||
|
||||
- [ ] DevTools → Network: after the first successful `display_flight` call, verify the tool result contains an `a2ui_operations` container whose `createSurface` carries `surfaceId: "flight-fixed-schema"` and `catalogId: "copilotkit://flight-fixed-catalog"` (matches `SURFACE_ID` / `CATALOG_ID` in `src/agents/a2ui_fixed.py` and `CATALOG_ID` in `src/app/demos/a2ui-fixed-schema/a2ui/catalog.ts`)
|
||||
- [ ] Verify the same container's `updateComponents` carries the full `FLIGHT_SCHEMA` tree (12 nodes from `src/agents/a2ui_schemas/flight_schema.json`: `root`, `content`, `title`, `route`, `from`, `arrow`, `to`, `meta`, `airline`, `price`, `bookButton`, `bookButtonLabel`) and that `updateDataModel` writes `{origin, destination, airline, price}` at path `/`
|
||||
|
||||
#### Search-Flights Prompt (`display_flight` tool → `flight_schema.json`)
|
||||
|
||||
- [ ] Click the "Find SFO → JFK" suggestion; within 20s verify a single flight card renders in-transcript with `data-testid="a2ui-fixed-card"`, assembled per `flight_schema.json`:
|
||||
- outer `Card` (`max-w-md`, 20px padding) wrapping a `Column` of children in this order: title row, route row, meta row, book button
|
||||
- `Title` node renders the eyebrow "Itinerary" above the literal schema text "Flight Details", with an outline mono badge "1-stop · economy" on the right
|
||||
- `route` row shows `Airport` "SFO" → `Arrow` (an SVG chevron flanked by two hairline separators) → `Airport` "JFK" (both monospaced, `text-2xl`, semibold, wide tracking)
|
||||
- `meta` row shows `AirlineBadge` "UNITED" (secondary pill, uppercase, `0.08em` tracking) on the left and `PriceTag` — the eyebrow "Total" followed by monospaced "$289" — on the right
|
||||
- `Button` renders full-width with label "Book flight"
|
||||
- [ ] Verify all four data-model fields resolved correctly (origin=`SFO`, destination=`JFK`, airline=`United`, price=`$289`) — each is a `{ path: "/..." }` binding in the schema and must reach the DOM as a plain string via the binder (no literal `{path}` leak and no React error #31)
|
||||
|
||||
#### Book-Flight Button (inert — pure presentation)
|
||||
|
||||
- [ ] Verify the "Book flight" button renders the schema-declared label and is clickable, but the click is a no-op: the agent is not invoked, no schema swap occurs, and the button does not transition to a "Booked" state. The schema declares an `action` (`book_flight`) purely for fidelity; the `Button` renderer in `a2ui/renderers.tsx` deliberately drops it — schema-swap-on-action waits on the Python SDK exposing `action_handlers=` on `a2ui.render` (see the `Button` comment in `a2ui/renderers.tsx`). `src/agents/a2ui_schemas/booked_schema.json` ships alongside but is not loaded by any code path
|
||||
|
||||
#### Follow-up Prompt (data-model refresh)
|
||||
|
||||
- [ ] Send "Find me a flight from LAX to ORD on Delta for $412."; within 20s verify the card updates in place with origin=`LAX`, destination=`ORD`, airline=`DELTA`, price=`$412` (same schema, new data model — proves the fixed-schema pattern: schema once, data streams)
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Send "What is the capital of France?"; verify the agent replies in plain text without invoking `display_flight` (no flight card rendered, no `a2ui_operations` in the response)
|
||||
- [ ] DevTools → Console: walk through all flows above; verify no uncaught errors and specifically no React error #31 ("objects are not valid as a React child, found: object with keys {path}") — the `DynString` union in `a2ui/definitions.ts` is what prevents this, so a single occurrence is a regression
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3s; plain-text response within 10s; flight card renders within 20s of the search prompt
|
||||
- `display_flight` is called exactly once per search prompt; result contains an `a2ui_operations` container with `catalogId: "copilotkit://flight-fixed-catalog"` and the full 12-node flight schema
|
||||
- All custom renderers in `a2ui/renderers.tsx` (`Card`, `Title`, `Airport`, `Arrow`, `AirlineBadge`, `PriceTag`, `Button`) render at least once per search-flights run
|
||||
- Clicking "Book flight" is a no-op (inert presentation button)
|
||||
- No UI layout breaks, no `{path}` leak into the DOM, no uncaught console errors
|
||||
@@ -0,0 +1,47 @@
|
||||
# QA: A2UI Error Recovery — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/a2ui-recovery` on the dashboard host
|
||||
- Next.js host is healthy (`GET /api/health`); the Python backend is reachable (`GET /api/copilotkit` reports `agent_status: "reachable"`); `ANTHROPIC_API_KEY` is set. `AGENT_URL` (default `http://localhost:8000`) points at `src/agent_server.py`, which exposes `@app.post("/a2ui-recovery")` → `run_a2ui_recovery_agent`. There is **no** LangGraph deployment in this integration
|
||||
- **The recovery loop is NATIVE here.** The langgraph-python reference owns `generate_a2ui` via `ag_ui_langgraph.get_a2ui_tools` and runs the validate→retry loop inside the toolkit. claude-sdk-python uses its own adapter (`ag-ui-claude-sdk` + `claude-agent-sdk`) and does not depend on `ag_ui_langgraph` / `ag_ui_a2ui_toolkit`, so `src/agents/recovery_agent.py` re-implements the loop: `_validate_a2ui_components` (structural checks — `empty_components`, `missing_id`, `missing_component_type`, `unresolved_child`, `no_root`) driven by `_run_render_with_recovery` with `MAX_A2UI_ATTEMPTS = 3`, one inner `render_a2ui` Claude call **per attempt**, and `_wrap_recovery_exhausted_envelope` returning `{"error": …, "code": "a2ui_recovery_exhausted", "attempts": [...]}` on cap
|
||||
- Backend-owned wiring: `src/app/api/copilotkit-a2ui-recovery/route.ts` sets `injectA2UITool: false` (load-bearing — the backend owns `generate_a2ui`, whose only argument is `intent`) plus `defaultCatalogId: "declarative-gen-ui-catalog"`
|
||||
- Reuses the **declarative-gen-ui** catalog (`myCatalog`, `catalogId: "declarative-gen-ui-catalog"`) and the Vantage Threads sales context (`useSalesAnalystContext`) — no new components
|
||||
- The `building` / `retrying` / `failed` lifecycle chrome comes from `@copilotkit/react-core/v2` (`A2UIRecoveryStates.tsx`, mounted by `A2UIMessageRenderer`)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/a2ui-recovery`; verify the page renders within 3s and a single `CopilotChat` pane is centered (`max-w-4xl`, `rounded-2xl`, full viewport height)
|
||||
- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-a2ui-recovery"` and `agent="a2ui-recovery"` (DevTools → Network: sending a message hits that endpoint, not `/api/copilotkit`)
|
||||
- [ ] Verify both suggestion pills are visible with verbatim titles:
|
||||
- "Recover a bad render"
|
||||
- "Show an unrecoverable failure"
|
||||
|
||||
### 2. Healing path
|
||||
|
||||
- [ ] Click "Recover a bad render" ("Build my Q2 revenue summary and self-correct a malformed first attempt.")
|
||||
- [ ] Attempt 1 of the inner `render_a2ui` returns a **structurally invalid** surface (`root` lists a child id that no component defines → `unresolved_child`). Verify the native loop rejects it, retries, and attempt 2 paints — no broken surface, no error banner. A "Building interface" skeleton is expected; the "Retrying generation… (N/M attempts)" sub-label is threshold-gated (2 attempts / 2000ms) so it may or may not become visible — do not fail the run on its absence
|
||||
- [ ] Verify the **painted** surface is valid: at least two `declarative-metric` tiles — "QUARTERLY REVENUE / $4.2M / ↑ +12% QoQ" and "WIN RATE / 31% / ↓ -2 pts"
|
||||
- [ ] DevTools → Network: verify the `generate_a2ui` tool result carries an `a2ui_operations` container (and **no** `a2ui_recovery_exhausted`)
|
||||
- [ ] Verify the chat reply is one short sentence noting the heal
|
||||
- [ ] If backend stdout is reachable, verify two `[a2ui recovery] attempt N: …` log lines — attempt 1 `invalid` with an `unresolved_child` error, attempt 2 `valid` (emitted by `_log_attempt` on the `agents.recovery_agent` logger)
|
||||
|
||||
### 3. Hard-fail (recovery exhausted) path
|
||||
|
||||
- [ ] Click "Show an unrecoverable failure" ("Build a report that fails every validation pass so I can preview the fallback.")
|
||||
- [ ] Verify the lifecycle ends in the tasteful `failed` card — amber panel reading "Couldn't generate the UI" over "Something went wrong rendering this. You can keep chatting and try again." — and NOT a broken/half-rendered surface, and NOT a silent drop. No new `declarative-metric` tile may appear for this pill
|
||||
- [ ] DevTools → Network: verify `render_a2ui` was attempted up to the cap (3 attempts, all invalid) and the `generate_a2ui` result is an `a2ui_recovery_exhausted` envelope with a 3-entry `attempts` array (no `a2ui_operations` painted)
|
||||
- [ ] Verify the chat reply gracefully explains the fallback (one short sentence)
|
||||
|
||||
### 4. Regression / isolation
|
||||
|
||||
- [ ] Verify the recovery demo does not affect the `declarative-gen-ui` or `beautiful-chat` demos (separate routes and agents, even though the catalog is shared)
|
||||
- [ ] Re-run each pill a second time and verify the same lifecycle
|
||||
|
||||
## Notes
|
||||
|
||||
- The malformed renders are forced by aimock fixtures (`showcase/aimock/d6/claude-sdk-python/a2ui-recovery.json`): the inner `render_a2ui` calls are matched by `userMessage` + `toolName=render_a2ui` (+ `sequenceIndex` 0/1 for the heal pill's two attempts), and the outer narration by the emit's unique `toolCallId`. The retry DECISION is made live by the native loop in `recovery_agent.py` — the fixture only supplies the render args.
|
||||
- The pill prompts are unique per integration on purpose: the inner `render_a2ui` calls carry no `x-aimock-context`, so identical prompts across integrations would collide in the shared aimock matcher. Keep `src/app/demos/a2ui-recovery/suggestions.ts` in sync with `showcase/harness/src/probes/scripts/d5-a2ui-recovery.ts`.
|
||||
- Heads-up on a stale comment: `suggestions.ts` still describes the heal as `parse_and_fix` healing sloppy JSON-string args in one pass. That is the toolkit's mechanism, not this integration's. Here the heal is a genuine **invalid → retry → valid** two-attempt loop; the fixture `_comment` fields are authoritative.
|
||||
@@ -0,0 +1,93 @@
|
||||
# QA: Beautiful Chat — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/beautiful-chat` on the dashboard host
|
||||
- Agent backend is healthy: `GET /api/health` returns `{"status":"ok","integration":"claude-sdk-python"}`, and `GET /api/copilotkit` reports `agent_status: "reachable"` (it polls `${AGENT_URL}/health` on the FastAPI backend)
|
||||
- `ANTHROPIC_API_KEY` is set on the deployment; `ANTHROPIC_MODEL` defaults to `claude-opus-4-8`; `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI server exposing `POST /beautiful-chat` (`src/agent_server.py`)
|
||||
- Note: the demo source contains no `data-testid` attributes. Checks below rely on verbatim visible text and DOM structure.
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/beautiful-chat`; verify the page renders within 3s with the "CopilotKit" wordmark plus the logo mark (`img[alt="CopilotKit"]`, src `/copilotkit-logo-mark.svg`) top-left of the chat pane
|
||||
- [ ] Verify the `Chat` / `App` mode pill is fixed top-right, `Chat` active (highlighted) by default, and the right-side canvas region is collapsed (width 0)
|
||||
- [ ] Verify the `CopilotChat` input is rendered with no disclaimer text below it, and that attachments are enabled
|
||||
- [ ] Verify all 9 suggestion pills are visible with verbatim titles (`showcase.json` is `"default"`, so no pill carries a highlight class):
|
||||
- "Pie Chart (Controlled Generative UI)"
|
||||
- "Bar Chart (Controlled Generative UI)"
|
||||
- "Schedule Meeting (Human In The Loop)"
|
||||
- "Search Flights (A2UI Fixed Schema)"
|
||||
- "Sales Dashboard (A2UI Dynamic)"
|
||||
- "Excalidraw Diagram (MCP App)"
|
||||
- "Calculator App (Open Generative UI)"
|
||||
- "Toggle Theme (Frontend Tools)"
|
||||
- "Task Manager (Shared State)"
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s
|
||||
- [ ] DevTools → Network: verify the send POSTs to `/api/copilotkit-beautiful-chat` (the dedicated runtime, not the shared `/api/copilotkit`)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Mode Toggle (frontend tools `enableAppMode` / `enableChatMode`)
|
||||
|
||||
- [ ] Click `App`; verify the canvas expands to ~2/3 width showing the TodoList empty state: pencil emoji, heading "No todos yet", subtext "Create your first task to get started", enabled "Add a task" button
|
||||
- [ ] Click `Chat`; verify the canvas collapses back to width 0
|
||||
|
||||
#### Shared State — Task Manager (agent tools `manage_todos`, `get_todos`)
|
||||
|
||||
- [ ] Click the "Task Manager (Shared State)" pill; verify the mode auto-switches to App (the system prompt in `src/agents/agent.py` routes todos through `enableAppMode` first) and within 15s the "To Do" column renders exactly 3 todo cards (each with emoji, title, description)
|
||||
- [ ] Verify the "Done" column is empty (shows "No completed todos yet")
|
||||
- [ ] Click a todo's checkbox; verify the card moves from "To Do" to "Done"
|
||||
|
||||
#### Controlled Generative UI — Pie Chart (agent tool `query_data` + frontend component `pieChart`)
|
||||
|
||||
- [ ] Click "Pie Chart (Controlled Generative UI)"; within 15s verify a pie-chart card renders in-transcript with a non-empty `CardTitle` and `CardDescription`
|
||||
- [ ] Verify the donut SVG renders at least 2 `<circle>` slice elements inside the card
|
||||
- [ ] Verify the legend renders one row per slice: colored dot, label, comma-formatted value, and a percentage ending in "%"; percentages sum to 100%
|
||||
|
||||
#### Controlled Generative UI — Bar Chart (agent tool `query_data` + frontend component `barChart`)
|
||||
|
||||
- [ ] Click "Bar Chart (Controlled Generative UI)"; within 15s verify a bar-chart card renders with `CardTitle`, `CardDescription`, and a bar-chart icon in the header
|
||||
- [ ] Verify the recharts `ResponsiveContainer` (height 280px) renders at least 2 bar rectangles with X-axis labels matching the `label` field values; bars animate in via the `barSlideIn` keyframe on first render
|
||||
|
||||
#### Human-in-the-Loop — Schedule Meeting (frontend tool `scheduleTime`)
|
||||
|
||||
- [ ] Click "Schedule Meeting (Human In The Loop)"; within 15s verify a MeetingTimePicker card renders with a clock icon, a heading (agent-supplied `reasonForScheduling` or default "Schedule a Meeting"), 3 time-slot buttons each with date + time + a "30 min" duration badge, and a "None of these work" ghost button
|
||||
- [ ] Click a time slot; verify the card switches to the confirmed state with heading "Meeting Scheduled", the chosen date/time, and a green check icon
|
||||
- [ ] Re-trigger, click "None of these work"; verify the card shows heading "No Time Selected" and subtext "Looking for a better time that works for you"
|
||||
|
||||
#### A2UI Fixed Schema — Search Flights (agent tool `search_flights`)
|
||||
|
||||
- [ ] Click "Search Flights (A2UI Fixed Schema)"; within 20s verify exactly 2 flight cards render in-transcript, each with airline name, airline logo image, flight number, origin/destination, date, departure/arrival times, duration, a colored status dot, a status label (e.g. "On Time"), and a price
|
||||
- [ ] Note: the beautiful-chat runtime sets `injectA2UITool: false` with `defaultCatalogId: "copilotkit://app-dashboard-catalog"` — the same id the `demonstrationCatalog` registers, so cards must resolve against the local catalog rather than an injected tool
|
||||
|
||||
#### A2UI Dynamic — Sales Dashboard (agent tool `generate_a2ui`)
|
||||
|
||||
- [ ] Click "Sales Dashboard (A2UI Dynamic)"; within 30s verify a dynamic dashboard surface renders containing total-revenue metric, new-customers metric, conversion-rate metric, a pie chart (revenue by category), and a bar chart (monthly sales)
|
||||
|
||||
#### MCP App — Excalidraw Diagram
|
||||
|
||||
- [ ] Click "Excalidraw Diagram (MCP App)"; within 30s verify an Excalidraw embed renders a diagram with a router, 2 switches, and 4 computers (no console errors referencing `MCP_SERVER_URL`, default `https://mcp.excalidraw.com`, pinned `serverId: "excalidraw"`)
|
||||
|
||||
#### Open Generative UI — Calculator App (`generateSandboxedUi`)
|
||||
|
||||
- [ ] Click "Calculator App (Open Generative UI)"; within 30s verify a sandboxed calculator UI renders with digit/operator buttons plus labeled metric shortcut buttons
|
||||
- [ ] Click a metric shortcut button; verify its value is inserted into the calculator display
|
||||
|
||||
#### Frontend Tool — Toggle Theme (`toggleTheme`)
|
||||
|
||||
- [ ] Click "Toggle Theme (Frontend Tools)"; verify the `html` element's `class` attribute flips between containing `dark` and containing `light` (the `ThemeProvider` removes both classes then adds the active one)
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Attempt to send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Send a ~500-character message; verify it wraps in-transcript without horizontal scroll or layout break
|
||||
- [ ] With the FastAPI backend stopped, send a message; verify the UI surfaces a visible error path rather than hanging silently, and DevTools → Console shows no uncaught errors during any flow above
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds; plain-text response within 10 seconds
|
||||
- Controlled charts (pie/bar) render within 15 seconds of prompt; A2UI surfaces within 20–30 seconds
|
||||
- No UI layout breaks, no flash of unstyled content, no uncaught console errors
|
||||
- All 5 agent tools reachable via `BEAUTIFUL_CHAT_TOOLS` in `src/agents/agent.py` (`query_data`, `search_flights`, `generate_a2ui`, `manage_todos`, `get_todos`) are exercised by at least one check above
|
||||
@@ -1,49 +0,0 @@
|
||||
# QA: BYOC hashbrown — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
- `ANTHROPIC_API_KEY` is set on the deployment
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/byoc-hashbrown`
|
||||
- [ ] Verify the header "BYOC: Hashbrown" renders
|
||||
- [ ] Verify the description paragraph mentions `@hashbrownai/react`
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Q4 Sales Summary (mixed catalog)
|
||||
|
||||
- [ ] Send "Show me a Q4 sales summary" (or click a suggestion)
|
||||
- [ ] Verify a `data-testid="metric-card"` renders with a formatted value
|
||||
- [ ] Verify a `data-testid="pie-chart"` renders with at least three
|
||||
legend rows
|
||||
- [ ] Verify a `data-testid="bar-chart"` renders with at least three
|
||||
columns
|
||||
- [ ] Verify at least one Markdown heading renders inline
|
||||
|
||||
#### Deal card
|
||||
|
||||
- [ ] Ask "Show me a sample deal in the negotiation stage"
|
||||
- [ ] Verify a `data-testid="hashbrown-deal-card"` renders with a stage
|
||||
badge
|
||||
|
||||
### 3. Streaming behaviour
|
||||
|
||||
- [ ] Observe components progressively appear as Claude streams the
|
||||
JSON envelope — no full-refresh flash at the end of streaming
|
||||
|
||||
### 4. Error Handling
|
||||
|
||||
- [ ] No console errors during normal usage.
|
||||
- [ ] No hashbrown schema-validation errors logged.
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Agent responds within 15 seconds
|
||||
- Backend emits the JSON envelope (`{ui: [...]}`), NEVER XML
|
||||
@@ -1,59 +0,0 @@
|
||||
# QA: BYOC json-render — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
- `ANTHROPIC_API_KEY` is set on the deployment
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/byoc-json-render`
|
||||
- [ ] Verify the chat surface loads inside the centered 4xl container
|
||||
- [ ] Verify the three suggestion pills are visible:
|
||||
"Sales dashboard", "Revenue by category", "Expense trend"
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Sales dashboard (MetricCard + BarChart)
|
||||
|
||||
- [ ] Click the "Sales dashboard" suggestion pill
|
||||
- [ ] Verify a `data-testid="metric-card"` element renders with a label
|
||||
and a dollar-formatted value
|
||||
- [ ] Verify a `data-testid="bar-chart"` element renders inside the same
|
||||
`data-testid="json-render-root"` wrapper
|
||||
|
||||
#### Revenue by category (PieChart)
|
||||
|
||||
- [ ] Click "Revenue by category"
|
||||
- [ ] Verify a `data-testid="pie-chart"` element renders with at least
|
||||
three legend rows
|
||||
|
||||
#### Expense trend (BarChart)
|
||||
|
||||
- [ ] Click "Expense trend"
|
||||
- [ ] Verify `data-testid="bar-chart"` renders with three months of data
|
||||
|
||||
### 3. Streaming behaviour
|
||||
|
||||
- [ ] Observe the raw JSON streaming into the chat bubble briefly while
|
||||
the model emits the spec
|
||||
- [ ] Verify the catalog components swap in cleanly once the JSON
|
||||
becomes valid — no flicker, no duplicate render
|
||||
|
||||
### 4. Error Handling
|
||||
|
||||
- [ ] Ask a free-form question that has nothing to do with dashboards
|
||||
(e.g. "What is 2+2?"). The agent should still reply with a JSON
|
||||
spec — it may emit a single MetricCard — and the page must NOT
|
||||
white-screen.
|
||||
- [ ] No console errors during normal usage.
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Agent responds within 15 seconds (Claude opus)
|
||||
- Components render from the json-render catalog wrapped in a single
|
||||
`<JSONUIProvider>` (no missing-provider crashes)
|
||||
@@ -0,0 +1,76 @@
|
||||
# QA: Declarative Generative UI (A2UI — Dynamic Schema) — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/declarative-gen-ui` on the dashboard host
|
||||
- Next.js host is healthy (`GET /api/health`) and the Python backend is reachable (`GET /api/copilotkit` reports `agent_status: "reachable"`, derived from `${AGENT_URL}/health`)
|
||||
- `ANTHROPIC_API_KEY` is set — this cell's model calls go to Anthropic, not OpenAI. `ANTHROPIC_MODEL` is optional; `.env.example` sets `claude-opus-4-8` and `src/agents/a2ui_dynamic.py` falls back to the same id for BOTH the outer agent call and the inner design call
|
||||
- `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI agent server `src/agent_server.py`, which exposes `@app.post("/declarative-gen-ui")` → `run_a2ui_dynamic_agent`. There is **no** LangGraph deployment and no graph registration in this integration
|
||||
- Backend-owned wiring: `src/app/api/copilotkit-declarative-gen-ui/route.ts` sets `injectA2UITool: false` (the backend owns `generate_a2ui`) plus `defaultCatalogId: "declarative-gen-ui-catalog"`. `generate_a2ui(context)` in `src/agents/a2ui_dynamic.py` runs a SECOND Claude call forced onto the `render_a2ui` schema via `tool_choice`, then converts the args with `build_a2ui_operations_from_tool_call` from `tools/generate_a2ui.py` (a symlink to `showcase/shared/python/tools/generate_a2ui.py`)
|
||||
- The demo plays a sales analyst for the fictional **Vantage Threads** company. The dataset and per-question composition rules are registered as agent context in `src/app/demos/declarative-gen-ui/sales-context.ts` — surfaces should reflect those numbers ($4.2M Q2 revenue, 4 regions, 5 reps, 3 at-risk accounts, Meridian Apparel Group as top account)
|
||||
- Each custom renderer carries a stable `data-testid`: `declarative-card`, `declarative-metric`, `declarative-pie-chart`, `declarative-bar-chart`, `declarative-status-badge`, `declarative-data-table`, `declarative-info-row` (see `src/app/demos/declarative-gen-ui/a2ui/renderers.tsx`)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/declarative-gen-ui`; verify the page renders within 3s and a single `CopilotChat` pane is centered (max-width ~896px / `max-w-4xl`, `rounded-2xl`, full viewport height)
|
||||
- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-declarative-gen-ui"` and `agent="declarative-gen-ui"` (DevTools → Network: sending a message hits that endpoint, not `/api/copilotkit`)
|
||||
- [ ] Verify all 4 suggestion pills are visible with verbatim titles:
|
||||
- "Show my sales dashboard"
|
||||
- "Team performance"
|
||||
- "Anything at risk?"
|
||||
- "Top account details"
|
||||
- [ ] Verify no pill mentions a chart type — chart steering lives in `SYSTEM_PROMPT` (`src/agents/a2ui_dynamic.py`) and in the `COMPOSITION_RULES` context entry, not in the user prompt (OSS-136)
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s (no A2UI surface rendered for plain text)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Catalog Wiring (provider `a2ui={{ catalog: myCatalog }}`)
|
||||
|
||||
- [ ] DevTools → Network: on the first tool-driven response, verify the `generate_a2ui` tool result contains an `a2ui_operations` container with `catalogId: "declarative-gen-ui-catalog"` (matches `createCatalog(..., { catalogId: "declarative-gen-ui-catalog" })` in `a2ui/catalog.ts` and the route's `defaultCatalogId`)
|
||||
- [ ] Verify only ONE `generate_a2ui` tool call is emitted per surface-producing prompt — the runtime must not inject a second A2UI tool on top of the backend's (that is what `injectA2UITool: false` prevents)
|
||||
|
||||
#### Hero Pill — Composed Sales Dashboard
|
||||
|
||||
- [ ] Click "Show my sales dashboard" ("Show me my sales dashboard for this quarter."); within 60s verify ONE composed surface renders containing ALL of (no surrounding `declarative-card` — the charts carry their own card chrome):
|
||||
- a bare row of 4 `declarative-metric` KPI tiles (uppercase label, 1.5rem value, trend arrow with delta — green `↑` `#059669` for up, red `↓` `#dc2626` for down, e.g. "↑ +12% QoQ")
|
||||
- a `declarative-pie-chart` (recharts donut, `innerRadius` 40 / `outerRadius` 80, `paddingAngle` 2, one `.recharts-pie-sector` per slice, tooltip on hover, no legend) showing revenue by region
|
||||
- a `declarative-bar-chart` (recharts, 200px tall, single blue `#3b82f6` bars with rounded tops `[4,4,0,0]`, dashed `3 3` grid) showing monthly revenue for all six months Jan–Jun
|
||||
- [ ] Verify the surface is a single composed dashboard, NOT a lonely single widget — this is the regression OSS-136 was filed about
|
||||
- [ ] Verify the pie slices cycle through `CHART_COLORS` (`#3b82f6`, `#8b5cf6`, `#ec4899`, `#f59e0b`, `#10b981`, `#6366f1`) and bars are uniform blue `#3b82f6`; every chart sits in the shared `CardShell` chrome (12px radius, 20px padding, soft shadow)
|
||||
- [ ] Verify the chat reply text beneath the surface is one short sentence (per `SYSTEM_PROMPT`: "Keep chat replies to one short sentence; let the UI do the talking.")
|
||||
- [ ] Verify metric numbers match the Vantage Threads dataset (revenue $4.2M, 186 new customers, 31% win rate, $22.6k avg deal)
|
||||
|
||||
#### Team Performance — DataTable
|
||||
|
||||
- [ ] Click "Team performance" ("How are our sales reps performing against quota?"); within 60s verify a `declarative-data-table` renders inside a `declarative-card`: uppercase column headers (rep / attainment / pipeline), one body row per rep (5 reps, Dana Whitfield 124% through Elena Vasquez 71%), tabular numerals
|
||||
- [ ] Verify a quota-attainment `declarative-bar-chart` renders alongside the table (dashboardy, not a bare table); no `declarative-status-badge` or `declarative-info-row`
|
||||
|
||||
#### At Risk — StatusBadge Cards
|
||||
|
||||
- [ ] Click "Anything at risk?" ("Are any accounts or pipeline deals at risk this quarter?"); within 60s verify a risk panel: a strip of 3 `declarative-metric` tiles (ARR at risk $615k, accounts at risk 3, biggest exposure Northwind $340k) above three side-by-side `declarative-card`s (Northwind Retail, Cascadia Outfitters, Atlas Goods), each with a content-sized `declarative-status-badge` (`error` for high severity, `warning` for medium) and a one-line reason + recommended next action
|
||||
- [ ] Verify the badges are content-sized pills (not full-width banners) and that no chart or table renders for this pill
|
||||
|
||||
#### Top Account — InfoRow Facts
|
||||
|
||||
- [ ] Click "Top account details" ("Pull up the details on our biggest account."); within 60s verify a `declarative-card` for Meridian Apparel Group with at least 3 `declarative-info-row` label/value rows (owner, region, ARR $612k, renewal Sep 30, last contact), each separated by a 1px bottom border with no trailing border on the last row
|
||||
- [ ] Verify a product-line `declarative-pie-chart` renders next to the fact card (grounded in Meridian's product mix: Outerwear $260k, Footwear $180k, Accessories $112k, Custom $60k); no data table or status badge
|
||||
|
||||
#### Cross-Pill Differentiation (mirrors the D5 probe)
|
||||
|
||||
- [ ] Run all 4 pills in one conversation; verify each pill mounts its distinguishing component fresh (the D5 probe `showcase/harness/src/probes/scripts/d5-gen-ui-declarative.ts` asserts a newly-mounted testid per pill — leftovers from earlier pills must not be the only match)
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Send "What is 2+2?"; verify the agent replies in plain text without invoking `generate_a2ui` (no `a2ui_operations` in the response stream, no surface rendered)
|
||||
- [ ] DevTools → Console: walk through all flows above; verify no uncaught errors, no React error #31, no A2UI render-error banners ("Cannot create component root without a type", "Catalog not found"), and no `Invalid chart value` warnings (the chart renderers log that when the model emits a non-numeric `value`)
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3s; plain-text response within 10s; A2UI surfaces render within 60s of prompt (the inner `render_a2ui` Claude call can be slow on cold start)
|
||||
- `generate_a2ui` is called exactly once per surface-producing prompt; result contains a valid `a2ui_operations` container with `catalogId: "declarative-gen-ui-catalog"`
|
||||
- The hero pill produces a composed dashboard (4 KPI tile metrics + 1 PieChart + 1 BarChart in one surface, with NO surrounding Card per OSS-136); pills 2–4 produce their distinguishing component (data-table / status-badge / info-row)
|
||||
- Numbers are consistent with the Vantage Threads dataset across all four pills
|
||||
- No UI layout breaks, no flash of unstyled content, no uncaught console errors
|
||||
@@ -0,0 +1,87 @@
|
||||
# QA: Declarative UI — Hashbrown — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo deployed at `/demos/declarative-hashbrown`
|
||||
- Agent backend healthy (`GET /api/health` returns
|
||||
`{"status":"ok","integration":"claude-sdk-python"}`)
|
||||
- `ANTHROPIC_API_KEY` set in the deployment environment (`ANTHROPIC_MODEL`
|
||||
defaults to `claude-opus-4-8`)
|
||||
- `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI server
|
||||
exposing `POST /declarative-hashbrown` (`src/agent_server.py`, prompt in
|
||||
`src/agents/byoc_hashbrown_agent.py`)
|
||||
- `@hashbrownai/core` + `@hashbrownai/react` installed in the package
|
||||
(pinned to `0.5.0-beta.4`)
|
||||
- Frontend runtime: `/api/copilotkit-declarative-hashbrown`, agent name
|
||||
`declarative-hashbrown-demo`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Page load
|
||||
|
||||
- [ ] Navigate to `/demos/declarative-hashbrown`
|
||||
- [ ] Header "Declarative UI: Hashbrown" visible
|
||||
- [ ] Short description mentioning `@hashbrownai/react` visible
|
||||
- [ ] Chat composer visible at the bottom of the chat area
|
||||
- [ ] 3 suggestion pills visible inside the composer with labels:
|
||||
"Sales dashboard", "Revenue by category", "Expense trend"
|
||||
- [ ] No red console errors (amber hydration warnings tolerated)
|
||||
|
||||
### 2. Sales dashboard suggestion
|
||||
|
||||
- [ ] Click the "Sales dashboard" pill
|
||||
- [ ] The prompt is dispatched automatically (useConfigureSuggestions sends
|
||||
the message on pill click)
|
||||
- [ ] Within 45 seconds, at least one MetricCard (`data-testid="metric-card"`)
|
||||
renders in the transcript
|
||||
- [ ] Within 45 seconds, at least one chart
|
||||
(`data-testid="bar-chart"` or `data-testid="pie-chart"`) renders
|
||||
- [ ] At least one Markdown heading renders inline (the prompt's worked
|
||||
example leads with `## Q4 Sales Summary`)
|
||||
- [ ] Rendered content streams progressively — partial UI appears before the
|
||||
full response completes (optional visual check)
|
||||
|
||||
### 3. Revenue by category
|
||||
|
||||
- [ ] Click "Revenue by category"
|
||||
- [ ] Within 45s, a pie chart (`data-testid="pie-chart"`) renders
|
||||
- [ ] Legend shows at least 4 segments with readable labels and values
|
||||
|
||||
### 4. Expense trend
|
||||
|
||||
- [ ] Click "Expense trend"
|
||||
- [ ] Within 45s, a bar chart (`data-testid="bar-chart"`) renders
|
||||
- [ ] Chart has at least 3 bars with month-like labels
|
||||
|
||||
### 5. Free-form prompt
|
||||
|
||||
- [ ] Type "Show me revenue trends for the last six months" and press Enter
|
||||
- [ ] Verify at least one catalog component renders (metric, chart, or deal)
|
||||
|
||||
### 6. Deal card
|
||||
|
||||
- [ ] Ask "Show me a sample deal in the negotiation stage"
|
||||
- [ ] Verify a `data-testid="hashbrown-deal-card"` renders with a stage badge
|
||||
reading `negotiation` and a `$`-formatted value
|
||||
|
||||
### 7. Multi-turn
|
||||
|
||||
- [ ] After a first render completes, send a follow-up prompt
|
||||
(e.g. "Now break it down by region")
|
||||
- [ ] A new render appears alongside prior renders in the transcript
|
||||
|
||||
### 8. Error handling
|
||||
|
||||
- [ ] Empty send is a no-op (button stays disabled)
|
||||
- [ ] Console remains clean during successful flows
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Suggestion pills produce a hashbrown render within 45 seconds
|
||||
- The backend emits the JSON envelope (`{"ui": [...]}`), NEVER the `<ui>` XML
|
||||
form — the XML in `useUiKit({ examples })` is hashbrown's prompt DSL, not
|
||||
the wire format this demo consumes
|
||||
- Streaming renders assemble progressively as JSON chunks arrive
|
||||
- No uncaught errors; no `HashBrownRenderMessage must be used within
|
||||
HashBrownDashboard` errors
|
||||
- Multi-turn works without clearing prior renders
|
||||
@@ -0,0 +1,68 @@
|
||||
# QA: Declarative UI — json-render — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo deployed at `/demos/declarative-json-render`
|
||||
- Agent backend healthy (`GET /api/health` returns
|
||||
`{"status":"ok","integration":"claude-sdk-python"}`)
|
||||
- `ANTHROPIC_API_KEY` set in the deployment environment (`ANTHROPIC_MODEL`
|
||||
defaults to `claude-opus-4-8`)
|
||||
- `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI server
|
||||
exposing `POST /declarative-json-render` (`src/agent_server.py`, prompt in
|
||||
`src/agents/byoc_json_render_agent.py`)
|
||||
- `@json-render/core` + `@json-render/react` present in `package.json`
|
||||
(pinned to `0.18.0`)
|
||||
- Frontend runtime: `/api/copilotkit-declarative-json-render`, agent name
|
||||
`byoc_json_render`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Page load
|
||||
|
||||
- [ ] Navigate to `/demos/declarative-json-render`.
|
||||
- [ ] The chat surface loads inside the centered `max-w-4xl` container.
|
||||
- [ ] Chat composer is visible.
|
||||
- [ ] Three suggestion pills appear with titles: "Sales dashboard", "Revenue by category", "Expense trend".
|
||||
- [ ] No console errors.
|
||||
|
||||
### 2. Sales dashboard suggestion
|
||||
|
||||
- [ ] Click the "Sales dashboard" suggestion.
|
||||
- [ ] Within 60 seconds, a `data-testid="json-render-root"` wrapper appears in the assistant bubble.
|
||||
- [ ] A `data-testid="metric-card"` renders inside the wrapper.
|
||||
- [ ] A chart (`data-testid="bar-chart"` or `data-testid="pie-chart"`) renders inside the wrapper.
|
||||
- [ ] No raw JSON text is shown once rendering finishes — the streaming JSON is replaced by components.
|
||||
|
||||
### 3. Revenue by category
|
||||
|
||||
- [ ] Click the "Revenue by category" suggestion.
|
||||
- [ ] Within 60 seconds, a `data-testid="pie-chart"` renders with multiple category slices + legend.
|
||||
|
||||
### 4. Expense trend
|
||||
|
||||
- [ ] Click the "Expense trend" suggestion.
|
||||
- [ ] Within 60 seconds, a `data-testid="bar-chart"` renders with month labels.
|
||||
|
||||
### 5. Free-form prompt
|
||||
|
||||
- [ ] Type "Show me a metric for quarterly revenue" and send.
|
||||
- [ ] Verify at least one `metric-card` renders; no console errors.
|
||||
|
||||
### 6. Multi-turn
|
||||
|
||||
- [ ] After a previous render is visible, send a follow-up prompt ("Now break that down by region").
|
||||
- [ ] A new assistant message appears with a new json-render rendering — prior renders stay in the transcript.
|
||||
|
||||
### 7. Malformed output handling
|
||||
|
||||
- [ ] Force non-spec output by asking "tell me a joke". The renderer's
|
||||
`parseSpec` returns null for anything that is not a `{ root, elements }`
|
||||
object whose element `type`s are all in `MetricCard` / `BarChart` /
|
||||
`PieChart`, so the chat falls back to the default
|
||||
`CopilotChatAssistantMessage` bubble. No crash, no stuck spinner.
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Suggestion renders land within 60 seconds. Budget is slightly higher than the hashbrown demo because a JSON `{ root, elements }` spec is more verbose than hashbrown's token stream.
|
||||
- No uncaught errors in the console.
|
||||
- Streaming falls back to plain text until the JSON parses, then swaps to rendered components wrapped in a single `<JSONUIProvider>` (no missing-provider crashes).
|
||||
@@ -0,0 +1,76 @@
|
||||
# QA: In-Chat HITL via useInterrupt — Claude Agent SDK (Python)
|
||||
|
||||
> **STATUS: QUARANTINED.** `gen-ui-interrupt` is listed under
|
||||
> `not_supported_features` in `manifest.yaml` (alongside `interrupt-headless`),
|
||||
> with this reason: turn-2 fails on a `useInterrupt` / `useHeadlessInterrupt`
|
||||
> **resume-path** bug in `@copilotkit/react-core/v2` — the backend resumes and
|
||||
> streams (HTTP 200) but the frontend never appends the confirmation assistant
|
||||
> bubble, so the harness DOM settle-check times out. The fix is a
|
||||
> published-package change. The reference integration (langgraph-python)
|
||||
> quarantines the same cell for the same reason. The demo stays fully wired.
|
||||
>
|
||||
> Sections 1–2 below are verifiable today. Section 3 is the **re-qualification
|
||||
> checklist**: do not tick it, and do not report the cell green off the back of
|
||||
> it — run it only once the upstream react-core fix lands, and report the result
|
||||
> to whoever owns the manifest.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/gen-ui-interrupt` on the dashboard host
|
||||
- Next.js host is healthy (`GET /api/health`) and the Python backend is reachable (`GET /api/copilotkit` reports `agent_status: "reachable"`); `ANTHROPIC_API_KEY` is set. `ANTHROPIC_MODEL` is optional (`.env.example` sets `claude-opus-4-8`, which is also the in-code fallback in `src/agents/interrupt_agent.py`)
|
||||
- `AGENT_URL` (default `http://localhost:8000`) points at `src/agent_server.py`. This demo goes through the SHARED runtime `/api/copilotkit`: `dedicatedAgentPaths` in `src/app/api/copilotkit/route.ts` maps agent name `gen-ui-interrupt` → `${AGENT_URL}/interrupt-adapted` → `run_interrupt_agent`. There is **no** LangGraph deployment and no `interrupt()` primitive in this integration
|
||||
- **How this integration adapts the demo.** The Claude Agent SDK has no LangGraph checkpoint/resume `interrupt()`. `src/agents/interrupt_agent.py` instead forwards the frontend tool definitions it receives in `input_data.tools` straight to Claude and, per its `SYSTEM_PROMPT` ("you MUST call the `schedule_meeting` tool" with a `topic` and optional `attendee`), emits a `schedule_meeting` tool call. Two facts to hold onto while testing, both verified in this integration's source:
|
||||
- the Python backend emits **no** AG-UI interrupt signal — no `on_interrupt` custom event and no `RUN_FINISHED` `outcome: "interrupt"` — anywhere in `src/agents/`
|
||||
- `src/app/demos/gen-ui-interrupt/page.tsx` registers **no** `useFrontendTool` / `useRenderTool`; it wires only `useInterrupt({ agentId: "gen-ui-interrupt", renderInChat: true, render })`
|
||||
So `useInterrupt`'s `render` callback has no event source here, and the picker card is expected NOT to mount. If it DOES mount, that is new information — record it and escalate, because it changes the quarantine rationale
|
||||
- The picker component is `src/app/demos/gen-ui-interrupt/_components/time-picker-card.tsx` (testids `time-picker-card`, `time-picker-slot`, `time-picker-cancel`, `time-picker-picked`, `time-picker-cancelled`); fallback slot labels come from `src/app/demos/_shared/interrupt-fallback-slots.ts`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality (verifiable today)
|
||||
|
||||
- [ ] Navigate to `/demos/gen-ui-interrupt`; verify the page renders within 3s with the `CopilotChat` centered in a `max-w-4xl` container filling full viewport height, `rounded-2xl`
|
||||
- [ ] Verify the `CopilotChat` input placeholder is visible and the transcript is empty on first load
|
||||
- [ ] Send "Hello" and verify the agent responds with a text-only reply (no picker — the prompt only instructs `schedule_meeting` for booking/scheduling requests)
|
||||
|
||||
### 2. Feature-Specific Checks (verifiable today)
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify both suggestion pills are visible with verbatim titles:
|
||||
- "Book a call with sales" (message: "Book an intro call with the sales team to discuss pricing.")
|
||||
- "Schedule a 1:1 with Alice" (message: "Schedule a 1:1 with Alice next week to review Q2 goals.")
|
||||
|
||||
#### Turn 1 — Backend tool call reaches the client
|
||||
|
||||
- [ ] Click "Book a call with sales"
|
||||
- [ ] DevTools → Network: verify the request goes to `/api/copilotkit` and the SSE stream carries `TOOL_CALL_START` / `TOOL_CALL_ARGS` / `TOOL_CALL_END` for `schedule_meeting` with a `topic` (and `attendee` where the prompt names one) — this is the backend half of the adaptation and it should work
|
||||
- [ ] Record whether a `data-testid="time-picker-card"` element mounts inside the chat transcript. Per the Prerequisites, the expected answer on this integration is **no** — the tool call arrives with no interrupt signal and no frontend handler, so the transcript shows the assistant text and however `CopilotChat` renders an unhandled tool call. Note down exactly what you see — that observation is the useful output of this step
|
||||
- [ ] Verify the page does not crash: no uncaught console errors, no blank pane, chat input still accepts a second message
|
||||
|
||||
#### Contract Check — Interrupt Is Low-Level
|
||||
|
||||
- [ ] Confirm a plain conversational message ("What's the weather?") does not render a picker and does not call `schedule_meeting`
|
||||
- [ ] Confirm no approval-dialog-style modal appears at any point (this demo is inline, not modal — contrast `hitl-in-app`, which portals a modal)
|
||||
|
||||
### 3. Re-qualification checklist — BLOCKED, do not tick
|
||||
|
||||
Run only after the `@copilotkit/react-core/v2` resume-path fix ships AND an interrupt signal exists on this backend. These are the reference behaviors this cell owes; each one is currently unreachable.
|
||||
|
||||
- Picker renders INLINE in the transcript (`time-picker-card`), a descendant of the chat container, NOT portaled to `<body>`
|
||||
- Card header shows the outline badge "Book a call", the agent-supplied topic as the title, "With <attendee>" when present, and the description "Pick a time that works for you."
|
||||
- A 2-column grid of `time-picker-slot` buttons; with no backend-supplied slots the fallback labels are "Tomorrow 10:00 AM", "Tomorrow 2:00 PM", "Monday 9:00 AM", "Monday 3:30 PM"
|
||||
- A ghost `time-picker-cancel` button labeled "None of these work" below the grid
|
||||
- Pick path: card switches to `time-picker-picked` (green-tinted, "Booked" badge + bold slot label), all buttons disable, and `resolve({chosen_time, chosen_label})` fires after the deliberate 500ms commit delay in `page.tsx`
|
||||
- Cancel path: card switches to `time-picker-cancelled` ("Cancelled" badge + "No time picked.") and `resolve({cancelled: true})` fires
|
||||
- **Turn 2 (the quarantined step):** the agent resumes and appends a confirmation assistant bubble naming the chosen slot, or noting the cancellation. This is exactly what the react-core bug drops
|
||||
- Multi-turn: a fresh independent picker renders for a follow-up booking prompt while the earlier card stays in its resolved state
|
||||
- Double-click a slot button rapidly: only one selection commits
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds; plain-text response within 10 seconds
|
||||
- A booking prompt produces a `schedule_meeting` tool call on the wire within 20 seconds
|
||||
- No inline picker card, and therefore no pick/cancel/resume flow, while the cell is quarantined — this is the accepted outcome, honestly marked skipped-incapable in `manifest.yaml` (not green, not red)
|
||||
- No UI layout breaks and no uncaught console errors at any point
|
||||
- Anything that contradicts the two verified facts in the Prerequisites (a picker that mounts, an `on_interrupt` event on the wire) is a finding worth reporting, not a pass
|
||||
@@ -0,0 +1,28 @@
|
||||
# QA: Interrupt (Headless) — Claude Agent SDK (Python)
|
||||
|
||||
> Stub — authored for column completeness. This is a testing-kind demo
|
||||
> (see `kind: "testing"` in `showcase/shared/feature-registry.json`) and
|
||||
> does not warrant a full manual checklist. The cell is additionally
|
||||
> quarantined under `not_supported_features` in `manifest.yaml` (shared
|
||||
> upstream `@copilotkit/react-core/v2` resume-path defect).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy; `ANTHROPIC_API_KEY` set. The cell uses the shared
|
||||
`/api/copilotkit` runtime with agent `interrupt-headless`, mapped to the
|
||||
FastAPI `POST /interrupt-adapted` endpoint (`src/agents/interrupt_agent.py`)
|
||||
|
||||
## Test Steps
|
||||
|
||||
- [ ] Navigate to /demos/interrupt-headless and verify the left app surface
|
||||
(`data-testid="interrupt-headless-app-surface"`) shows the empty state
|
||||
(`data-testid="interrupt-headless-empty"`, "Nothing scheduled yet")
|
||||
- [ ] Send a scheduling prompt (e.g. the "Book a call with sales" pill) and verify a time-slot picker popup (`data-testid="interrupt-headless-popup"`) appears in the left app surface, not in the chat
|
||||
- [ ] Click one of the time-slot buttons and verify the popup disappears and the agent confirms the booking back in the chat
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads without errors
|
||||
- Interrupt resolves via the plain button grid (no `useInterrupt` render prop, no in-chat picker) and the agent continues the run with the picked slot
|
||||
- Known deviation: the turn-2 confirmation bubble may never append — that is the quarantined react-core resume-path bug, not an integration regression
|
||||
@@ -0,0 +1,64 @@
|
||||
# QA: MCP Apps — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/mcp-apps` on the dashboard host
|
||||
- Agent backend is healthy; `ANTHROPIC_API_KEY` is set on the deployment (`ANTHROPIC_MODEL` defaults to `claude-opus-4-8`); `AGENT_URL` (default `http://localhost:8000`) points at the FastAPI server exposing `POST /mcp-apps` (`src/agent_server.py`, handler in `src/agents/mcp_apps_agent.py`), registered as agent name `mcp-apps` — see `src/app/api/copilotkit-mcp-apps/route.ts`
|
||||
- MCP server target: the public Excalidraw MCP app at `https://mcp.excalidraw.com` (override via `MCP_SERVER_URL`). Pinned `serverId: "excalidraw"` so URL changes don't silently break persisted activities
|
||||
- Note: the demo source contains no `data-testid` attributes and registers no custom activity renderer — CopilotKit's built-in `MCPAppsActivityRenderer` handles the sandboxed iframe automatically. Checks below rely on verbatim visible text, network traffic, and the iframe DOM
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/mcp-apps`; verify the page renders within 3s and a single `CopilotChat` pane is centered (max-width ~896px via `max-w-4xl`, `rounded-2xl`, full-height)
|
||||
- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-mcp-apps"` and `agent="mcp-apps"` (DevTools → Network: sending a message hits that endpoint)
|
||||
- [ ] Verify both suggestion pills are visible with verbatim titles:
|
||||
- "Draw a flowchart"
|
||||
- "Sketch a system diagram"
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s (no MCP activity iframe for plain text)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### MCP Server Connection (runtime `mcpApps.servers`)
|
||||
|
||||
- [ ] Send the first flow-chart prompt; in DevTools → Network, verify the POST to `/api/copilotkit-mcp-apps` succeeds (status 200) and the server-side runtime resolves tools from `https://mcp.excalidraw.com` (the MCP Apps middleware appends them to the AG-UI request's `tools` array, which `_build_anthropic_tools` in `src/agents/mcp_apps_agent.py` forwards verbatim to Claude — notably `create_view`)
|
||||
- [ ] Verify no console errors mentioning the MCP server URL, auth, or tool-schema parse failures
|
||||
|
||||
#### MCP Tool Invocation (`create_view`)
|
||||
|
||||
- [ ] Click "Draw a flowchart"; within 60s verify the agent calls the `create_view` MCP tool exactly ONCE (per `SYSTEM_PROMPT` in `src/agents/mcp_apps_agent.py`: "Call `create_view` ONCE with 3-5 elements total") — confirm via DevTools → Network stream or backend logs
|
||||
- [ ] Verify the tool payload contains 3-5 Excalidraw elements (shapes + arrows + optional title text), each with a unique string `id`, and ends with ONE `cameraUpdate` sized `600x450` or `800x600`
|
||||
|
||||
#### Activity Renderer (built-in `MCPAppsActivityRenderer`)
|
||||
|
||||
- [ ] Within 60s of the tool call, verify a sandboxed `<iframe>` renders inline in the chat transcript (activity-message slot) pointed at the Excalidraw MCP UI resource
|
||||
- [ ] Verify the iframe has a `sandbox` attribute (CopilotKit's built-in renderer always sandboxes MCP UI resources)
|
||||
- [ ] Verify the iframe paints a flow-chart-shaped diagram: at least 3 shape nodes (rectangles, ellipses, or diamonds with text labels) connected by arrows, framed within the viewport (camera-update step from the system prompt)
|
||||
- [ ] Verify the assistant text below the iframe is a single short sentence describing what was drawn (per system prompt)
|
||||
|
||||
#### Server-Driven UI Update (second prompt, same thread)
|
||||
|
||||
- [ ] Without reloading, send the second suggestion "Sketch a system diagram"; within 60s verify a new activity iframe renders in-transcript containing a client → server → database layout (3 labeled shapes + 2 arrows)
|
||||
- [ ] Verify the previous flow-chart iframe is still present and un-stale in the scrollback (activity messages persist, matching the rationale for the pinned `serverId: "excalidraw"` in the runtime config)
|
||||
|
||||
#### End-to-End MCP Interaction (concrete, single-case)
|
||||
|
||||
- [ ] Send an explicit prompt: `"Use Excalidraw to draw exactly 2 rectangles labelled 'A' and 'B' connected by one arrow from A to B."`
|
||||
- [ ] Within 60s verify: (1) `create_view` is called ONCE with exactly 3 elements (2 rectangles + 1 arrow) plus the trailing `cameraUpdate`; (2) an iframe renders showing two labelled rectangles with a connecting arrow; (3) the assistant reply is one short sentence; (4) no duplicate `create_view` invocations or retries appear in network / logs
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Send "What is 2+2?"; verify the agent replies in plain text without invoking `create_view` (no iframe, no MCP activity in the stream)
|
||||
- [ ] Verify no assistant bubble ever starts with `Agent error:` — `run_mcp_apps_agent` streams caught exceptions into the transcript with that prefix rather than breaking the SSE stream, so its presence means the backend threw
|
||||
- [ ] DevTools → Console: walk through all flows above; verify no uncaught errors, no CORS failures referencing `mcp.excalidraw.com`, and no "sandbox" / iframe-permission warnings
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3s; plain-text response within 10s; MCP-backed iframe renders within 60s of prompt (bias is "correct-enough diagram fast" per system prompt, one `create_view` call)
|
||||
- MCP server connection to `https://mcp.excalidraw.com` succeeds and the Excalidraw tool set (including `create_view`) is advertised to Claude at request time
|
||||
- At least one concrete end-to-end MCP interaction completes: user prompt → `create_view` tool call → activity event → sandboxed iframe painting the requested diagram
|
||||
- The built-in `MCPAppsActivityRenderer` is used (no app-side `useRenderActivityMessage` / `renderActivityMessages` registration exists in `src/app/demos/mcp-apps/page.tsx` — per the `@region[no-frontend-renderer-needed]` contract)
|
||||
- The backend runs no server-side tool loop: it emits `TOOL_CALL_*` events and stops, and the runtime middleware resolves the MCP call and re-invokes it with the result
|
||||
- No UI layout breaks, no uncaught console errors, no duplicate `create_view` invocations within a single prompt turn
|
||||
@@ -0,0 +1,24 @@
|
||||
# QA: Reasoning (Default) — Claude Agent SDK (Python)
|
||||
|
||||
> Stub — authored for column completeness. This demo verifies the
|
||||
> built-in `CopilotChatReasoningMessage` renders without a custom slot
|
||||
> and does not warrant a full manual checklist.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy; `ANTHROPIC_API_KEY` set. The cell uses the shared
|
||||
`/api/copilotkit` runtime with agent `reasoning-default`, mapped to the
|
||||
FastAPI `POST /reasoning` endpoint (`src/agents/reasoning_agent.py`), which
|
||||
emits AG-UI `REASONING_MESSAGE_*` events
|
||||
|
||||
## Test Steps
|
||||
|
||||
- [ ] Navigate to /demos/reasoning-default
|
||||
- [ ] Click the "Show reasoning" suggestion pill (prompt: "Explain step by step why the sky appears blue during the day but red at sunset.") and verify the built-in `CopilotChatReasoningMessage` collapsible card renders the reasoning tokens
|
||||
- [ ] Verify no custom reasoning slot is wired (default styling only — no `ReasoningBlock` or bespoke container; `page.tsx` passes no `messageView.reasoningMessage`)
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads without errors
|
||||
- Reasoning renders via CopilotKit's default `CopilotChatReasoningMessage` component with zero frontend configuration
|
||||
@@ -1,41 +0,0 @@
|
||||
# QA: Shared State (Writing) — Claude Agent SDK (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the shared-state-write demo page
|
||||
- [ ] Verify the chat interface loads with title "Shared State (Writing)"
|
||||
- [ ] Verify the chat input placeholder "Type a message..." is visible
|
||||
- [ ] Send a basic message (e.g. "Hello! What can you do?")
|
||||
- [ ] Verify the agent responds
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Get started" suggestion button is visible
|
||||
|
||||
#### Note: Stub Demo
|
||||
|
||||
> **Status: Stub** — This demo is currently a stub (TODO: implement)
|
||||
|
||||
- [ ] Verify the basic CopilotChat loads and accepts messages
|
||||
- [ ] Verify the agent responds to messages
|
||||
- [ ] No custom UI components are expected beyond the chat interface
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Agent responds within 10 seconds
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,43 @@
|
||||
# Thread ID Frontend Tool Round Trip
|
||||
|
||||
## Scope
|
||||
|
||||
Regression checklist for ENT-658: a `CopilotChat` wrapped by
|
||||
`CopilotChatConfigurationProvider` must keep its SDK-generated non-explicit
|
||||
thread active across a frontend tool call and follow-up run. On this
|
||||
integration the chat is wired to `runtimeUrl="/api/copilotkit"` with agent
|
||||
`threadid-frontend-tool-roundtrip`, which has no dedicated FastAPI endpoint —
|
||||
it falls through to the root `POST /` agent in `src/agents/agent.py`.
|
||||
|
||||
## Manual QA
|
||||
|
||||
- [ ] Navigate to `/demos/threadid-frontend-tool-roundtrip`.
|
||||
- [ ] Verify the chat input is visible and `Explicit threadId` is unchecked,
|
||||
and `data-testid="ent-658-thread-mode"` reads `SDK-generated thread`.
|
||||
- [ ] Send `invoke testFrontendToolCalling with label X`.
|
||||
- [ ] Verify the user message remains visible.
|
||||
- [ ] Verify the `data-testid="ent-658-tool-card"` card remains visible and
|
||||
shows `label: X` and `result: handled X`.
|
||||
- [ ] Verify an assistant follow-up reply appears (`followUp: true` on the
|
||||
tool guarantees a second run; the recorded fixture asserts the verbatim
|
||||
`Frontend tool finished for X.` — against a live model the wording may
|
||||
differ, but a follow-up assistant message MUST appear).
|
||||
- [ ] Verify the chat does not return to the empty state.
|
||||
- [ ] Refresh the page or open a new tab, check `Explicit threadId` before
|
||||
sending any message (the mode line must flip to `Explicit thread`), send
|
||||
the same prompt again, and verify the same message/tool/reply
|
||||
persistence behavior.
|
||||
- [ ] Optionally toggle `Explicit threadId` after a generated-thread
|
||||
conversation and verify the chat switches to the explicit thread's
|
||||
history. An empty explicit thread on first use is expected.
|
||||
|
||||
## Automated Coverage
|
||||
|
||||
- `showcase/harness/src/probes/scripts/d5-threadid-frontend-tool-roundtrip.ts`
|
||||
is the shared probe run against every integration (it asserts the
|
||||
`Frontend tool finished for X.` page text).
|
||||
- `showcase/aimock/d6/claude-sdk-python/threadid-frontend-tool-roundtrip.json`
|
||||
is this integration's fixture: turn 1 emits the `testFrontendToolCalling`
|
||||
call, turn 2 confirms the tool result without changing threads.
|
||||
- `packages/react-core/src/v2/components/chat/__tests__/CopilotChat.absentThreadConnect.test.tsx`
|
||||
covers the SDK-generated thread handoff at the component level.
|
||||
@@ -0,0 +1,25 @@
|
||||
# QA: Tool Rendering (Reasoning Chain) — Claude Agent SDK (Python)
|
||||
|
||||
> Stub — authored for column completeness. This is a testing-kind demo
|
||||
> (see `kind: "testing"` in `showcase/shared/feature-registry.json`) and
|
||||
> does not warrant a full manual checklist.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy; `ANTHROPIC_API_KEY` set. The cell uses the shared
|
||||
`/api/copilotkit` runtime with agent `tool-rendering-reasoning-chain`,
|
||||
mapped to the FastAPI `POST /tool-rendering-reasoning-chain` endpoint
|
||||
(`src/agents/tool_rendering_reasoning_chain_agent.py`)
|
||||
|
||||
## Test Steps
|
||||
|
||||
- [ ] Navigate to /demos/tool-rendering-reasoning-chain
|
||||
- [ ] Click the "Flights + destination weather" pill ("Find flights from SFO to JFK and show me the weather there.") and verify reasoning blocks interleave with sequential tool cards — `search_flights` → `FlightListCard`, `get_weather` → `WeatherCard`
|
||||
- [ ] Click "Compare two stocks" or "Chain of dice rolls" and verify the backend-only tools (`get_stock_price`, `roll_dice`) fall through to `CustomCatchallRenderer` via `useDefaultRenderTool`
|
||||
- [ ] Verify reasoning tokens stream into the custom `ReasoningBlock` slot alongside the tool cards in the same message view
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads without errors
|
||||
- Reasoning tokens and tool-call cards render side-by-side in a single sequential chain, each tool matched to its typed renderer
|
||||
@@ -114,6 +114,18 @@ from agents.mcp_apps_agent import run_mcp_apps_agent
|
||||
from agents.multimodal_agent import SYSTEM_PROMPT as MULTIMODAL_SYSTEM_PROMPT
|
||||
from agents.multimodal_agent import convert_part_for_claude
|
||||
from agents.reasoning_agent import run_reasoning_agent
|
||||
|
||||
# Shared State (Frontend Context) — the reference
|
||||
# (langgraph-python/src/agents/readonly_state_agent_context.py) mounts a
|
||||
# DEDICATED graph with `tools=[]` and a system prompt telling the agent to
|
||||
# consult the READ-ONLY context the frontend publishes via `useAgentContext`.
|
||||
# Import the matching prompt hint so our dedicated
|
||||
# `/readonly-state-agent-context` endpoint applies it via
|
||||
# system_prompt_override (parity, not generic fallback). See
|
||||
# dedicatedAgentPaths in the copilotkit route.
|
||||
from agents.readonly_state_agent_context import (
|
||||
SYSTEM_PROMPT_HINT as READONLY_STATE_AGENT_CONTEXT_SYSTEM_PROMPT,
|
||||
)
|
||||
from agents.shared_state_read_write_agent import (
|
||||
run_shared_state_read_write_agent,
|
||||
)
|
||||
@@ -373,6 +385,35 @@ async def shared_state_read_write_endpoint(request: Request) -> StreamingRespons
|
||||
)
|
||||
|
||||
|
||||
@app.post("/readonly-state-agent-context")
|
||||
async def readonly_state_agent_context_endpoint(
|
||||
request: Request,
|
||||
) -> StreamingResponse:
|
||||
"""Shared State (Frontend Context) — read-only `useAgentContext` consumer.
|
||||
|
||||
Parity with langgraph-python, whose dedicated `readonly_state_agent_context`
|
||||
graph is `tools=[]` plus a system prompt telling the agent to consult the
|
||||
read-only context (name, timezone, recent activity) the frontend publishes
|
||||
via `useAgentContext`. The demo page registers no frontend tools of its own,
|
||||
so this endpoint is that prompt plus an empty backend tool set and nothing
|
||||
else — the CopilotKit runtime already routes the context entries into the
|
||||
message history.
|
||||
|
||||
Without the dedicated endpoint the demo fell through to the generic root
|
||||
agent, whose sales-assistant SYSTEM_PROMPT and ~10 backend tools say nothing
|
||||
about consuming frontend context: the GOTCHAS #8 masking bug (the fixture
|
||||
replays a context-aware answer, so D6 is green while the wrong agent is
|
||||
live).
|
||||
"""
|
||||
body = await request.json()
|
||||
input_data = RunAgentInput(**body)
|
||||
return _stream_agent_response(
|
||||
input_data,
|
||||
system_prompt_override=READONLY_STATE_AGENT_CONTEXT_SYSTEM_PROMPT,
|
||||
tools_override=[],
|
||||
)
|
||||
|
||||
|
||||
@app.post("/reasoning")
|
||||
async def reasoning_endpoint(request: Request) -> StreamingResponse:
|
||||
"""Reasoning demo backend — emits AG-UI REASONING_MESSAGE_* events.
|
||||
|
||||
@@ -6,11 +6,13 @@ be edited by the agent, but the agent reads this context on every turn
|
||||
via the CopilotKit runtime, which routes the context entries into the
|
||||
model's message history.
|
||||
|
||||
The shared Claude backend in `src/agents/agent.py` handles this demo via
|
||||
the `readonly-state-agent-context` agent name registered in the
|
||||
copilotkit route. This module exists so the manifest's `highlight` path
|
||||
references a per-demo Python reference, mirroring the langgraph-python
|
||||
layout.
|
||||
Mirrors langgraph-python's dedicated `readonly_state_agent_context`
|
||||
graph, which is `tools=[]` plus the system prompt below. Here that same
|
||||
prompt drives the dedicated `/readonly-state-agent-context` endpoint in
|
||||
`src/agent_server.py` (with `tools_override=[]`); the copilotkit route
|
||||
maps the `readonly-state-agent-context` agent name to that path via
|
||||
`dedicatedAgentPaths`. The demo registers no tools of its own on either
|
||||
side — reading the context the frontend published is the whole feature.
|
||||
"""
|
||||
|
||||
SYSTEM_PROMPT_HINT = (
|
||||
|
||||
@@ -87,6 +87,14 @@ const dedicatedAgentPaths: Record<string, string> = {
|
||||
"tool-rendering-custom-catchall": "/tool-rendering",
|
||||
"shared-state-read-write": "/shared-state-read-write",
|
||||
"shared-state-streaming": "/shared-state-streaming",
|
||||
// Shared State (Frontend Context) needs its own backend so the agent gets
|
||||
// the tailored "consult the read-only useAgentContext entries" prompt and a
|
||||
// tools=[] surface (mirrors langgraph-python's dedicated
|
||||
// readonly_state_agent_context graph). Without this it fell through to the
|
||||
// generic root agent's sales-assistant prompt + ~10 backend tools — the
|
||||
// GOTCHAS #8 masking bug: the fixture replays a context-aware answer so D6
|
||||
// is green, while live the wrong agent handles the turn.
|
||||
"readonly-state-agent-context": "/readonly-state-agent-context",
|
||||
subagents: "/subagents",
|
||||
// Reasoning demos share a single backend that emits AG-UI
|
||||
// REASONING_MESSAGE_* events (parsed out of <reasoning>...</reasoning>
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
import type { Page } from "@playwright/test";
|
||||
|
||||
// QA reference: qa/a2ui-recovery.md
|
||||
// Demo source: src/app/demos/a2ui-recovery/{page.tsx, chat.tsx, suggestions.ts}
|
||||
//
|
||||
// A2UI error recovery (OSS-158 / OSS-375). Assert the STABLE end-states — the
|
||||
// recovered surface paints (heal) and the hard-failure UI shows (exhaust) — and
|
||||
// deliberately do NOT assert the transient "Retrying generation… (N/M)" label,
|
||||
// which is threshold-gated + timing dependent (see @copilotkit/react-core/v2
|
||||
// A2UIRecoveryStates) and would be flaky.
|
||||
//
|
||||
// The aimock fixtures (showcase/aimock/d6/langgraph-python/a2ui-recovery.json)
|
||||
// drive the inner render_a2ui sub-agent two ways: HEAL emits free-form/sloppy
|
||||
// args (components/data as JSON strings) the middleware heals via parse_and_fix;
|
||||
// EXHAUST is structurally invalid (unresolved child) on every attempt, so the
|
||||
// validate->retry loop hits the cap and returns the a2ui_recovery_exhausted
|
||||
// envelope. Healing + the loop run live in the backend ag_ui_langgraph
|
||||
// get_a2ui_tools factory (injectA2UITool=false); the failure UI text ("Couldn't
|
||||
// generate the UI") comes from @copilotkit/react-core/v2. The recovered surface
|
||||
// reuses the declarative-gen-ui catalog, so it carries the `declarative-metric`
|
||||
// testid.
|
||||
//
|
||||
// Requires the stack running with aimock so the malformed renders fire
|
||||
// deterministically; against a real LLM the demo would not reliably produce the
|
||||
// invalid attempts.
|
||||
|
||||
/** Click a suggestion pill and confirm the message dispatched (the user
|
||||
* bubble with the pill's full message text appears). The user bubble is
|
||||
* matched by `data-testid="copilot-user-message"`. Slow hydration can swallow
|
||||
* the first click, so we retry; never re-click once dispatched. */
|
||||
async function clickPill(page: Page, title: string, message: string) {
|
||||
const pill = page
|
||||
.locator('[data-testid="copilot-suggestion"]')
|
||||
.filter({ hasText: title })
|
||||
.first();
|
||||
await expect(pill).toBeVisible({ timeout: 15_000 });
|
||||
const userBubble = page
|
||||
.locator('[data-testid="copilot-user-message"]')
|
||||
.filter({ hasText: message })
|
||||
.first();
|
||||
await expect(async () => {
|
||||
if ((await userBubble.count()) === 0) {
|
||||
await pill.click();
|
||||
}
|
||||
await expect(userBubble).toBeVisible({ timeout: 3_000 });
|
||||
}).toPass({ timeout: 30_000 });
|
||||
}
|
||||
|
||||
const HEAL_PILL = "Recover a bad render";
|
||||
const HEAL_MSG =
|
||||
"Build my Q2 revenue summary and self-correct a malformed first attempt.";
|
||||
const EXHAUST_PILL = "Show an unrecoverable failure";
|
||||
const EXHAUST_MSG =
|
||||
"Build a report that fails every validation pass so I can preview the fallback.";
|
||||
|
||||
test.describe("A2UI Error Recovery", () => {
|
||||
test.setTimeout(120_000);
|
||||
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/a2ui-recovery");
|
||||
});
|
||||
|
||||
test("page loads with both recovery pills", async ({ page }) => {
|
||||
await expect(page.getByPlaceholder("Type a message")).toBeVisible();
|
||||
const suggestions = page.locator('[data-testid="copilot-suggestion"]');
|
||||
for (const title of [HEAL_PILL, EXHAUST_PILL]) {
|
||||
await expect(suggestions.filter({ hasText: title }).first()).toBeVisible({
|
||||
timeout: 15_000,
|
||||
});
|
||||
}
|
||||
// No surface on first paint.
|
||||
await expect(page.getByTestId("declarative-metric")).toHaveCount(0);
|
||||
});
|
||||
|
||||
test("heal: free-form/sloppy render is healed into a valid surface", async ({
|
||||
page,
|
||||
}) => {
|
||||
await clickPill(page, HEAL_PILL, HEAL_MSG);
|
||||
|
||||
// The model emits free-form (stringified) A2UI args; the middleware heals
|
||||
// them via parse_and_fix into a valid surface that paints. Allow 90s for
|
||||
// the sub-agent round-trip.
|
||||
const metrics = page.locator('[data-testid="declarative-metric"]');
|
||||
await expect
|
||||
.poll(async () => await metrics.count(), { timeout: 90_000 })
|
||||
.toBeGreaterThanOrEqual(2);
|
||||
|
||||
// The healed surface, NOT the hard-failure UI, and no render-error banners.
|
||||
await expect(page.getByText("Couldn't generate the UI")).toHaveCount(0);
|
||||
await expect(page.getByText(/Catalog not found/i)).toHaveCount(0);
|
||||
await expect(
|
||||
page.getByText(/Cannot create component .* without a type/i),
|
||||
).toHaveCount(0);
|
||||
});
|
||||
|
||||
test("exhaust: always-invalid render shows the hard-failure UI, no faulty surface", async ({
|
||||
page,
|
||||
}) => {
|
||||
await clickPill(page, EXHAUST_PILL, EXHAUST_MSG);
|
||||
|
||||
// Hard-failure UI appears once the attempt cap is hit (A2UIRecoveryStates
|
||||
// renders this on status: "failed" = the a2ui_recovery_exhausted envelope).
|
||||
await expect(
|
||||
page.getByText("Couldn't generate the UI").first(),
|
||||
).toBeVisible({ timeout: 90_000 });
|
||||
|
||||
// No faulty surface ever paints (server-side no-wipe guarantee: middleware
|
||||
// gate + adapter recovery loop).
|
||||
await expect(page.getByTestId("declarative-metric")).toHaveCount(0);
|
||||
|
||||
// Conversation remains usable after the hard failure.
|
||||
await expect(page.getByPlaceholder("Type a message")).toBeEnabled();
|
||||
});
|
||||
});
|
||||
@@ -169,10 +169,10 @@ test.describe("Beautiful Chat", () => {
|
||||
// `_design_a2ui_surface` (renamed from `render_a2ui` to avoid the A2UI
|
||||
// middleware's default tool-call intercept on `render_a2ui`);
|
||||
// both calls hit aimock fixtures
|
||||
// (showcase/aimock/feature-parity.json — userMessage + toolName matchers
|
||||
// differentiate primary vs secondary calls; a toolCallId match breaks the
|
||||
// post-tool loop). The render_a2ui fixture ships a 3-metric + 2-chart
|
||||
// dashboard tree against `copilotkit://app-dashboard-catalog`.
|
||||
// (showcase/aimock/d4/claude-sdk-python/chat.json — userMessage + toolName
|
||||
// matchers differentiate primary vs secondary calls; a toolCallId match
|
||||
// breaks the post-tool loop). The render_a2ui fixture ships a 3-metric +
|
||||
// 2-chart dashboard tree against `copilotkit://app-dashboard-catalog`.
|
||||
//
|
||||
// Visual fingerprint: a Metric label "Total Revenue", plus a recharts
|
||||
// ResponsiveContainer (the Pie/BarChart custom renderers wrap their
|
||||
@@ -196,6 +196,13 @@ test.describe("Beautiful Chat", () => {
|
||||
timeout: 90_000,
|
||||
});
|
||||
|
||||
// Regression guard (#4733 / #4734 / #5425): the deployed Sales Dashboard
|
||||
// used to surface "A2UI render error: Catalog not found: ..." when the
|
||||
// model omitted `catalogId` and no `defaultCatalogId` was configured on
|
||||
// the route. Hard-assert the error is absent regardless of whether the
|
||||
// charts rendered — the error banner paints even when the surface fails.
|
||||
await expect(page.getByText(/Catalog not found/i)).toHaveCount(0);
|
||||
|
||||
// Soft assertion: if the full A2UI pipeline fires, recharts containers
|
||||
// should appear. When running against aimock-only (no secondary LLM),
|
||||
// only the text narration renders — so we don't hard-fail on missing
|
||||
@@ -207,10 +214,6 @@ test.describe("Beautiful Chat", () => {
|
||||
.catch(() => false);
|
||||
|
||||
if (chartsRendered) {
|
||||
// Regression guard (#4733 / #4734): the deployed Sales Dashboard used
|
||||
// to surface "A2UI render error: Catalog not found: ...". Assert the
|
||||
// error string is absent so any future revert trips this test.
|
||||
await expect(page.getByText(/Catalog not found/i)).toHaveCount(0);
|
||||
await expect(
|
||||
page.getByText(/Cannot create component .* without a type/i),
|
||||
).toHaveCount(0);
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
test.describe("BYOC hashbrown", () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/byoc-hashbrown");
|
||||
});
|
||||
|
||||
test("header renders", async ({ page }) => {
|
||||
await expect(page.getByText("BYOC: Hashbrown")).toBeVisible();
|
||||
});
|
||||
|
||||
test("chat composer is visible", async ({ page }) => {
|
||||
await expect(
|
||||
page.locator('textarea, [placeholder*="message"]').first(),
|
||||
).toBeVisible({ timeout: 10000 });
|
||||
});
|
||||
});
|
||||
@@ -1,21 +0,0 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
test.describe("BYOC json-render", () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/byoc-json-render");
|
||||
});
|
||||
|
||||
test("page loads with chat composer", async ({ page }) => {
|
||||
await expect(
|
||||
page.locator('textarea, [placeholder*="message"]').first(),
|
||||
).toBeVisible({ timeout: 10000 });
|
||||
});
|
||||
|
||||
test("suggestion pills are rendered", async ({ page }) => {
|
||||
await expect(
|
||||
page.getByText("Sales dashboard", { exact: false }).first(),
|
||||
).toBeVisible({
|
||||
timeout: 10000,
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,27 +1,60 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
import type { Page } from "@playwright/test";
|
||||
|
||||
// QA reference: qa/declarative-gen-ui.md
|
||||
// Demo source: src/app/demos/declarative-gen-ui/{page.tsx, a2ui/*}
|
||||
//
|
||||
// Pattern: A2UI dynamic-schema BYOC. The frontend registers a 7-component
|
||||
// catalog (Card, StatusBadge, Metric, InfoRow, PrimaryButton, PieChart,
|
||||
// BarChart) via `a2ui={{ catalog: myCatalog }}`. The Python agent
|
||||
// (`src/agents/a2ui_dynamic.py`) owns the `generate_a2ui` tool and emits an
|
||||
// `a2ui_operations` container with `catalogId: "declarative-gen-ui-catalog"`.
|
||||
// The secondary LLM inside `generate_a2ui` produces a JSON component tree
|
||||
// that the A2UI renderer binds to the registered React catalog.
|
||||
// Pattern: A2UI dynamic-schema BYOC. The frontend registers a custom catalog
|
||||
// (Row, Column, Card, StatusBadge, Metric, InfoRow, DataTable, PrimaryButton,
|
||||
// PieChart, BarChart) via `a2ui={{ catalog: myCatalog }}`. The agent plays a
|
||||
// sales analyst for the fictional "Vantage Threads" company; the dataset and
|
||||
// per-question composition rules are registered as agent context in
|
||||
// `declarative-gen-ui/sales-context.ts`. Suggestion pills are natural
|
||||
// business questions — chart-type steering lives in the agent system prompt,
|
||||
// not the user prompt (OSS-136).
|
||||
//
|
||||
// There is no `data-testid` in the demo source. We rely on verbatim
|
||||
// suggestion-pill text and the inline-style fingerprints exported by
|
||||
// `a2ui/renderers.tsx` (donut SVG, recharts markers, lilac/mint brand
|
||||
// colours, etc.). Because the secondary-LLM render is multi-step, the
|
||||
// surface can take 30-60s to paint — all render assertions use a 60s budget.
|
||||
// Each renderer carries a stable `data-testid` (declarative-card, -metric,
|
||||
// -pie-chart, -bar-chart, -status-badge, -data-table, -info-row). Because
|
||||
// the secondary-LLM render is multi-step, the surface can take 30-60s to
|
||||
// paint — render assertions use a 90s budget.
|
||||
//
|
||||
// W8-7 (resolved): KPI and StatusReport were skipped due to Railway
|
||||
// slowness. The root cause was aimock fixtures returning content+toolCalls
|
||||
// in one response — the frontend closed the assistant turn before the A2UI
|
||||
// tool call rendered. Fixed by splitting fixtures (2436adba6); all 4 pills
|
||||
// now test reliably with aimock.
|
||||
// W8-7 (resolved): aimock fixtures must split content and toolCalls into
|
||||
// separate responses — a combined response closes the assistant turn before
|
||||
// the A2UI tool call renders (see 2436adba6).
|
||||
|
||||
/** Click a suggestion pill and confirm the message actually dispatched
|
||||
* (the user bubble with the pill's full message text appears). On slow
|
||||
* dev-server hydration the first click can land before the chat send
|
||||
* pipeline is wired and is silently swallowed — retry until the bubble
|
||||
* shows up.
|
||||
*
|
||||
* The "dispatched" assertion is scoped to the chat-message list
|
||||
* (`[data-message-role="user"]`), NOT a bare `getByText` — the pill
|
||||
* button itself contains the message text, so an unscoped match would
|
||||
* satisfy the locator with the pill rather than the resulting user
|
||||
* bubble, neutering the dispatch guard. Before each retry we also
|
||||
* check whether the user bubble already exists; if it does the
|
||||
* earlier click DID dispatch and we must NOT re-click (which would
|
||||
* send a duplicate user message). */
|
||||
async function clickPill(page: Page, title: string, message: string) {
|
||||
const pill = page
|
||||
.locator('[data-testid="copilot-suggestion"]')
|
||||
.filter({ hasText: title })
|
||||
.first();
|
||||
await expect(pill).toBeVisible({ timeout: 15_000 });
|
||||
const userBubble = page
|
||||
.locator('[data-message-role="user"]')
|
||||
.filter({ hasText: message })
|
||||
.first();
|
||||
await expect(async () => {
|
||||
// If the previous attempt's click already produced the user bubble,
|
||||
// skip the click — re-clicking dispatches a duplicate user message.
|
||||
if ((await userBubble.count()) === 0) {
|
||||
await pill.click();
|
||||
}
|
||||
await expect(userBubble).toBeVisible({ timeout: 3_000 });
|
||||
}).toPass({ timeout: 30_000 });
|
||||
}
|
||||
|
||||
test.describe("Declarative Generative UI (A2UI dynamic schema)", () => {
|
||||
test.setTimeout(120_000);
|
||||
@@ -44,10 +77,10 @@ test.describe("Declarative Generative UI (A2UI dynamic schema)", () => {
|
||||
}) => {
|
||||
const suggestions = page.locator('[data-testid="copilot-suggestion"]');
|
||||
const expected = [
|
||||
"Show a KPI dashboard",
|
||||
"Pie chart — sales by region",
|
||||
"Bar chart — quarterly revenue",
|
||||
"Status report",
|
||||
"Show my sales dashboard",
|
||||
"Team performance",
|
||||
"Anything at risk?",
|
||||
"Top account details",
|
||||
];
|
||||
for (const title of expected) {
|
||||
await expect(suggestions.filter({ hasText: title }).first()).toBeVisible({
|
||||
@@ -56,105 +89,149 @@ test.describe("Declarative Generative UI (A2UI dynamic schema)", () => {
|
||||
}
|
||||
});
|
||||
|
||||
test("PieChart pill renders a donut SVG with slice circles + legend %", async ({
|
||||
test("sales dashboard pill renders a composed surface: KPI strip + pie + bar (no surrounding card)", async ({
|
||||
page,
|
||||
}) => {
|
||||
// The custom DonutChart renderer (a2ui/renderers.tsx) builds an inline
|
||||
// <svg> with one grey background <circle> + one stroked <circle> per
|
||||
// slice, wrapped in `transform: scaleX(-1)`. The legend rows end in a
|
||||
// percentage like "45%". This is the strongest visual fingerprint of a
|
||||
// correctly-bound catalog PieChart node.
|
||||
const suggestions = page.locator('[data-testid="copilot-suggestion"]');
|
||||
await suggestions
|
||||
.filter({ hasText: "Pie chart — sales by region" })
|
||||
.first()
|
||||
.click();
|
||||
await clickPill(
|
||||
page,
|
||||
"Show my sales dashboard",
|
||||
"Show me my sales dashboard for this quarter.",
|
||||
);
|
||||
|
||||
// At least background circle + 2 slice circles. 90s budget: on
|
||||
// The hero surface must contain a 4-tile KPI Metric row AND both
|
||||
// charts (no surrounding Card — the charts carry their own card
|
||||
// chrome). Composition rule (sales-context.ts) + D5 probe + aimock
|
||||
// fixtures all pin the hero at 4 Metric tiles. A single lonely
|
||||
// widget is the regression OSS-136 was filed about. 90s budget: on
|
||||
// cold starts the secondary-LLM `generate_a2ui` pass can eat most
|
||||
// of a minute before emitting the PieChart node.
|
||||
const circles = page.locator("svg circle");
|
||||
// of a minute.
|
||||
const metrics = page.locator('[data-testid="declarative-metric"]');
|
||||
await expect
|
||||
.poll(async () => await circles.count(), { timeout: 90_000 })
|
||||
.toBeGreaterThanOrEqual(3);
|
||||
.poll(async () => await metrics.count(), { timeout: 90_000 })
|
||||
.toBeGreaterThanOrEqual(4);
|
||||
|
||||
// A legend row with an integer percentage (e.g. "45%").
|
||||
await expect(page.getByText(/\b\d+%/).first()).toBeVisible({
|
||||
timeout: 10_000,
|
||||
});
|
||||
});
|
||||
// PieChart: recharts donut (mirrors beautiful-chat's sales dashboard) —
|
||||
// one sector path per slice.
|
||||
const pie = page.locator('[data-testid="declarative-pie-chart"]');
|
||||
await expect(pie.first()).toBeVisible({ timeout: 60_000 });
|
||||
const sectors = pie.locator(".recharts-pie-sector");
|
||||
await expect
|
||||
.poll(async () => await sectors.count(), { timeout: 15_000 })
|
||||
.toBeGreaterThanOrEqual(2);
|
||||
|
||||
test("BarChart pill renders a recharts bar chart with rectangles", async ({
|
||||
page,
|
||||
}) => {
|
||||
// BarChart renderer uses a recharts ResponsiveContainer (height 280) +
|
||||
// a custom shape (AnimatedBar with `barSlideIn` keyframe). We only
|
||||
// assert on stable recharts markers (class names unchanged across
|
||||
// versions) — the keyframe-specific CSS is a visual detail not worth
|
||||
// asserting via DOM.
|
||||
const suggestions = page.locator('[data-testid="copilot-suggestion"]');
|
||||
await suggestions
|
||||
.filter({ hasText: "Bar chart — quarterly revenue" })
|
||||
.first()
|
||||
.click();
|
||||
|
||||
// 90s budget for the same cold-start reason as PieChart above.
|
||||
const barChartRoot = page.locator(".recharts-responsive-container").first();
|
||||
await expect(barChartRoot).toBeVisible({ timeout: 90_000 });
|
||||
|
||||
// At least 2 bar rectangles should render. The custom shape renders a
|
||||
// recharts <Rectangle> inside a <g>, which keeps the standard class.
|
||||
// BarChart: recharts markers are stable across versions.
|
||||
const bar = page.locator('[data-testid="declarative-bar-chart"]');
|
||||
await expect(bar.first()).toBeVisible({ timeout: 60_000 });
|
||||
const bars = page.locator(".recharts-bar-rectangle");
|
||||
await expect
|
||||
.poll(async () => await bars.count(), { timeout: 15_000 })
|
||||
.toBeGreaterThanOrEqual(2);
|
||||
|
||||
// Regression guard (#4734): the deployed KPI / dashboard pills used to
|
||||
// loop with "A2UI render error: Cannot create component root without a
|
||||
// type" because the secondary LLM's `render_a2ui` tool call was
|
||||
// intercepted by the A2UI middleware before our defensive validation
|
||||
// could drop malformed components. Renaming to `_design_a2ui_surface`
|
||||
// killed the bypass; assert no A2UI render-error banners are visible.
|
||||
// Regression guard (#4734): no A2UI render-error banners (malformed
|
||||
// secondary-LLM output used to loop with "Cannot create component root
|
||||
// without a type").
|
||||
await expect(
|
||||
page.getByText(/Cannot create component .* without a type/i),
|
||||
).toHaveCount(0);
|
||||
await expect(page.getByText(/Catalog not found/i)).toHaveCount(0);
|
||||
|
||||
// Regression guard: only one bar chart surface (one ResponsiveContainer)
|
||||
// should render — looping renders would stack multiple.
|
||||
// Regression guard: exactly one composed surface — pie + bar each use a
|
||||
// ResponsiveContainer, so the hero dashboard yields exactly 2.
|
||||
// Fewer = under-composed surface (a lonely chart, OSS-136 regression);
|
||||
// more = looping/duplicated renders.
|
||||
const allCharts = page.locator(".recharts-responsive-container");
|
||||
await expect
|
||||
.poll(async () => await allCharts.count(), { timeout: 5_000 })
|
||||
.toBeLessThanOrEqual(1);
|
||||
.toEqual(2);
|
||||
|
||||
// Composition rule (OSS-136 — QA `qa/declarative-gen-ui.md`): the hero
|
||||
// dashboard has NO surrounding Card. The charts carry their own card
|
||||
// chrome, so wrapping them in an extra Card is a planner-side
|
||||
// over-composition regression. Assert zero `declarative-card` mounts.
|
||||
await expect(page.getByTestId("declarative-card")).toHaveCount(0);
|
||||
});
|
||||
|
||||
test("KPI dashboard pill renders at least 3 Metric tiles", async ({
|
||||
test("team performance pill renders a DataTable with rep rows", async ({
|
||||
page,
|
||||
}) => {
|
||||
const suggestions = page.locator('[data-testid="copilot-suggestion"]');
|
||||
await suggestions
|
||||
.filter({ hasText: "Show a KPI dashboard" })
|
||||
.first()
|
||||
.click();
|
||||
await clickPill(
|
||||
page,
|
||||
"Team performance",
|
||||
"How are our sales reps performing against quota?",
|
||||
);
|
||||
|
||||
// Each Metric renderer emits `data-testid="declarative-metric"`.
|
||||
// The component tree is: label (uppercase) + value + optional trend arrow.
|
||||
const metrics = page.locator('[data-testid="declarative-metric"]');
|
||||
const table = page.locator('[data-testid="declarative-data-table"]');
|
||||
await expect(table.first()).toBeVisible({ timeout: 90_000 });
|
||||
|
||||
// At least 2 body rows — a header-only table is an under-specified
|
||||
// surface (the planner forgot the `rows` prop).
|
||||
const rows = table.locator("tbody tr");
|
||||
await expect
|
||||
.poll(async () => await metrics.count(), { timeout: 90_000 })
|
||||
.toBeGreaterThanOrEqual(3);
|
||||
.poll(async () => await rows.count(), { timeout: 15_000 })
|
||||
.toBeGreaterThanOrEqual(2);
|
||||
|
||||
// The surface is dashboardy, not a bare table: a quota-attainment
|
||||
// BarChart accompanies it.
|
||||
await expect(
|
||||
page.locator('[data-testid="declarative-bar-chart"]').first(),
|
||||
).toBeVisible({ timeout: 15_000 });
|
||||
});
|
||||
|
||||
test("Status report pill renders a Card with a StatusBadge pill", async ({
|
||||
page,
|
||||
}) => {
|
||||
const suggestions = page.locator('[data-testid="copilot-suggestion"]');
|
||||
await suggestions.filter({ hasText: "Status report" }).first().click();
|
||||
test("at-risk pill renders StatusBadge pills", async ({ page }) => {
|
||||
await clickPill(
|
||||
page,
|
||||
"Anything at risk?",
|
||||
"Are any accounts or pipeline deals at risk this quarter?",
|
||||
);
|
||||
|
||||
// StatusBadge renderer emits `data-testid="declarative-status-badge"`.
|
||||
// One severity badge per at-risk account (3 in the dataset).
|
||||
const badges = page.locator('[data-testid="declarative-status-badge"]');
|
||||
await expect
|
||||
.poll(async () => await badges.count(), { timeout: 90_000 })
|
||||
.toBeGreaterThanOrEqual(1);
|
||||
.toBeGreaterThanOrEqual(3);
|
||||
|
||||
// The surface is a risk panel, not bare cards: a KPI strip of three
|
||||
// tiles (ARR at risk / accounts at risk / biggest exposure) leads
|
||||
// it. QA + composition rule require all three — fewer is an
|
||||
// under-specified surface.
|
||||
const metrics = page.locator('[data-testid="declarative-metric"]');
|
||||
await expect
|
||||
.poll(async () => await metrics.count(), { timeout: 15_000 })
|
||||
.toBeGreaterThanOrEqual(3);
|
||||
|
||||
// Composition rule (QA `qa/declarative-gen-ui.md`): the at-risk pill
|
||||
// renders StatusBadge cards + a KPI strip — NO charts or tables.
|
||||
// Any chart/table mount here is a planner over-composition regression.
|
||||
await expect(page.getByTestId("declarative-pie-chart")).toHaveCount(0);
|
||||
await expect(page.getByTestId("declarative-bar-chart")).toHaveCount(0);
|
||||
await expect(page.getByTestId("declarative-data-table")).toHaveCount(0);
|
||||
});
|
||||
|
||||
test("top account pill renders InfoRow facts", async ({ page }) => {
|
||||
await clickPill(
|
||||
page,
|
||||
"Top account details",
|
||||
"Pull up the details on our biggest account.",
|
||||
);
|
||||
|
||||
// The account card stacks label/value facts (owner, region, ARR,
|
||||
// renewal, last contact) — require at least 3 InfoRows.
|
||||
const infoRows = page.locator('[data-testid="declarative-info-row"]');
|
||||
await expect
|
||||
.poll(async () => await infoRows.count(), { timeout: 90_000 })
|
||||
.toBeGreaterThanOrEqual(3);
|
||||
|
||||
// The surface is dashboardy, not a bare fact list: a product-line
|
||||
// PieChart accompanies it.
|
||||
await expect(
|
||||
page.locator('[data-testid="declarative-pie-chart"]').first(),
|
||||
).toBeVisible({ timeout: 15_000 });
|
||||
|
||||
// Composition rule (QA `qa/declarative-gen-ui.md`): the top-account
|
||||
// pill renders a Card of InfoRow facts + a product-line PieChart —
|
||||
// NO DataTable, NO StatusBadge. Either is a planner over-composition
|
||||
// regression.
|
||||
await expect(page.getByTestId("declarative-data-table")).toHaveCount(0);
|
||||
await expect(page.getByTestId("declarative-status-badge")).toHaveCount(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
/**
|
||||
* E2E spec for the Declarative UI: Hashbrown demo. Selectors match the
|
||||
* chart/metric components' `data-testid` hooks. Covers 3 suggestion
|
||||
* flows + page-load smoke; timeouts are streaming-friendly because
|
||||
* hashbrown assembles UI progressively from structured output.
|
||||
*/
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
test.describe("Declarative UI: Hashbrown", () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/declarative-hashbrown");
|
||||
});
|
||||
|
||||
test("page loads with header, suggestion pills, and chat composer", async ({
|
||||
page,
|
||||
}) => {
|
||||
await expect(
|
||||
page.getByRole("heading", { name: "Declarative UI: Hashbrown" }),
|
||||
).toBeVisible();
|
||||
await expect(page.getByText("Sales dashboard").first()).toBeVisible();
|
||||
await expect(page.getByText("Revenue by category").first()).toBeVisible();
|
||||
await expect(page.getByText("Expense trend").first()).toBeVisible();
|
||||
});
|
||||
|
||||
test("sales-dashboard suggestion triggers a hashbrown render", async ({
|
||||
page,
|
||||
}) => {
|
||||
await page.getByText("Sales dashboard").first().click();
|
||||
|
||||
const metricCard = page.locator('[data-testid="metric-card"]').first();
|
||||
const chart = page
|
||||
.locator('[data-testid="bar-chart"], [data-testid="pie-chart"]')
|
||||
.first();
|
||||
|
||||
await expect(metricCard).toBeVisible({ timeout: 60000 });
|
||||
await expect(chart).toBeVisible({ timeout: 60000 });
|
||||
});
|
||||
|
||||
test("revenue-by-category suggestion renders a pie chart", async ({
|
||||
page,
|
||||
}) => {
|
||||
await page.getByText("Revenue by category").first().click();
|
||||
await expect(page.locator('[data-testid="pie-chart"]').first()).toBeVisible(
|
||||
{ timeout: 60000 },
|
||||
);
|
||||
});
|
||||
|
||||
test("expense-trend suggestion renders a bar chart", async ({ page }) => {
|
||||
await page.getByText("Expense trend").first().click();
|
||||
await expect(page.locator('[data-testid="bar-chart"]').first()).toBeVisible(
|
||||
{ timeout: 60000 },
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
/**
|
||||
* E2E spec for the Declarative UI: json-render demo. Structurally
|
||||
* mirrors `gen-ui-tool-based.spec.ts` so the dashboard's BYOC rows
|
||||
* exercise the same surfaces (json-render-root + metric-card + chart).
|
||||
*/
|
||||
test.describe("Declarative UI: json-render", () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/declarative-json-render");
|
||||
});
|
||||
|
||||
test("page loads with chat composer and suggestion pills", async ({
|
||||
page,
|
||||
}) => {
|
||||
// Chat composer
|
||||
await expect(
|
||||
page.locator('textarea, [placeholder*="message"]').first(),
|
||||
).toBeVisible({ timeout: 10000 });
|
||||
|
||||
// Suggestion pills driven by useConfigureSuggestions. The
|
||||
// CopilotChat welcome screen renders titles as buttons/links.
|
||||
await expect(page.getByText("Sales dashboard")).toBeVisible({
|
||||
timeout: 10000,
|
||||
});
|
||||
await expect(page.getByText("Revenue by category")).toBeVisible();
|
||||
await expect(page.getByText("Expense trend")).toBeVisible();
|
||||
});
|
||||
|
||||
test("sales dashboard request renders a json-render tree", async ({
|
||||
page,
|
||||
}) => {
|
||||
const input = page.locator('textarea, [placeholder*="message"]').first();
|
||||
await input.fill(
|
||||
"Show me the sales dashboard with metrics and a revenue chart",
|
||||
);
|
||||
await input.press("Enter");
|
||||
|
||||
// The JsonRenderAssistantMessage slot wraps renders in this testid.
|
||||
await expect(
|
||||
page.locator('[data-testid="json-render-root"]').first(),
|
||||
).toBeVisible({ timeout: 60000 });
|
||||
|
||||
// A MetricCard should appear in the rendered tree.
|
||||
await expect(
|
||||
page.locator('[data-testid="metric-card"]').first(),
|
||||
).toBeVisible({ timeout: 60000 });
|
||||
|
||||
// ...plus at least one chart (either shape).
|
||||
await expect(
|
||||
page
|
||||
.locator('[data-testid="bar-chart"], [data-testid="pie-chart"]')
|
||||
.first(),
|
||||
).toBeVisible({ timeout: 60000 });
|
||||
});
|
||||
|
||||
test("revenue-by-category request renders a pie chart", async ({ page }) => {
|
||||
const input = page.locator('textarea, [placeholder*="message"]').first();
|
||||
await input.fill("Break down revenue by category as a pie chart");
|
||||
await input.press("Enter");
|
||||
|
||||
await expect(page.locator('[data-testid="pie-chart"]').first()).toBeVisible(
|
||||
{ timeout: 60000 },
|
||||
);
|
||||
});
|
||||
|
||||
test("expense-trend request renders a bar chart", async ({ page }) => {
|
||||
const input = page.locator('textarea, [placeholder*="message"]').first();
|
||||
await input.fill("Show me monthly expenses as a bar chart");
|
||||
await input.press("Enter");
|
||||
|
||||
await expect(page.locator('[data-testid="bar-chart"]').first()).toBeVisible(
|
||||
{ timeout: 60000 },
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,119 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
// QA reference: qa/reasoning-custom.md
|
||||
// Demo source: src/app/demos/reasoning-custom/{page.tsx, reasoning-block.tsx}
|
||||
//
|
||||
// The demo mounts a custom `reasoningMessage` slot (`ReasoningBlock`) that
|
||||
// renders an amber banner with `data-testid="reasoning-block"`. The label
|
||||
// inside the banner reads "Thinking…" while the agent is streaming, then
|
||||
// flips to "Agent reasoning" once streaming settles. The backend uses
|
||||
// `deepagents.create_deep_agent` with a reasoning-capable OpenAI model
|
||||
// (`gpt-5-mini` by default, override via `OPENAI_REASONING_MODEL`) routed
|
||||
// through the Responses API so the model's chain of thought streams as
|
||||
// AG-UI REASONING_MESSAGE_* events.
|
||||
//
|
||||
// Streaming assertions exercise the aimock fixture in
|
||||
// `showcase/aimock/d5-all.json` — its `reasoning` field makes aimock emit
|
||||
// `response.reasoning_summary_text.delta` events deterministically, no
|
||||
// real LLM call required. Local stack: a "Show reasoning" suggestion pill
|
||||
// fires the same prompt, so a single click reproduces the streaming UX.
|
||||
//
|
||||
// Selectors are testid / role / stable text only. No LLM-text assertions.
|
||||
|
||||
const REASONING_PROMPT = "show your reasoning step by step";
|
||||
|
||||
test.describe("Reasoning: Custom", () => {
|
||||
test.setTimeout(120_000);
|
||||
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/reasoning-custom");
|
||||
});
|
||||
|
||||
test("page loads with chat input", async ({ page }) => {
|
||||
await expect(page.getByPlaceholder("Type a message")).toBeVisible();
|
||||
});
|
||||
|
||||
test("send button is visible alongside the input", async ({ page }) => {
|
||||
await expect(page.getByPlaceholder("Type a message")).toBeVisible();
|
||||
await expect(
|
||||
page.locator('[data-testid="copilot-send-button"]').first(),
|
||||
).toBeVisible();
|
||||
});
|
||||
|
||||
test("typing a prompt and submitting does not crash the UI", async ({
|
||||
page,
|
||||
}) => {
|
||||
const input = page.getByPlaceholder("Type a message");
|
||||
await input.fill("Say hello in one short sentence.");
|
||||
await page.locator('[data-testid="copilot-send-button"]').first().click();
|
||||
|
||||
await expect(input).toHaveValue("", { timeout: 10_000 });
|
||||
});
|
||||
|
||||
// --- Reasoning-block streaming coverage --------------------------------
|
||||
|
||||
test("reasoning prompt renders a reasoning-block before the answer", async ({
|
||||
page,
|
||||
}) => {
|
||||
const input = page.getByPlaceholder("Type a message");
|
||||
await input.fill(REASONING_PROMPT);
|
||||
await page.locator('[data-testid="copilot-send-button"]').first().click();
|
||||
|
||||
const reasoningBlock = page
|
||||
.locator('[data-testid="reasoning-block"]')
|
||||
.first();
|
||||
await expect(reasoningBlock).toBeVisible({ timeout: 60_000 });
|
||||
await expect(
|
||||
reasoningBlock.getByText("Reasoning", { exact: true }),
|
||||
).toBeVisible({ timeout: 10_000 });
|
||||
});
|
||||
|
||||
test("reasoning-block label flips from Thinking to Agent reasoning", async ({
|
||||
page,
|
||||
}) => {
|
||||
const input = page.getByPlaceholder("Type a message");
|
||||
await input.fill(REASONING_PROMPT);
|
||||
await page.locator('[data-testid="copilot-send-button"]').first().click();
|
||||
|
||||
const reasoningBlock = page
|
||||
.locator('[data-testid="reasoning-block"]')
|
||||
.first();
|
||||
await expect(reasoningBlock).toBeVisible({ timeout: 60_000 });
|
||||
await expect(reasoningBlock.getByText("Agent reasoning")).toBeVisible({
|
||||
timeout: 90_000,
|
||||
});
|
||||
});
|
||||
|
||||
test("reasoning-block accumulates italic reasoning content", async ({
|
||||
page,
|
||||
}) => {
|
||||
const input = page.getByPlaceholder("Type a message");
|
||||
await input.fill(REASONING_PROMPT);
|
||||
await page.locator('[data-testid="copilot-send-button"]').first().click();
|
||||
|
||||
const reasoningBlock = page
|
||||
.locator('[data-testid="reasoning-block"]')
|
||||
.first();
|
||||
await expect(reasoningBlock).toBeVisible({ timeout: 60_000 });
|
||||
// ReasoningBlock renders content inside a div with italic class when
|
||||
// `message.content` is non-empty (see reasoning-block.tsx).
|
||||
await expect(reasoningBlock.locator(".italic")).toBeVisible({
|
||||
timeout: 45_000,
|
||||
});
|
||||
});
|
||||
|
||||
test("Show reasoning suggestion pill fires the reasoning prompt", async ({
|
||||
page,
|
||||
}) => {
|
||||
// The page wires `useConfigureSuggestions` with a single "Show reasoning"
|
||||
// pill whose message exactly matches the aimock fixture key.
|
||||
const pill = page.getByRole("button", { name: /Show reasoning/i }).first();
|
||||
await expect(pill).toBeVisible({ timeout: 30_000 });
|
||||
await pill.click();
|
||||
|
||||
const reasoningBlock = page
|
||||
.locator('[data-testid="reasoning-block"]')
|
||||
.first();
|
||||
await expect(reasoningBlock).toBeVisible({ timeout: 60_000 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,41 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
// QA reference: qa/reasoning-default.md
|
||||
// Demo source: src/app/demos/reasoning-default/page.tsx
|
||||
//
|
||||
// This cell does NOT override the `reasoningMessage` slot. CopilotKit's
|
||||
// built-in `CopilotChatReasoningMessage` renders the reasoning as a
|
||||
// collapsible card. The page exposes a "Show reasoning" suggestion pill
|
||||
// whose message matches the aimock fixture in showcase/aimock/d5-all.json,
|
||||
// so streaming is deterministic in CI.
|
||||
|
||||
test.describe("Reasoning: Default", () => {
|
||||
test.setTimeout(120_000);
|
||||
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/reasoning-default");
|
||||
});
|
||||
|
||||
test("page renders without errors", async ({ page }) => {
|
||||
await expect(
|
||||
page.locator('[data-testid="copilot-chat-input"]'),
|
||||
).toBeVisible();
|
||||
});
|
||||
|
||||
test("Show reasoning pill renders a reasoning-role message", async ({
|
||||
page,
|
||||
}) => {
|
||||
const pill = page.getByRole("button", { name: /Show reasoning/i }).first();
|
||||
await expect(pill).toBeVisible({ timeout: 30_000 });
|
||||
await pill.click();
|
||||
|
||||
// The cell uses CopilotKit's default `CopilotChatReasoningMessage`,
|
||||
// which doesn't emit a testid — its visible signal is the
|
||||
// streaming/complete header label ("Thinking…" while streaming,
|
||||
// "Thought for …" once complete). Asserting on either label proves
|
||||
// the reasoning collapsible mounted.
|
||||
await expect(page.getByText(/Thinking…|Thought for/i).first()).toBeVisible({
|
||||
timeout: 60_000,
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -72,7 +72,7 @@ test.describe("Shared State (Read + Write)", () => {
|
||||
});
|
||||
// Negative assertion: the wrong-fixture response is gone.
|
||||
await expect(assistantMessage).not.toContainText(
|
||||
/Research the topic[\s\S]*Outline key points/i,
|
||||
/Research the topic.*Outline key points/is,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
test.describe("Shared State (Writing)", () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/shared-state-write");
|
||||
});
|
||||
|
||||
test("page loads with Sales Pipeline dashboard", async ({ page }) => {
|
||||
await expect(page.getByText("Sales Pipeline")).toBeVisible({
|
||||
timeout: 10000,
|
||||
});
|
||||
});
|
||||
|
||||
test("sidebar is open with Sales Pipeline Assistant title", async ({
|
||||
page,
|
||||
}) => {
|
||||
await expect(page.getByText("Sales Pipeline Assistant")).toBeVisible({
|
||||
timeout: 10000,
|
||||
});
|
||||
});
|
||||
|
||||
test("dashboard shows pipeline summary with Total Pipeline metric", async ({
|
||||
page,
|
||||
}) => {
|
||||
await expect(page.getByText("Total Pipeline")).toBeVisible({
|
||||
timeout: 10000,
|
||||
});
|
||||
});
|
||||
|
||||
test("can send message through sidebar and get response", async ({
|
||||
page,
|
||||
}) => {
|
||||
const input = page.locator('textarea, [placeholder*="message"]').first();
|
||||
await input.fill("Add a new deal for Acme Corp worth $50,000");
|
||||
await input.press("Enter");
|
||||
|
||||
await expect(page.locator('[data-role="assistant"]').first()).toBeVisible({
|
||||
timeout: 30000,
|
||||
});
|
||||
});
|
||||
|
||||
test("deal list shows empty state or active deals with interactive elements", async ({
|
||||
page,
|
||||
}) => {
|
||||
// Dashboard should show either deals or the empty state with an "Add a deal" button
|
||||
const addButton = page.getByRole("button", { name: /add a deal/i });
|
||||
const activeDealsList = page.getByText("Active Deals");
|
||||
|
||||
await expect(addButton.or(activeDealsList).first()).toBeVisible({
|
||||
timeout: 10000,
|
||||
});
|
||||
});
|
||||
|
||||
test("agent can modify dashboard state through chat", async ({ page }) => {
|
||||
const input = page.locator('textarea, [placeholder*="message"]').first();
|
||||
await input.fill(
|
||||
"Create three sample deals: Widget Co for $10,000, Gadget Inc for $25,000, and Tech LLC for $50,000",
|
||||
);
|
||||
await input.press("Enter");
|
||||
|
||||
// Wait for agent to respond
|
||||
await expect(page.locator('[data-role="assistant"]').first()).toBeVisible({
|
||||
timeout: 45000,
|
||||
});
|
||||
|
||||
// After the agent writes state, the dashboard should reflect deals
|
||||
// Either "Active Deals" heading appears or the pipeline values change
|
||||
await expect(
|
||||
page.getByText("Active Deals").or(page.getByText(/\$\d{1,3}(,\d{3})*/)),
|
||||
).toBeVisible({ timeout: 30000 });
|
||||
});
|
||||
|
||||
test("can add a deal via UI button", async ({ page }) => {
|
||||
// Look for an "Add" or "+" button on the dashboard
|
||||
const addBtn = page.locator('button:has-text("Add"), button:has-text("+")');
|
||||
await expect(addBtn.first()).toBeVisible({ timeout: 5000 });
|
||||
const initialCount = await page
|
||||
.locator('[data-testid="todo-card"], .todo-card')
|
||||
.count();
|
||||
await addBtn.first().click();
|
||||
// New item should appear
|
||||
await expect(
|
||||
page.locator('[data-testid="todo-card"], .todo-card'),
|
||||
).toHaveCount(initialCount + 1, { timeout: 5000 });
|
||||
});
|
||||
|
||||
test("can toggle a deal completion via checkbox", async ({ page }) => {
|
||||
// First create some deals via chat so checkboxes exist
|
||||
const input = page.locator('textarea, [placeholder*="message"]').first();
|
||||
await input.fill("Add a deal for Demo Corp worth $20,000");
|
||||
await input.press("Enter");
|
||||
|
||||
await expect(page.locator('[data-role="assistant"]').first()).toBeVisible({
|
||||
timeout: 30000,
|
||||
});
|
||||
|
||||
const toggleBtn = page.locator('[data-testid="toggle-completed"]').first();
|
||||
await expect(toggleBtn).toBeVisible({ timeout: 5000 });
|
||||
|
||||
// The toggle is a styled button, not a native checkbox.
|
||||
// Check whether the card has the completed opacity class before/after click.
|
||||
const card = page.locator('[data-testid="todo-card"]').first();
|
||||
const hadOpacity = await card.evaluate((el) =>
|
||||
el.classList.contains("opacity-60"),
|
||||
);
|
||||
await toggleBtn.click();
|
||||
if (hadOpacity) {
|
||||
await expect(card).not.toHaveClass(/opacity-60/, { timeout: 3000 });
|
||||
} else {
|
||||
await expect(card).toHaveClass(/opacity-60/, { timeout: 3000 });
|
||||
}
|
||||
});
|
||||
});
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
// QA reference: qa/threadid-frontend-tool-roundtrip.md
|
||||
// Demo source: src/app/demos/threadid-frontend-tool-roundtrip/page.tsx
|
||||
//
|
||||
// The source-level regression for ENT-658 lives in react-core. This smoke keeps
|
||||
// the showcase demo route and generated-thread toggle covered without depending
|
||||
// on fixture-driven tool execution in the standalone showcase package.
|
||||
|
||||
test.describe("Thread ID frontend-tool round trip", () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await page.goto("/demos/threadid-frontend-tool-roundtrip");
|
||||
});
|
||||
|
||||
test("page loads with generated-thread mode selected", async ({ page }) => {
|
||||
await expect(page.getByPlaceholder("Type a message")).toBeVisible();
|
||||
await expect(page.getByLabel("Explicit threadId")).not.toBeChecked();
|
||||
await expect(page.getByTestId("ent-658-thread-mode")).toHaveText(
|
||||
/SDK-generated thread/i,
|
||||
);
|
||||
|
||||
await page.getByLabel("Explicit threadId").check();
|
||||
await expect(page.getByTestId("ent-658-thread-mode")).toHaveText(
|
||||
/explicit thread/i,
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user