botpress__skills/skills/adk-evals/references/eval-format.md

Eval File Format

Evals are TypeScript files in the evals/ directory. Each file exports one or more eval definitions using new Eval.

File Location

Evals live in evals/ at the project root (not inside src/). Create it if it doesn't exist:

my-agent/
├── agent.config.ts
├── src/
│   ├── actions/
│   └── workflows/
└── evals/              ← create this
    ├── greeting.eval.ts
    └── billing.eval.ts

• Naming: *.eval.ts convention (recommended)
• Auto-discovery: All files in evals/ are picked up by adk evals

Full Structure

import { Eval } from '@botpress/evals'

export default new Eval({
  name: 'my-eval', // unique identifier (required)
  description: 'What this tests', // optional
  type: 'regression', // 'capability' or 'regression' — optional, for filtering
  tags: ['tools', 'multi-turn'], // optional, for filtering

  setup: {
    // Seed state or trigger a workflow before the conversation (optional)
  },

  conversation: [
    {
      user: 'message from user', // or: event, expectSilence
      assert: {
        /* per-turn assertions */
      },
    },
  ],

  outcome: {
    /* post-conversation assertions (optional) */
  },

  options: {
    /* per-eval overrides (optional) */
  },
})

Types

Type	Purpose
capability	Verify the bot can do something new
regression	Verify the bot still does it correctly

Multiple Evals Per File

export const greeting = new Eval({ name: 'greeting', ... })
export const farewell = new Eval({ name: 'farewell', ... })

---

Conversation Turns

Each entry in conversation is one turn. A turn must have either user or event.

User Message

{
  user: 'What is my account balance?',
  assert: { /* assertions on the bot's response */ },
}

Event Trigger

Push a custom event instead of a user message. The turn carries only a payload; the bot receives it as a chat:custom event and reads the data at event.payload.payload.

{
  event: {
    payload: { orderId: 'ORD-001', total: 49.99 },
  },
  assert: {
    workflow: [{ name: 'orderConfirmation', entered: true }],
  },
}

Expect Silence

Assert the bot does not respond. Add expectSilence: true to any turn.

// Silence after a user message
{ user: 'Please ignore this.', expectSilence: true }

// Silence after an event
{ event: { payload: { kind: 'ping' } }, expectSilence: true }

Note: expectSilence is mutually exclusive with assert.response. Every turn must have user or event — expectSilence is a flag on top of that, not a standalone turn type.

---

Assertion Categories

Response

What the bot said back.

assert: {
  response: [
    { contains: 'ticket' },                    // substring present
    { not_contains: 'error' },                 // substring absent
    { matches: 'TKT-\\d{3}' },               // regex match
    { llm_judge: 'Response confirms the ticket was created' }, // AI judge, scores 1–5
  ],
}

Tools

Which tools the bot called and with what parameters.

assert: {
  tools: [
    { called: 'createTicket' },                          // tool was invoked
    { called: 'createTicket', params: {                  // with specific params
        priority: { equals: 'high' },
        department: { contains: 'Engineering' },
    }},
    { not_called: 'deleteTicket' },                      // tool was NOT invoked
    { call_order: ['lookupUser', 'createTicket'] },      // ordered calls
  ],
}

State

Bot, user, or conversation state values after the turn.

assert: {
  state: [
    { path: 'conversation.topic', equals: 'support' },  // exact value
    { path: 'conversation.topic', changed: true },       // value changed from before
    { path: 'bot.ticketCount', equals: 3 },
  ],
}

Workflow

Workflow execution (verified via trace spans).

assert: {
  workflow: [
    { name: 'onboarding', entered: true },    // workflow was started
    { name: 'onboarding', completed: true },  // workflow finished
  ],
}

Timing

How long the bot took to respond (milliseconds).

assert: {
  timing: [
    { response_time: { lte: 5000 } },  // must respond within 5s
    { response_time: { gte: 100 } },   // sanity-check: not suspiciously fast
  ],
}

---

Match Operators

Used in tool params, state values, and workflow params:

Operator	Example	Description
equals	{ equals: 'urgent' }	Exact match
contains	{ contains: 'HR' }	Substring
not_contains	{ not_contains: 'test' }	Excludes substring
matches	{ matches: '\\d+' }	Regex
in	{ in: ['high', 'urgent'] }	One of
exists	{ exists: true }	Property exists

Numeric-only operators (for timing and numeric state values):

Operator	Example	Description
gte	{ gte: 100 }	Greater than or equal
lte	{ lte: 5000 }	Less than or equal

---

Eval Setup

Use setup to put the bot in a known state before the conversation starts.

Seed State

Pre-populate bot, user, or conversation state.

setup: {
  state: {
    bot: { welcomeMessageSent: true },
    user: { plan: 'pro' },
    conversation: { topic: 'billing' },
  },
}

The seeded state becomes the baseline for changed assertions — { changed: false } passes if the value matches the seeded value at the end.

Trigger a Workflow

Start a workflow before the conversation begins.

setup: {
  workflow: {
    trigger: 'onboarding',
    input: { userId: 'test-user-1' },
  },
}

Both can be combined:

setup: {
  state: {
    bot: { sleepDurationMs: 3000 }, // 3 seconds instead of default 10 minutes
  },
  workflow: {
    trigger: 'reminderFlow',
    input: { userId: 'test-user-1' },
  },
}

Testing step.sleep(): Server-side scheduling can't be fast-forwarded. Workaround: make the sleep duration configurable by reading it from bot state in the workflow, then seed a short value in ms (e.g. 3000) in setup.state.bot.

---

Outcome Assertions

Run once after all conversation turns complete. Supports state and workflow (not response or tools — those are per-turn only).

outcome: {
  state: [
    { path: 'conversation.resolved', equals: true },
  ],
  workflow: [
    { name: 'ticketFlow', completed: true },
  ],
}

---

Options

Override defaults for a specific eval. Cascades: eval options → agent config → default.

options: {
  idleTimeout: 60000,       // ms to wait for bot response (default: 30000)
  judgePassThreshold: 4,    // llm_judge score required to pass, 1–5 (default: 3)
}

Agent-level defaults in agent.config.ts:

export default defineConfig({
  evals: {
    idleTimeout: 20000,
    judgePassThreshold: 3,
    judgeModel: 'fast', // 'fast', 'best', or a model ref like 'openai:gpt-4o'
  },
})

---

Common Mistakes

❌ Turn with neither user nor event

// WRONG — every turn needs a trigger
{
  expectSilence: true
}

✅ Correct

{ user: 'hello', expectSilence: true }

---

❌ expectSilence with assert.response

// WRONG — mutually exclusive
{ user: 'hello', expectSilence: true, assert: { response: [{ contains: 'hi' }] } }

✅ Correct — pick one

{ user: 'hello', expectSilence: true }
// or
{ user: 'hello', assert: { response: [{ contains: 'hi' }] } }

---

❌ Both user and event on the same turn

// WRONG — mutually exclusive
{ user: 'hello', event: { payload: { amount: 50 } } }

✅ Correct — use separate turns

{
  event: {
    payload: {
      amount: 50
    }
  }
}

See Also

• testing-workflow.md — Running evals, interpreting output, the write → test → iterate loop
• test-patterns.md — Per-primitive testing patterns