mirror of
https://github.com/vercel/eve.git
synced 2026-09-20 05:35:39 +08:00
feat(evals): add a Datadog Experiments reporter (#2860)
This commit is contained in:
mehulsonowal
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"eve": patch
|
||||
---
|
||||
|
||||
Add a Datadog eval reporter that creates one LLM Observability Experiment per eve eval run and submits eval assertion metrics through the optional `dd-trace` package. Opted-in eval inputs are pushed as versioned dataset records and linked to their experiment rows. The integration is tested against the public dataset and external Experiment APIs in `dd-trace@6.13.0`.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Datadog eval reporter fixture
|
||||
|
||||
This deterministic mock-model app exercises the Datadog eval reporter against
|
||||
`dd-trace@6.13.0`.
|
||||
|
||||
```bash
|
||||
export DD_API_KEY="..."
|
||||
export DD_APP_KEY="..."
|
||||
export DD_SITE="datadoghq.com"
|
||||
|
||||
pnpm --filter datadog-eval-reporter-fixture eval
|
||||
```
|
||||
|
||||
Without both Datadog keys, the fixture omits the reporter so the shared local,
|
||||
Postgres, and Vercel e2e matrices remain hermetic.
|
||||
|
||||
The credentialed run creates a Datadog dataset with one record containing `Say hello.`,
|
||||
creates one LLM Observability Experiment against that dataset version, links the
|
||||
record to the smoke eval's synthetic experiment span, attaches the assertion
|
||||
metrics, and prints the Dataset and Experiment URLs. The fixture does not export
|
||||
the agent runtime's OpenTelemetry spans; Datadog's public dataset and external
|
||||
Experiment APIs generate the record, trace, and span identifiers.
|
||||
@@ -0,0 +1,9 @@
|
||||
import { e2eAgentConfig } from "@eve-e2e/config";
|
||||
import { defineAgent } from "eve";
|
||||
import { mockModel } from "eve/evals";
|
||||
|
||||
export default defineAgent({
|
||||
...e2eAgentConfig(),
|
||||
model: mockModel(),
|
||||
modelContextWindowTokens: 1_000_000,
|
||||
});
|
||||
@@ -0,0 +1 @@
|
||||
You are a deterministic test agent used to develop the Datadog eval reporter.
|
||||
@@ -0,0 +1,8 @@
|
||||
import { defineEvalConfig } from "eve/evals";
|
||||
import { Datadog } from "eve/evals/reporters";
|
||||
|
||||
const hasDatadogCredentials = Boolean(process.env.DD_API_KEY && process.env.DD_APP_KEY);
|
||||
|
||||
export default defineEvalConfig({
|
||||
reporters: hasDatadogCredentials ? [Datadog({ recordInputs: true })] : [],
|
||||
});
|
||||
@@ -0,0 +1,10 @@
|
||||
import { defineEval } from "eve/evals";
|
||||
|
||||
export default defineEval({
|
||||
description: "Datadog reporter smoke eval.",
|
||||
|
||||
async test(t) {
|
||||
await t.send("Say hello.");
|
||||
t.succeeded();
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "datadog-eval-reporter-fixture",
|
||||
"version": "0.0.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"eval": "eve eval --strict"
|
||||
},
|
||||
"dependencies": {
|
||||
"@eve-e2e/config": "workspace:*",
|
||||
"@workflow/world-postgres": "catalog:",
|
||||
"dd-trace": "6.13.0",
|
||||
"eve": "workspace:*"
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Reporters"
|
||||
description: "Ship eval results to Braintrust experiments or JUnit XML. eve runs and scores everything itself."
|
||||
description: "Ship eval results to Braintrust, Datadog, or JUnit XML. eve runs and scores everything itself."
|
||||
---
|
||||
|
||||
eve runs and grades everything itself; reporters ship the results out. The CLI prints a console summary by default (one line per eval, with failed assertions and their messages), and reporters from `eve/evals/reporters` add destinations on top.
|
||||
@@ -44,6 +44,25 @@ A reporter instance observes the evals that reference it. Share one instance acr
|
||||
|
||||
Braintrust needs its SDK installed in the app and credentials in the environment: install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run the eval without shipping results, which also suppresses config reporters and is useful locally when iterating.
|
||||
|
||||
## Datadog
|
||||
|
||||
`Datadog(...)` uploads eval assertion scores to a Datadog LLM Observability Experiment. Put one instance in the config so it covers the whole run:
|
||||
|
||||
```ts title="evals/evals.config.ts"
|
||||
import { defineEvalConfig } from "eve/evals";
|
||||
import { Datadog } from "eve/evals/reporters";
|
||||
|
||||
export default defineEvalConfig({
|
||||
reporters: [Datadog({ projectName: "weather-agent" })],
|
||||
});
|
||||
```
|
||||
|
||||
The reporter creates one Datadog Experiment, submits one synthetic experiment span per completed eve eval, associates that eval's assertion scores as experiment metrics, and prints the Datadog Experiment URL after the run completes. Without input recording, the external Experiment uses a placeholder dataset and spans are submitted as evals finish. With `recordInputs: true`, the reporter instead creates and pushes a versioned Datadog dataset after the evals finish, adds one record per eval, starts the Experiment against that dataset version, passes each dataset record id to its corresponding experiment span, and prints both Dataset and Experiment URLs. Buffering is necessary because an imperative eval's first `t.send(...)` input is only known after that eval executes. `dd-trace` generates dataset record, experiment trace, and experiment span identifiers; the reporter does not mutate or link the agent runtime's OpenTelemetry spans.
|
||||
|
||||
Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, and repeated or reserved labels receive numeric suffixes. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Pass `recordAssertionDetails: true` only when the destination is also approved for raw assertion-name tags and failed assertion messages in row metadata.
|
||||
|
||||
Datadog needs `dd-trace` installed in the app and Datadog credentials in the environment. The reporter is tested against `dd-trace@6.13.0`; install it with `npm install dd-trace@6.13.0` and set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate. By default, the reporter records assertion scores, eval metadata, and the target URL origin only. Pass `recordInputs: true`, `recordOutputs: true`, `recordExpectedOutputs: true`, or `recordErrors: true` if your destination is approved for eval prompts, outputs, authored expectations, or execution error messages. `recordInputs` stores the eval input in both the linked dataset record and experiment span. Input is read from the eval's first `t.send(...)` message, falling back to the eval description when no message event is available. When `recordExpectedOutputs` is enabled, expected output is read from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` and stored in both the dataset record and experiment span; when no expected output is authored, the input-only dataset record is still valid. Those expected-output keys are excluded from the general metadata payload. Target URL credentials, paths, query parameters, and fragments are never reported.
|
||||
|
||||
## JUnit
|
||||
|
||||
`JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, usually the better fit because CI owns the output path, not the eval:
|
||||
|
||||
@@ -498,6 +498,7 @@
|
||||
"chat": "4.34.0",
|
||||
"chokidar": "5.0.0",
|
||||
"commander": "14.0.3",
|
||||
"dd-trace": "6.13.0",
|
||||
"emulate": "0.6.0",
|
||||
"env-runner": "0.1.16",
|
||||
"eventsource-parser": "3.1.0",
|
||||
@@ -527,6 +528,7 @@
|
||||
"@opentelemetry/api": "^1.0.0",
|
||||
"ai": "catalog:",
|
||||
"braintrust": "^3.0.0",
|
||||
"dd-trace": "^6.13.0",
|
||||
"just-bash": "^3.1.0",
|
||||
"microsandbox": "^0.5.0"
|
||||
},
|
||||
@@ -537,6 +539,9 @@
|
||||
"braintrust": {
|
||||
"optional": true
|
||||
},
|
||||
"dd-trace": {
|
||||
"optional": true
|
||||
},
|
||||
"just-bash": {
|
||||
"optional": true
|
||||
},
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
export { Braintrust, type BraintrustReporterConfig } from "#evals/runner/reporters/braintrust.js";
|
||||
export { Console, type ConsoleReporterConfig } from "#evals/runner/reporters/console.js";
|
||||
export { Datadog, type DatadogReporterConfig } from "#evals/runner/reporters/datadog.js";
|
||||
export { JUnit, type JUnitReporterConfig } from "#evals/runner/reporters/junit.js";
|
||||
export type {
|
||||
EvalReporter,
|
||||
|
||||
@@ -0,0 +1,464 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { createEmptyDerivedFacts } from "#evals/runner/derive-run-facts.js";
|
||||
import { Datadog, type DatadogReporterConfig } from "#evals/reporters/index.js";
|
||||
import type { EveEval, EveEvalResult, EveEvalRunSummary, EveEvalTarget } from "#evals/types.js";
|
||||
|
||||
function makeTarget(kind: "local" | "remote" = "local"): EveEvalTarget {
|
||||
return {
|
||||
capabilities: { devRoutes: kind === "local" },
|
||||
kind,
|
||||
url: kind === "local" ? "http://127.0.0.1:3000" : "https://test.vercel.app",
|
||||
};
|
||||
}
|
||||
|
||||
function makeEval(overrides: Partial<EveEval> = {}): EveEval {
|
||||
return {
|
||||
_tag: "EveEval",
|
||||
id: "eval-1",
|
||||
description: "Say hello",
|
||||
tags: ["smoke"],
|
||||
metadata: {
|
||||
suite: "unit",
|
||||
expectedOutput: "helpful onboarding answer",
|
||||
expected: "legacy expected answer",
|
||||
expected_output: "legacy snake-case expected answer",
|
||||
},
|
||||
async test() {},
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function makeEvalResult(overrides: Partial<EveEvalResult> = {}): EveEvalResult {
|
||||
return {
|
||||
id: "eval-1",
|
||||
result: {
|
||||
output: "actual output",
|
||||
finalMessage: "actual output",
|
||||
status: "completed",
|
||||
events: [
|
||||
{
|
||||
type: "message.received",
|
||||
data: {
|
||||
message: "What should I send you?",
|
||||
sequence: 1,
|
||||
turnId: "turn-1",
|
||||
},
|
||||
meta: { at: "2026-01-01T00:00:00.000Z", id: "event-1" },
|
||||
},
|
||||
],
|
||||
derived: {
|
||||
...createEmptyDerivedFacts(),
|
||||
toolCalls: [
|
||||
{
|
||||
name: "search",
|
||||
input: { query: "test" },
|
||||
output: null,
|
||||
status: "completed",
|
||||
turnIndex: 0,
|
||||
sessionId: "session-123",
|
||||
},
|
||||
],
|
||||
toolCallCount: 1,
|
||||
messageCount: 1,
|
||||
},
|
||||
sessionId: "session-123",
|
||||
traceContexts: [],
|
||||
},
|
||||
assertions: [
|
||||
{ name: "succeeded", score: 1, severity: "gate", passed: true },
|
||||
{ name: "similarity", score: 0.9, severity: "soft", threshold: 0.6, passed: true },
|
||||
{ name: "judge.autoevals.closedQA", score: 1, severity: "soft", passed: true },
|
||||
],
|
||||
verdict: "passed",
|
||||
startedAt: "2026-01-01T00:00:00.000Z",
|
||||
completedAt: "2026-01-01T00:00:01.000Z",
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function makeSummary(result: EveEvalResult = makeEvalResult()): EveEvalRunSummary {
|
||||
return {
|
||||
target: makeTarget(),
|
||||
results: [result],
|
||||
startedAt: "2026-01-01T00:00:00.000Z",
|
||||
completedAt: "2026-01-01T00:00:02.000Z",
|
||||
passed: result.verdict === "passed" ? 1 : 0,
|
||||
failed: result.verdict === "failed" ? 1 : 0,
|
||||
scored: result.verdict === "scored" ? 1 : 0,
|
||||
skipped: result.verdict === "skipped" ? 1 : 0,
|
||||
errored: result.error ? 1 : 0,
|
||||
};
|
||||
}
|
||||
|
||||
function makeConfig(overrides: Partial<DatadogReporterConfig> = {}) {
|
||||
const span = {
|
||||
experimentId: "exp-1",
|
||||
spanId: "span-1",
|
||||
traceId: "trace-1",
|
||||
url: "https://dd.test/span",
|
||||
};
|
||||
const experiment = {
|
||||
experimentId: vi.fn(() => "exp-1"),
|
||||
url: vi.fn(() => "https://dd.test/experiment"),
|
||||
submitSpan: vi.fn(async (_row: unknown) => span),
|
||||
submitEvaluationMetrics: vi.fn(async (_span: unknown, _metrics: unknown) => undefined),
|
||||
close: vi.fn(async () => undefined),
|
||||
};
|
||||
let datasetName = "dataset-1";
|
||||
let datasetRecords: Array<{
|
||||
id: string;
|
||||
inputData: unknown;
|
||||
expectedOutput?: unknown;
|
||||
metadata?: Readonly<Record<string, unknown>>;
|
||||
}> = [];
|
||||
const dataset = {
|
||||
id: vi.fn(() => "dataset-1"),
|
||||
name: vi.fn(() => datasetName),
|
||||
version: vi.fn(() => 3),
|
||||
records: vi.fn(() => datasetRecords),
|
||||
url: vi.fn(() => "https://dd.test/dataset"),
|
||||
push: vi.fn(async () => ({
|
||||
pushedCount: datasetRecords.length,
|
||||
totalCount: datasetRecords.length,
|
||||
})),
|
||||
};
|
||||
const client = {
|
||||
createDataset: vi.fn(
|
||||
(
|
||||
name: string,
|
||||
options?: {
|
||||
records?: Array<{
|
||||
inputData: unknown;
|
||||
expectedOutput?: unknown;
|
||||
metadata?: Readonly<Record<string, unknown>>;
|
||||
}>;
|
||||
},
|
||||
) => {
|
||||
datasetName = name;
|
||||
datasetRecords = (options?.records ?? []).map((record, index) => ({
|
||||
id: `record-${index + 1}`,
|
||||
...record,
|
||||
}));
|
||||
return dataset;
|
||||
},
|
||||
),
|
||||
startExperiment: vi.fn(async (_options: unknown) => experiment),
|
||||
};
|
||||
const lines: string[] = [];
|
||||
const config = {
|
||||
projectName: "test-project",
|
||||
client,
|
||||
log: (line: string) => lines.push(line),
|
||||
...overrides,
|
||||
} satisfies DatadogReporterConfig;
|
||||
|
||||
return { client, config, dataset, experiment, lines, span };
|
||||
}
|
||||
|
||||
describe("Datadog", () => {
|
||||
it("creates an experiment, submits one span, and attaches assertion metrics", async () => {
|
||||
const { client, config, experiment, span } = makeConfig({ experimentName: "run-1" });
|
||||
const reporter = Datadog(config);
|
||||
const evaluation = makeEval();
|
||||
const result = makeEvalResult();
|
||||
|
||||
await reporter.onRunStart([evaluation], makeTarget());
|
||||
await reporter.onEvalComplete(result);
|
||||
|
||||
expect(client.createDataset).not.toHaveBeenCalled();
|
||||
expect(client.startExperiment).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
name: "run-1",
|
||||
projectName: "test-project",
|
||||
dataset: { name: "run-1 dataset" },
|
||||
tags: expect.objectContaining({ source: "eve", target_kind: "local" }),
|
||||
metadata: expect.objectContaining({
|
||||
eveEvalIds: ["eval-1"],
|
||||
eveTargetKind: "local",
|
||||
eveTargetOrigin: "http://127.0.0.1:3000",
|
||||
}),
|
||||
}),
|
||||
);
|
||||
expect(experiment.submitSpan).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
name: "eval-1",
|
||||
durationMs: 1000,
|
||||
metadata: expect.objectContaining({
|
||||
suite: "unit",
|
||||
eveSessionId: "session-123",
|
||||
eveVerdict: "passed",
|
||||
eveToolCalls: ["search"],
|
||||
}),
|
||||
tags: expect.objectContaining({ eval_id: "eval-1", eval_verdict: "passed" }),
|
||||
}),
|
||||
);
|
||||
const submittedSpan = experiment.submitSpan.mock.calls[0]?.[0];
|
||||
expect(submittedSpan).not.toHaveProperty("id");
|
||||
expect(submittedSpan).not.toHaveProperty("input");
|
||||
expect(submittedSpan).not.toHaveProperty("output");
|
||||
expect(submittedSpan).not.toHaveProperty("expectedOutput");
|
||||
expect(submittedSpan).not.toHaveProperty("metadata.expectedOutput");
|
||||
expect(submittedSpan).not.toHaveProperty("metadata.expected");
|
||||
expect(submittedSpan).not.toHaveProperty("metadata.expected_output");
|
||||
expect(experiment.submitEvaluationMetrics).toHaveBeenCalledWith(span, [
|
||||
expect.objectContaining({ label: "gate_succeeded", value: 1 }),
|
||||
expect.objectContaining({ label: "similarity", value: 0.9 }),
|
||||
expect.objectContaining({ label: "judge_autoevals_closedQA", value: 1 }),
|
||||
expect.objectContaining({ label: "eve_tool_call_count", value: 1 }),
|
||||
expect.objectContaining({ label: "eve_subagent_call_count", value: 0 }),
|
||||
expect.objectContaining({ label: "eve_message_count", value: 1 }),
|
||||
expect.objectContaining({ label: "eve_reasoning_block_count", value: 0 }),
|
||||
]);
|
||||
expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).not.toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
tags: expect.objectContaining({ assertion_name: expect.anything() }),
|
||||
}),
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
it("redacts target URL secrets and execution errors by default", async () => {
|
||||
const { client, config, experiment } = makeConfig();
|
||||
const reporter = Datadog(config);
|
||||
const target: EveEvalTarget = {
|
||||
...makeTarget("remote"),
|
||||
url: "https://user:password@test.vercel.app/agent?token=private#fragment",
|
||||
};
|
||||
|
||||
await reporter.onRunStart([makeEval()], target);
|
||||
await reporter.onEvalComplete(makeEvalResult({ error: "private application error" }));
|
||||
|
||||
expect(client.startExperiment).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
metadata: expect.objectContaining({ eveTargetOrigin: "https://test.vercel.app" }),
|
||||
}),
|
||||
);
|
||||
expect(client.startExperiment.mock.calls[0]?.[0]).not.toHaveProperty("metadata.eveTargetUrl");
|
||||
expect(experiment.submitSpan.mock.calls[0]?.[0]).not.toHaveProperty("error");
|
||||
});
|
||||
|
||||
it("records execution errors only when enabled", async () => {
|
||||
const { config, experiment } = makeConfig({ recordErrors: true });
|
||||
const reporter = Datadog(config);
|
||||
|
||||
await reporter.onRunStart([makeEval()], makeTarget());
|
||||
await reporter.onEvalComplete(makeEvalResult({ error: "application error" }));
|
||||
|
||||
expect(experiment.submitSpan).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ error: "application error" }),
|
||||
);
|
||||
});
|
||||
|
||||
it("redacts failing assertion details by default", async () => {
|
||||
const { config, experiment } = makeConfig();
|
||||
const reporter = Datadog(config);
|
||||
const result = makeEvalResult({
|
||||
assertions: [
|
||||
{
|
||||
name: "messageIncludes(private expectation)",
|
||||
message: "got private assistant output",
|
||||
score: 0,
|
||||
severity: "gate",
|
||||
passed: false,
|
||||
},
|
||||
],
|
||||
verdict: "failed",
|
||||
});
|
||||
|
||||
await reporter.onRunStart([makeEval()], makeTarget());
|
||||
await reporter.onEvalComplete(result);
|
||||
|
||||
expect(experiment.submitSpan.mock.calls[0]?.[0]).not.toHaveProperty(
|
||||
"metadata.eveFailedAssertions",
|
||||
);
|
||||
expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
label: "gate_messageIncludes_private_expectation",
|
||||
tags: {
|
||||
assertion_index: "1",
|
||||
assertion_severity: "gate",
|
||||
assertion_passed: "false",
|
||||
},
|
||||
}),
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
it("records assertion name tags and failure messages only when enabled", async () => {
|
||||
const { config, experiment } = makeConfig({ recordAssertionDetails: true });
|
||||
const reporter = Datadog(config);
|
||||
const result = makeEvalResult({
|
||||
assertions: [
|
||||
{
|
||||
name: "messageIncludes(private expectation)",
|
||||
message: "got private assistant output",
|
||||
score: 0,
|
||||
severity: "gate",
|
||||
passed: false,
|
||||
},
|
||||
],
|
||||
verdict: "failed",
|
||||
});
|
||||
|
||||
await reporter.onRunStart([makeEval()], makeTarget());
|
||||
await reporter.onEvalComplete(result);
|
||||
|
||||
expect(experiment.submitSpan.mock.calls[0]?.[0]).toHaveProperty(
|
||||
"metadata.eveFailedAssertions",
|
||||
[
|
||||
{
|
||||
name: "messageIncludes(private expectation)",
|
||||
message: "got private assistant output",
|
||||
},
|
||||
],
|
||||
);
|
||||
expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
label: "gate_messageIncludes_private_expectation",
|
||||
tags: expect.objectContaining({
|
||||
assertion_name: "messageIncludes(private expectation)",
|
||||
}),
|
||||
}),
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
it("deduplicates assertion metric labels and reserves built-in labels", async () => {
|
||||
const { config, experiment } = makeConfig({ recordAssertionDetails: true });
|
||||
const reporter = Datadog(config);
|
||||
const result = makeEvalResult({
|
||||
assertions: [
|
||||
{ name: "same label", score: 1, severity: "soft", passed: true },
|
||||
{ name: "same@label", score: 1, severity: "soft", passed: true },
|
||||
{ name: "same label", score: 1, severity: "soft", passed: true },
|
||||
{ name: "eve_tool_call_count", score: 1, severity: "soft", passed: true },
|
||||
],
|
||||
});
|
||||
|
||||
await reporter.onRunStart([makeEval()], makeTarget());
|
||||
await reporter.onEvalComplete(result);
|
||||
|
||||
expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual([
|
||||
expect.objectContaining({ label: "same_label" }),
|
||||
expect.objectContaining({ label: "same_label_2" }),
|
||||
expect.objectContaining({ label: "same_label_3" }),
|
||||
expect.objectContaining({ label: "eve_tool_call_count_2" }),
|
||||
expect.objectContaining({ label: "eve_tool_call_count" }),
|
||||
expect.objectContaining({ label: "eve_subagent_call_count" }),
|
||||
expect.objectContaining({ label: "eve_message_count" }),
|
||||
expect.objectContaining({ label: "eve_reasoning_block_count" }),
|
||||
]);
|
||||
});
|
||||
|
||||
it("creates dataset records from opted-in eval inputs and links experiment rows", async () => {
|
||||
const { client, config, dataset, experiment, lines } = makeConfig({
|
||||
datasetName: "greeting inputs",
|
||||
experimentName: "run-1",
|
||||
recordInputs: true,
|
||||
recordOutputs: true,
|
||||
recordExpectedOutputs: true,
|
||||
});
|
||||
const reporter = Datadog(config);
|
||||
const result = makeEvalResult();
|
||||
|
||||
await reporter.onRunStart([makeEval()], makeTarget());
|
||||
await reporter.onEvalComplete(result);
|
||||
|
||||
expect(client.startExperiment).not.toHaveBeenCalled();
|
||||
expect(experiment.submitSpan).not.toHaveBeenCalled();
|
||||
|
||||
await reporter.onRunComplete(makeSummary(result));
|
||||
|
||||
expect(client.createDataset).toHaveBeenCalledWith(
|
||||
"greeting inputs",
|
||||
expect.objectContaining({
|
||||
projectName: "test-project",
|
||||
records: [
|
||||
{
|
||||
inputData: "What should I send you?",
|
||||
expectedOutput: "helpful onboarding answer",
|
||||
metadata: { eveEvalId: "eval-1" },
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(dataset.push).toHaveBeenCalledOnce();
|
||||
expect(dataset.push.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
client.startExperiment.mock.invocationCallOrder[0] ?? 0,
|
||||
);
|
||||
expect(client.startExperiment).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
name: "run-1",
|
||||
dataset: { id: "dataset-1", name: "greeting inputs", version: 3 },
|
||||
}),
|
||||
);
|
||||
expect(experiment.submitSpan).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
input: "What should I send you?",
|
||||
output: "actual output",
|
||||
expectedOutput: "helpful onboarding answer",
|
||||
datasetRecordId: "record-1",
|
||||
}),
|
||||
);
|
||||
expect(lines.join("\n")).toContain("Datadog dataset URL: https://dd.test/dataset");
|
||||
});
|
||||
|
||||
it("creates an input-only dataset record when no expected output is authored", async () => {
|
||||
const { client, config } = makeConfig({
|
||||
datasetName: "input-only dataset",
|
||||
recordInputs: true,
|
||||
recordExpectedOutputs: true,
|
||||
});
|
||||
const reporter = Datadog(config);
|
||||
const evaluation = makeEval({ metadata: { suite: "unit" } });
|
||||
const result = makeEvalResult();
|
||||
|
||||
await reporter.onRunStart([evaluation], makeTarget());
|
||||
await reporter.onEvalComplete(result);
|
||||
await reporter.onRunComplete(makeSummary(result));
|
||||
|
||||
const record = client.createDataset.mock.calls[0]?.[1]?.records?.[0];
|
||||
expect(record).toEqual({
|
||||
inputData: "What should I send you?",
|
||||
metadata: { eveEvalId: "eval-1" },
|
||||
});
|
||||
});
|
||||
|
||||
it("uses the dd-trace project environment variable by default", async () => {
|
||||
vi.stubEnv("DD_LLMOBS_PROJECT_NAME", "environment-project");
|
||||
try {
|
||||
const { client, config } = makeConfig({ projectName: undefined });
|
||||
const reporter = Datadog(config);
|
||||
|
||||
await reporter.onRunStart([makeEval()], makeTarget());
|
||||
|
||||
expect(client.startExperiment).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ projectName: "environment-project" }),
|
||||
);
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
});
|
||||
|
||||
it("closes the experiment and logs its URL", async () => {
|
||||
const { config, lines, experiment } = makeConfig();
|
||||
const reporter = Datadog(config);
|
||||
|
||||
await reporter.onRunStart([makeEval()], makeTarget());
|
||||
await reporter.onRunComplete(makeSummary());
|
||||
|
||||
expect(experiment.close).toHaveBeenCalledWith({ status: "completed", error: undefined });
|
||||
expect(lines.join("\n")).toContain("Datadog experiment URL: https://dd.test/experiment");
|
||||
});
|
||||
|
||||
it("is a no-op before the experiment is initialized", async () => {
|
||||
const reporter = Datadog(makeConfig().config);
|
||||
|
||||
await reporter.onEvalComplete(makeEvalResult());
|
||||
await reporter.onRunComplete(makeSummary());
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,630 @@
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
import type ddTrace from "dd-trace";
|
||||
|
||||
import type { EveEval, EveEvalResult, EveEvalRunSummary, EveEvalTarget } from "#evals/types.js";
|
||||
import type { EvalReporter } from "#evals/runner/reporters/types.js";
|
||||
import { resolveLocalGitMetadata } from "#evals/runner/resolve-git-metadata.js";
|
||||
import { parseJsonValue, type JsonValue } from "#shared/json.js";
|
||||
|
||||
/** Configuration for the Datadog reporter. */
|
||||
export interface DatadogReporterConfig {
|
||||
/** Datadog LLM Observability project name. Defaults to `DD_LLMOBS_PROJECT_NAME`, the configured ml_app, `DD_SERVICE`, or the first eval id. */
|
||||
readonly projectName?: string;
|
||||
/** Name for the dataset used by the experiment. Defaults to `<experimentName> dataset`. */
|
||||
readonly datasetName?: string;
|
||||
/** Name for the created experiment. Defaults to a timestamped eve eval run name. */
|
||||
readonly experimentName?: string;
|
||||
/** Optional experiment description. */
|
||||
readonly description?: string;
|
||||
/** Datadog service name used when the reporter initializes `dd-trace`. */
|
||||
readonly service?: string;
|
||||
/** Datadog env used when the reporter initializes `dd-trace`. */
|
||||
readonly env?: string;
|
||||
/** Datadog site used when the reporter initializes `dd-trace`. */
|
||||
readonly site?: string;
|
||||
/** LLM Observability ml_app used when the reporter initializes `dd-trace`. Defaults to `projectName`. */
|
||||
readonly mlApp?: string;
|
||||
/** Tags attached to the Datadog Experiment. */
|
||||
readonly tags?: Readonly<Record<string, string>>;
|
||||
/** JSON-serializable metadata attached to the Datadog Experiment. */
|
||||
readonly metadata?: Readonly<Record<string, unknown>>;
|
||||
/** JSON-serializable Datadog Experiment config. */
|
||||
readonly config?: Readonly<Record<string, unknown>>;
|
||||
/** Include the first sent message, falling back to the eval description, in the experiment row and a linked dataset record. Defaults to false. */
|
||||
readonly recordInputs?: boolean;
|
||||
/** Include eval outputs in the synthetic experiment row output. Defaults to false. */
|
||||
readonly recordOutputs?: boolean;
|
||||
/** Include `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` on experiment rows. Defaults to false. */
|
||||
readonly recordExpectedOutputs?: boolean;
|
||||
/** Include raw assertion names as metric tags and failure messages in row metadata. Defaults to false. */
|
||||
readonly recordAssertionDetails?: boolean;
|
||||
/** Include execution error messages, which may contain application data. Defaults to false. */
|
||||
readonly recordErrors?: boolean;
|
||||
/** Console hook used for tests. */
|
||||
readonly log?: (line: string) => void;
|
||||
/** eve-owned client seam for tests and custom Datadog SDK wiring. */
|
||||
readonly client?: DatadogExperimentsClient;
|
||||
}
|
||||
|
||||
type DatadogJsonValue =
|
||||
| string
|
||||
| number
|
||||
| boolean
|
||||
| null
|
||||
| DatadogJsonValue[]
|
||||
| { [key: string]: DatadogJsonValue };
|
||||
|
||||
interface DatadogDatasetRecordInput {
|
||||
inputData: DatadogJsonValue;
|
||||
expectedOutput?: DatadogJsonValue;
|
||||
metadata?: Record<string, DatadogJsonValue>;
|
||||
tags?: string[];
|
||||
}
|
||||
|
||||
interface DatadogDatasetRecord {
|
||||
readonly id: string | null;
|
||||
}
|
||||
|
||||
interface DatadogDataset {
|
||||
id(): string | null;
|
||||
name(): string;
|
||||
version(): number | null;
|
||||
records(): readonly DatadogDatasetRecord[];
|
||||
url(): string | null;
|
||||
push(): Promise<{ pushedCount: number; totalCount: number }>;
|
||||
}
|
||||
|
||||
interface DatadogExperimentsClient {
|
||||
createDataset?(
|
||||
name: string,
|
||||
options?: {
|
||||
projectName?: string;
|
||||
description?: string;
|
||||
records?: DatadogDatasetRecordInput[];
|
||||
},
|
||||
): DatadogDataset;
|
||||
startExperiment(options: DatadogStartExperimentOptions): Promise<DatadogExternalExperiment>;
|
||||
}
|
||||
|
||||
interface DatadogStartExperimentOptions {
|
||||
name: string;
|
||||
projectName?: string;
|
||||
description?: string;
|
||||
tags?: Record<string, string>;
|
||||
metadata?: Record<string, DatadogJsonValue>;
|
||||
config?: Record<string, DatadogJsonValue>;
|
||||
dataset?: {
|
||||
id?: string;
|
||||
version?: number;
|
||||
name?: string;
|
||||
description?: string;
|
||||
};
|
||||
}
|
||||
|
||||
interface DatadogExternalExperiment {
|
||||
experimentId(): string;
|
||||
url(): string | null;
|
||||
submitSpan(row: DatadogExternalExperimentSpanInput): Promise<DatadogExternalExperimentSpan>;
|
||||
submitEvaluationMetrics(
|
||||
span: Pick<DatadogExternalExperimentSpan, "experimentId" | "spanId" | "traceId">,
|
||||
metrics: DatadogEvaluationMetricInput[],
|
||||
): Promise<void>;
|
||||
close(options?: { status?: string; error?: string | Error }): Promise<void>;
|
||||
}
|
||||
|
||||
interface DatadogExternalExperimentSpanInput {
|
||||
name?: string;
|
||||
input?: DatadogJsonValue;
|
||||
output?: DatadogJsonValue;
|
||||
expectedOutput?: DatadogJsonValue;
|
||||
metadata?: Record<string, DatadogJsonValue>;
|
||||
tags?: Record<string, string>;
|
||||
startedAt?: Date | string | number;
|
||||
completedAt?: Date | string | number;
|
||||
durationMs?: number;
|
||||
error?: string | Error | { type?: string; name?: string; message?: string; stack?: string };
|
||||
datasetRecordId?: string;
|
||||
runId?: string;
|
||||
runIteration?: number;
|
||||
}
|
||||
|
||||
interface DatadogExternalExperimentSpan {
|
||||
experimentId: string;
|
||||
spanId: string;
|
||||
traceId: string;
|
||||
url: string | null;
|
||||
}
|
||||
|
||||
interface DatadogEvaluationMetricInput {
|
||||
label: string;
|
||||
value?: DatadogJsonValue;
|
||||
error?: string | Error;
|
||||
timestamp?: Date | string | number;
|
||||
tags?: Record<string, string>;
|
||||
source?: string;
|
||||
}
|
||||
|
||||
type DatadogTraceModule = typeof ddTrace;
|
||||
|
||||
/**
|
||||
* Creates an {@link EvalReporter} that uploads eval assertion scores to a
|
||||
* Datadog LLM Observability Experiment. Requires the optional `dd-trace`
|
||||
* package and `DD_API_KEY`/`DD_APP_KEY` credentials unless `config.client` is
|
||||
* provided.
|
||||
*/
|
||||
export function Datadog(config: DatadogReporterConfig = {}): EvalReporter {
|
||||
return new DatadogReporter(config);
|
||||
}
|
||||
|
||||
class DatadogReporter implements EvalReporter {
|
||||
readonly #config: DatadogReporterConfig;
|
||||
readonly #evaluations = new Map<string, EveEval>();
|
||||
#client: DatadogExperimentsClient | undefined;
|
||||
#experimentOptions: DatadogStartExperimentOptions | undefined;
|
||||
#experiment: DatadogExternalExperiment | undefined;
|
||||
#datasetUrl: string | undefined;
|
||||
|
||||
constructor(config: DatadogReporterConfig) {
|
||||
this.#config = config;
|
||||
}
|
||||
|
||||
async onRunStart(evaluations: readonly EveEval[], target: EveEvalTarget): Promise<void> {
|
||||
this.#client = undefined;
|
||||
this.#experimentOptions = undefined;
|
||||
this.#experiment = undefined;
|
||||
this.#datasetUrl = undefined;
|
||||
this.#evaluations.clear();
|
||||
for (const evaluation of evaluations) {
|
||||
this.#evaluations.set(evaluation.id, evaluation);
|
||||
}
|
||||
|
||||
const client = await resolveDatadogClient(this.#config, evaluations);
|
||||
const git = resolveLocalGitMetadata(process.cwd());
|
||||
|
||||
const experimentName = this.#config.experimentName ?? defaultExperimentName();
|
||||
const metadata = resolveExperimentMetadata(evaluations, target);
|
||||
if (git.sha) {
|
||||
metadata.eveGitCommit = git.sha;
|
||||
metadata.eveGitBranch = git.branch;
|
||||
}
|
||||
Object.assign(metadata, this.#config.metadata);
|
||||
|
||||
const experimentOptions: DatadogStartExperimentOptions = {
|
||||
name: experimentName,
|
||||
projectName: resolveProjectName(this.#config, evaluations),
|
||||
description: this.#config.description,
|
||||
dataset: { name: this.#config.datasetName ?? `${experimentName} dataset` },
|
||||
tags: resolveExperimentTags(this.#config, target),
|
||||
metadata: toDatadogJsonRecord(metadata),
|
||||
config: toDatadogJsonRecord(this.#config.config),
|
||||
};
|
||||
|
||||
if (this.#config.recordInputs) {
|
||||
if (!client.createDataset) {
|
||||
throw new Error(
|
||||
"The installed 'dd-trace' package does not expose tracer.llmobs.experiments.createDataset().",
|
||||
);
|
||||
}
|
||||
this.#client = client;
|
||||
this.#experimentOptions = experimentOptions;
|
||||
return;
|
||||
}
|
||||
|
||||
this.#experiment = await client.startExperiment(experimentOptions);
|
||||
}
|
||||
|
||||
async onEvalComplete(result: EveEvalResult): Promise<void> {
|
||||
if (this.#config.recordInputs || !this.#experiment) return;
|
||||
await this.#submitResult(result);
|
||||
}
|
||||
|
||||
async onRunComplete(summary: EveEvalRunSummary): Promise<void> {
|
||||
try {
|
||||
if (this.#config.recordInputs) {
|
||||
await this.#startDatasetBackedExperiment(summary.results);
|
||||
}
|
||||
if (!this.#experiment) return;
|
||||
|
||||
const failed = summary.failed > 0 || summary.errored > 0;
|
||||
await this.#experiment.close({
|
||||
status: failed ? "failed" : "completed",
|
||||
error: failed ? `${summary.failed} failed, ${summary.errored} errored` : undefined,
|
||||
});
|
||||
|
||||
const log = this.#config.log ?? console.log;
|
||||
if (this.#datasetUrl) {
|
||||
log(`Datadog dataset URL: ${this.#datasetUrl}\n`);
|
||||
}
|
||||
const experimentUrl = this.#experiment.url();
|
||||
if (experimentUrl) {
|
||||
log(`Datadog experiment URL: ${experimentUrl}\n`);
|
||||
}
|
||||
} finally {
|
||||
this.#client = undefined;
|
||||
this.#experimentOptions = undefined;
|
||||
this.#experiment = undefined;
|
||||
this.#datasetUrl = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
async #startDatasetBackedExperiment(results: readonly EveEvalResult[]): Promise<void> {
|
||||
const client = this.#client;
|
||||
const experimentOptions = this.#experimentOptions;
|
||||
if (!client?.createDataset || !experimentOptions) return;
|
||||
|
||||
const datasetName = experimentOptions.dataset?.name ?? `${experimentOptions.name} dataset`;
|
||||
const records = results.map((result): DatadogDatasetRecordInput => {
|
||||
const evaluation = this.#evaluations.get(result.id);
|
||||
const record: DatadogDatasetRecordInput = {
|
||||
inputData: toDatadogJsonValue(resolveInput(result, evaluation)),
|
||||
metadata: { eveEvalId: result.id },
|
||||
};
|
||||
if (this.#config.recordExpectedOutputs) {
|
||||
const expectedOutput = toOptionalDatadogJsonValue(resolveExpectedOutput(evaluation));
|
||||
if (expectedOutput !== undefined) {
|
||||
record.expectedOutput = expectedOutput;
|
||||
}
|
||||
}
|
||||
return record;
|
||||
});
|
||||
const dataset = client.createDataset(datasetName, {
|
||||
projectName: experimentOptions.projectName,
|
||||
description:
|
||||
this.#config.description ?? `Eve eval inputs for experiment '${experimentOptions.name}'.`,
|
||||
records,
|
||||
});
|
||||
|
||||
await dataset.push();
|
||||
const datasetId = dataset.id();
|
||||
if (!datasetId) {
|
||||
throw new Error(`Datadog dataset '${datasetName}' has no id after push().`);
|
||||
}
|
||||
const datasetRecords = dataset.records();
|
||||
if (datasetRecords.length !== results.length) {
|
||||
throw new Error(
|
||||
`Datadog dataset '${datasetName}' has ${datasetRecords.length} records for ${results.length} eval results.`,
|
||||
);
|
||||
}
|
||||
|
||||
const datasetOptions: NonNullable<DatadogStartExperimentOptions["dataset"]> = {
|
||||
id: datasetId,
|
||||
name: dataset.name(),
|
||||
};
|
||||
const datasetVersion = dataset.version();
|
||||
if (datasetVersion !== null) {
|
||||
datasetOptions.version = datasetVersion;
|
||||
}
|
||||
this.#datasetUrl = dataset.url() ?? undefined;
|
||||
this.#experiment = await client.startExperiment({
|
||||
...experimentOptions,
|
||||
dataset: datasetOptions,
|
||||
});
|
||||
|
||||
for (const [index, result] of results.entries()) {
|
||||
const datasetRecordId = datasetRecords[index]?.id;
|
||||
if (!datasetRecordId) {
|
||||
throw new Error(`Datadog dataset record ${index + 1} has no id after push().`);
|
||||
}
|
||||
await this.#submitResult(result, datasetRecordId);
|
||||
}
|
||||
}
|
||||
|
||||
async #submitResult(result: EveEvalResult, datasetRecordId?: string): Promise<void> {
|
||||
if (!this.#experiment) return;
|
||||
|
||||
const evaluation = this.#evaluations.get(result.id);
|
||||
const spanInput: DatadogExternalExperimentSpanInput = {
|
||||
name: result.id,
|
||||
metadata: toDatadogJsonRecord(
|
||||
resolveResultMetadata(result, evaluation, this.#config.recordAssertionDetails === true),
|
||||
),
|
||||
tags: resolveResultTags(result, evaluation),
|
||||
startedAt: result.startedAt,
|
||||
completedAt: result.completedAt,
|
||||
durationMs: elapsedMs(result.startedAt, result.completedAt),
|
||||
};
|
||||
if (this.#config.recordInputs) {
|
||||
spanInput.input = toOptionalDatadogJsonValue(resolveInput(result, evaluation));
|
||||
}
|
||||
if (this.#config.recordOutputs) {
|
||||
spanInput.output = toOptionalDatadogJsonValue(result.result.output);
|
||||
}
|
||||
if (this.#config.recordExpectedOutputs) {
|
||||
spanInput.expectedOutput = toOptionalDatadogJsonValue(resolveExpectedOutput(evaluation));
|
||||
}
|
||||
if (this.#config.recordErrors && result.error !== undefined) {
|
||||
spanInput.error = result.error;
|
||||
}
|
||||
if (datasetRecordId !== undefined) {
|
||||
spanInput.datasetRecordId = datasetRecordId;
|
||||
}
|
||||
|
||||
const span = await this.#experiment.submitSpan(spanInput);
|
||||
await this.#experiment.submitEvaluationMetrics(
|
||||
span,
|
||||
resolveEvaluationMetrics(result, this.#config.recordAssertionDetails === true),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const DD_TRACE_PACKAGE = "dd-trace";
|
||||
const EXPECTED_OUTPUT_METADATA_KEYS: ReadonlySet<string> = new Set([
|
||||
"expectedOutput",
|
||||
"expected",
|
||||
"expected_output",
|
||||
]);
|
||||
const BUILT_IN_METRIC_LABELS = [
|
||||
"eve_tool_call_count",
|
||||
"eve_subagent_call_count",
|
||||
"eve_message_count",
|
||||
"eve_reasoning_block_count",
|
||||
] as const;
|
||||
|
||||
async function resolveDatadogClient(
|
||||
config: DatadogReporterConfig,
|
||||
evaluations: readonly EveEval[],
|
||||
): Promise<DatadogExperimentsClient> {
|
||||
if (config.client) return config.client;
|
||||
|
||||
const sdk = await loadDatadogSdk();
|
||||
const projectName = resolveProjectName(config, evaluations);
|
||||
const tracer = sdk.init({
|
||||
service: config.service ?? process.env.DD_SERVICE ?? projectName,
|
||||
env: config.env ?? process.env.DD_ENV,
|
||||
site: config.site ?? process.env.DD_SITE,
|
||||
llmobs: {
|
||||
projectName,
|
||||
mlApp: config.mlApp ?? projectName,
|
||||
agentlessEnabled: true,
|
||||
},
|
||||
});
|
||||
|
||||
const experiments = tracer.llmobs?.experiments;
|
||||
if (!experiments?.startExperiment) {
|
||||
throw new Error(
|
||||
[
|
||||
"The installed 'dd-trace' package does not expose tracer.llmobs.experiments.startExperiment().",
|
||||
"Update to a release compatible with dd-trace@6.13.0.",
|
||||
].join("\n"),
|
||||
);
|
||||
}
|
||||
return experiments;
|
||||
}
|
||||
|
||||
async function loadDatadogSdk(): Promise<DatadogTraceModule> {
|
||||
try {
|
||||
const requireFromApp = createRequire(`${process.cwd()}/package.json`);
|
||||
return requireFromApp(DD_TRACE_PACKAGE) as DatadogTraceModule;
|
||||
} catch {
|
||||
try {
|
||||
const mod = (await import(DD_TRACE_PACKAGE)) as { default?: unknown };
|
||||
return (mod.default ?? mod) as DatadogTraceModule;
|
||||
} catch {
|
||||
throw new Error(
|
||||
[
|
||||
"The 'dd-trace' package is required for Datadog reporting but was not found.",
|
||||
"",
|
||||
"Install the tested release with:",
|
||||
" npm install dd-trace@6.13.0",
|
||||
].join("\n"),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function resolveProjectName(
|
||||
config: DatadogReporterConfig,
|
||||
evaluations: readonly EveEval[],
|
||||
): string {
|
||||
return (
|
||||
config.projectName ??
|
||||
process.env.DD_LLMOBS_PROJECT_NAME ??
|
||||
config.mlApp ??
|
||||
process.env.DD_LLMOBS_ML_APP ??
|
||||
process.env.DD_SERVICE ??
|
||||
evaluations[0]?.id ??
|
||||
"eve evals"
|
||||
);
|
||||
}
|
||||
|
||||
function defaultExperimentName(): string {
|
||||
return `eve evals ${new Date().toISOString()}`;
|
||||
}
|
||||
|
||||
function resolveExperimentTags(
|
||||
config: DatadogReporterConfig,
|
||||
target: EveEvalTarget,
|
||||
): Record<string, string> {
|
||||
return {
|
||||
source: "eve",
|
||||
target_kind: target.kind,
|
||||
...config.tags,
|
||||
};
|
||||
}
|
||||
|
||||
function resolveExperimentMetadata(
|
||||
evaluations: readonly EveEval[],
|
||||
target: EveEvalTarget,
|
||||
): Record<string, unknown> {
|
||||
const metadata: Record<string, unknown> = {
|
||||
eveEvalIds: evaluations.map((evaluation) => evaluation.id),
|
||||
eveTargetKind: target.kind,
|
||||
eveTimestamp: new Date().toISOString(),
|
||||
};
|
||||
const targetOrigin = resolveTargetOrigin(target.url);
|
||||
if (targetOrigin !== undefined) {
|
||||
metadata.eveTargetOrigin = targetOrigin;
|
||||
}
|
||||
return metadata;
|
||||
}
|
||||
|
||||
function resolveTargetOrigin(value: string): string | undefined {
|
||||
try {
|
||||
const origin = new URL(value).origin;
|
||||
return origin === "null" ? undefined : origin;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function resolveResultTags(
|
||||
result: EveEvalResult,
|
||||
evaluation: EveEval | undefined,
|
||||
): Record<string, string> {
|
||||
const tags: Record<string, string> = {
|
||||
eval_id: result.id,
|
||||
eval_verdict: result.verdict,
|
||||
eval_status: result.result.status,
|
||||
};
|
||||
if (evaluation?.tags?.length) {
|
||||
tags.eval_tags = evaluation.tags.join(",");
|
||||
}
|
||||
return tags;
|
||||
}
|
||||
|
||||
function resolveInput(result: EveEvalResult, evaluation: EveEval | undefined): unknown {
|
||||
for (const event of result.result.events) {
|
||||
if (event.type !== "message.received") continue;
|
||||
return event.data.message;
|
||||
}
|
||||
|
||||
return evaluation?.description ?? "";
|
||||
}
|
||||
|
||||
function resolveExpectedOutput(evaluation: EveEval | undefined): unknown {
|
||||
if (!evaluation?.metadata) return undefined;
|
||||
return (
|
||||
evaluation.metadata.expectedOutput ??
|
||||
evaluation.metadata.expected ??
|
||||
evaluation.metadata.expected_output
|
||||
);
|
||||
}
|
||||
|
||||
function resolveResultMetadata(
|
||||
result: EveEvalResult,
|
||||
evaluation: EveEval | undefined,
|
||||
recordAssertionDetails: boolean,
|
||||
): Record<string, unknown> {
|
||||
const metadata: Record<string, unknown> = {};
|
||||
for (const [key, value] of Object.entries(evaluation?.metadata ?? {})) {
|
||||
if (!EXPECTED_OUTPUT_METADATA_KEYS.has(key)) {
|
||||
metadata[key] = value;
|
||||
}
|
||||
}
|
||||
Object.assign(metadata, {
|
||||
eveSessionId: result.result.sessionId,
|
||||
eveStatus: result.result.status,
|
||||
eveVerdict: result.verdict,
|
||||
eveSkipReason: result.skipReason,
|
||||
eveToolCalls: result.result.derived.toolCalls.map((call) => call.name),
|
||||
eveSubagentCalls: result.result.derived.subagentCalls.map((call) => call.name),
|
||||
eveParked: result.result.derived.parked,
|
||||
});
|
||||
if (recordAssertionDetails) {
|
||||
const failedAssertions = result.assertions
|
||||
.filter((assertion) => !assertion.passed)
|
||||
.map((assertion) => ({ name: assertion.name, message: assertion.message }));
|
||||
if (failedAssertions.length > 0) {
|
||||
metadata.eveFailedAssertions = failedAssertions;
|
||||
}
|
||||
}
|
||||
if (result.result.derived.failureCode) {
|
||||
metadata.eveFailureCode = result.result.derived.failureCode;
|
||||
}
|
||||
return metadata;
|
||||
}
|
||||
|
||||
function resolveEvaluationMetrics(
|
||||
result: EveEvalResult,
|
||||
recordAssertionDetails: boolean,
|
||||
): DatadogEvaluationMetricInput[] {
|
||||
const metrics: DatadogEvaluationMetricInput[] = [];
|
||||
const usedLabels = new Set<string>(BUILT_IN_METRIC_LABELS);
|
||||
|
||||
for (const [index, assertion] of result.assertions.entries()) {
|
||||
const rawLabel = assertion.severity === "gate" ? `gate_${assertion.name}` : assertion.name;
|
||||
const tags: Record<string, string> = {
|
||||
assertion_index: String(index + 1),
|
||||
assertion_severity: assertion.severity,
|
||||
assertion_passed: String(assertion.passed),
|
||||
};
|
||||
if (recordAssertionDetails) {
|
||||
tags.assertion_name = assertion.name;
|
||||
tags.assertion_label = rawLabel;
|
||||
}
|
||||
metrics.push({
|
||||
label: reserveDatadogMetricLabel(toDatadogMetricLabel(rawLabel), usedLabels),
|
||||
value: assertion.score,
|
||||
tags,
|
||||
});
|
||||
}
|
||||
|
||||
metrics.push(
|
||||
{ label: "eve_tool_call_count", value: result.result.derived.toolCallCount },
|
||||
{ label: "eve_subagent_call_count", value: result.result.derived.subagentCallCount },
|
||||
{ label: "eve_message_count", value: result.result.derived.messageCount },
|
||||
{ label: "eve_reasoning_block_count", value: result.result.derived.reasoningBlockCount },
|
||||
);
|
||||
|
||||
return metrics;
|
||||
}
|
||||
|
||||
function toDatadogMetricLabel(label: string): string {
|
||||
const normalized = label.replace(/[^a-zA-Z0-9_-]+/g, "_").replace(/^_+|_+$/g, "");
|
||||
return normalized || "metric";
|
||||
}
|
||||
|
||||
function reserveDatadogMetricLabel(baseLabel: string, usedLabels: Set<string>): string {
|
||||
let label = baseLabel;
|
||||
let suffix = 2;
|
||||
while (usedLabels.has(label)) {
|
||||
label = `${baseLabel}_${suffix}`;
|
||||
suffix += 1;
|
||||
}
|
||||
usedLabels.add(label);
|
||||
return label;
|
||||
}
|
||||
|
||||
function elapsedMs(startedAt: string, completedAt: string): number | undefined {
|
||||
const start = Date.parse(startedAt);
|
||||
const completed = Date.parse(completedAt);
|
||||
if (!Number.isFinite(start) || !Number.isFinite(completed)) return undefined;
|
||||
return Math.max(0, completed - start);
|
||||
}
|
||||
|
||||
function toOptionalDatadogJsonValue(value: unknown): DatadogJsonValue | undefined {
|
||||
return value === undefined ? undefined : toDatadogJsonValue(value);
|
||||
}
|
||||
|
||||
function toDatadogJsonRecord(
|
||||
value: Readonly<Record<string, unknown>> | undefined,
|
||||
): Record<string, DatadogJsonValue> | undefined {
|
||||
if (value === undefined) return undefined;
|
||||
|
||||
const output: Record<string, DatadogJsonValue> = {};
|
||||
for (const [key, entry] of Object.entries(value)) {
|
||||
if (entry !== undefined) {
|
||||
output[key] = toDatadogJsonValue(entry);
|
||||
}
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
function toDatadogJsonValue(value: unknown): DatadogJsonValue {
|
||||
return cloneDatadogJsonValue(parseJsonValue(value));
|
||||
}
|
||||
|
||||
function cloneDatadogJsonValue(value: JsonValue): DatadogJsonValue {
|
||||
if (Array.isArray(value)) {
|
||||
return value.map((entry) => cloneDatadogJsonValue(entry));
|
||||
}
|
||||
if (value !== null && typeof value === "object") {
|
||||
const output: Record<string, DatadogJsonValue> = {};
|
||||
for (const [key, entry] of Object.entries(value)) {
|
||||
output[key] = cloneDatadogJsonValue(entry);
|
||||
}
|
||||
return output;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
Generated
+372
-5
@@ -402,6 +402,21 @@ importers:
|
||||
specifier: 'catalog:'
|
||||
version: 7.0.2
|
||||
|
||||
apps/fixtures/datadog-eval-reporter:
|
||||
dependencies:
|
||||
'@eve-e2e/config':
|
||||
specifier: workspace:*
|
||||
version: link:../../../e2e/fixtures/e2e-config
|
||||
'@workflow/world-postgres':
|
||||
specifier: 'catalog:'
|
||||
version: 5.0.0-beta.40(@opentelemetry/api@1.9.1)(@types/pg@8.20.0)(@upstash/redis@1.38.4)(sql.js@1.14.1)(supports-color@10.2.2)(typescript@7.0.2)
|
||||
dd-trace:
|
||||
specifier: 6.13.0
|
||||
version: 6.13.0
|
||||
eve:
|
||||
specifier: workspace:*
|
||||
version: link:../../../packages/eve
|
||||
|
||||
apps/fixtures/weather-agent:
|
||||
dependencies:
|
||||
eve:
|
||||
@@ -1433,6 +1448,9 @@ importers:
|
||||
commander:
|
||||
specifier: 14.0.3
|
||||
version: 14.0.3
|
||||
dd-trace:
|
||||
specifier: 6.13.0
|
||||
version: 6.13.0
|
||||
emulate:
|
||||
specifier: 0.6.0
|
||||
version: 0.6.0
|
||||
@@ -2196,6 +2214,28 @@ packages:
|
||||
resolution: {integrity: sha512-ooWCrlZP11i8GImSjTHYHLkvFDP48nS4+204nGb1RiX/WXYHmJA2III9/e2DWVabCESdW7hBAEzHRqUn9OUVvQ==}
|
||||
engines: {node: '>=0.1.90'}
|
||||
|
||||
'@datadog/libdatadog@0.12.1':
|
||||
resolution: {integrity: sha512-4cKRaO1mB9npfklJjOizzJaNBdZvw1V62EVbSD6Y32zX92bTBq/vAno/TTN9dMAvzomXYmvADpGo4798E9fMoA==}
|
||||
|
||||
'@datadog/native-appsec@11.0.2':
|
||||
resolution: {integrity: sha512-Azs5fhwJx/BXHMnz+4PM2d9Is7Ik8uQPSM90nzvTCOcBYhSxuLhCY0EsDik/15CRknV9YoQrP1vmRrCVq3il5w==}
|
||||
engines: {node: '>=18'}
|
||||
|
||||
'@datadog/native-iast-taint-tracking@4.2.1':
|
||||
resolution: {integrity: sha512-KRLu3aTXvyAusouAZGTw2w1Xo67cRTw93yG4C9vTMpIgN9FXiqhlcvtNHvUzaA/KRrwtNwbaUELubOtk/kxOFg==}
|
||||
|
||||
'@datadog/native-metrics@4.0.0':
|
||||
resolution: {integrity: sha512-rS6Qc8WAbOgbrtDYdqK15gi2xbuIEyfuquUH05g+2DOdiNAswNAGrlIuSdJZQF/qoJzjcFvxb9kCHu8BjiPfQQ==}
|
||||
engines: {node: '>=18'}
|
||||
|
||||
'@datadog/pprof@5.18.1':
|
||||
resolution: {integrity: sha512-p8RnantBXCrcsT4pHvcdRtQzhqllls/t6yYb2j8hJrnnFcFfWO+u5WSdqLNxGFNjTEhfaFPu6TOoLhNk9iW9Lw==}
|
||||
engines: {node: '>=16'}
|
||||
|
||||
'@datadog/wasm-js-rewriter@5.0.4':
|
||||
resolution: {integrity: sha512-tSjbk51dkNzFMM3P/7y82tyrBj/9CWnPzcVb1JB8TUea/PIholdXSPuGC5F975YvvGB8KDujfmJIacN+EmuXMg==}
|
||||
engines: {node: '>= 10'}
|
||||
|
||||
'@dimforge/rapier3d-compat@0.12.0':
|
||||
resolution: {integrity: sha512-uekIGetywIgopfD97oDL5PfeezkFpNhwlzlaEYNOA0N6ghdsOvh/HYjSMek5Q2O1PYvRSDFcqFVJl4r4ZBwOow==}
|
||||
|
||||
@@ -4184,42 +4224,84 @@ packages:
|
||||
cpu: [arm]
|
||||
os: [android]
|
||||
|
||||
'@oxc-parser/binding-android-arm-eabi@0.132.0':
|
||||
resolution: {integrity: sha512-KrLaPWa5c9Y7LkW+rKkaUE3y7DBDrQtaf7rlsSDfv6KAHUjgzAIRA761Lrrp6//Yd/Rlie/yEOt9YENCoJnOcw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm]
|
||||
os: [android]
|
||||
|
||||
'@oxc-parser/binding-android-arm64@0.131.0':
|
||||
resolution: {integrity: sha512-nlGIod6gw75x1aEDgLS+srj+JRGY0HHm9MI9YgzE/B64l6d6+H3MSP9NOgp0+HTg8tp4vV9rVfgQGgd+TfVZcA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [android]
|
||||
|
||||
'@oxc-parser/binding-android-arm64@0.132.0':
|
||||
resolution: {integrity: sha512-SThDrSeamB/kG2+NxcJ5/wSLcV6dUqDknrPLqFYQ0ST/55mtBP4M7Q/f3QbubH6aAd11wpzZn/nwbVRSdobOpg==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [android]
|
||||
|
||||
'@oxc-parser/binding-darwin-arm64@0.131.0':
|
||||
resolution: {integrity: sha512-jukuV6xe5RbQKFo7QD34NDCLDZp4PSOm8rmckhNdH/60ymG5zXbDzGBEyc+nTkuLQNama2aSGCt+CPfpjNTqyw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [darwin]
|
||||
|
||||
'@oxc-parser/binding-darwin-arm64@0.132.0':
|
||||
resolution: {integrity: sha512-Lc0f/TYoKBghE5/2Gsv7bLXk+TJZunx2Tf61X8hG4ARXdc8UYI26dCGccFSd1AyFbK3jfaNXtMnupggDbjPXdQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [darwin]
|
||||
|
||||
'@oxc-parser/binding-darwin-x64@0.131.0':
|
||||
resolution: {integrity: sha512-g3JOo4khe9rslHm5WYaVDWb0HS/M1MLR3I9S8560MkKIcC96VQY00QjOlsuRyfSj/JDXj8i9T7ryPO2RidiXVg==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [x64]
|
||||
os: [darwin]
|
||||
|
||||
'@oxc-parser/binding-darwin-x64@0.132.0':
|
||||
resolution: {integrity: sha512-RG2eJIpf7C21z9HSSXFw1bTArdpKe7Y4fwcJTwRq1yCSe1vSavaN9GA1sm9KqzemTLAGVktQ+7qBTGp0vQeUZg==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [x64]
|
||||
os: [darwin]
|
||||
|
||||
'@oxc-parser/binding-freebsd-x64@0.131.0':
|
||||
resolution: {integrity: sha512-1hziITDTxjMePnX+dR9ocVT+EuZkQ8wm4FPAbmbEiKG+Phbo73J1ZnPAA6Y/aGsWF3McOFnQuZIktAFwalkfJQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [x64]
|
||||
os: [freebsd]
|
||||
|
||||
'@oxc-parser/binding-freebsd-x64@0.132.0':
|
||||
resolution: {integrity: sha512-wQIPntPLtJ8NcBpvKPbEv3NqzV6k8eP8tP/jE9Rg8HTg/j7urZGFSsTCPCW5k77Qfw2DM4vRvc9p3I4yq/Shvw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [x64]
|
||||
os: [freebsd]
|
||||
|
||||
'@oxc-parser/binding-linux-arm-gnueabihf@0.131.0':
|
||||
resolution: {integrity: sha512-9uRxfXwyKG9+MwmGQBo2ncPNwZH5HTmCETFM2WiuDBNDCW4NC5ttSQkwCAMrTAWgwMzVBH1CP8pM0v7nebCWXQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm]
|
||||
os: [linux]
|
||||
|
||||
'@oxc-parser/binding-linux-arm-gnueabihf@0.132.0':
|
||||
resolution: {integrity: sha512-PixKEpeSe3yxQWqNyOCBALRYc72+Tj7ILDofUl3iXo25cVOzLA6jHUhmOINRtWIPh7dbUie3QNeabwaQpZTw6w==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm]
|
||||
os: [linux]
|
||||
|
||||
'@oxc-parser/binding-linux-arm-musleabihf@0.131.0':
|
||||
resolution: {integrity: sha512-mgbLvzRShXOLBdWGInf08Af4q+pfj1xD8hSgLClDZ9of/BXkB6+LIhTH7fihiDUipqB3yoSkKBWaZ3Ejlf5Yag==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm]
|
||||
os: [linux]
|
||||
|
||||
'@oxc-parser/binding-linux-arm-musleabihf@0.132.0':
|
||||
resolution: {integrity: sha512-sCR+DzGHlyHKnbA2z9zWjTUhIo8Sy0enJl4RDsBwPmkxYynPatpwOAWe8W5127SlW0boqUWHGtr1NWn5UwIhXQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm]
|
||||
os: [linux]
|
||||
|
||||
'@oxc-parser/binding-linux-arm64-gnu@0.131.0':
|
||||
resolution: {integrity: sha512-OPT8++4aN6j2GJ8+3IZHS/byXoZP4aSBn+FoG6rgBJ2fKwPKXWF3MqrFMNW7NKHM28FLY579xYLxJSfgobEqPA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -4227,6 +4309,13 @@ packages:
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-arm64-gnu@0.132.0':
|
||||
resolution: {integrity: sha512-sQBix5P2cW+IpzTcCwYxnh9yALrKSIkKJThspBvMGcygSMnbzkSvhN7SfuX1hvBk8y1XEChsdkU3ET0V5DmzUw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-arm64-musl@0.131.0':
|
||||
resolution: {integrity: sha512-vtPiwmfVTAXzaxDKsOXG+LwgRAA7WEnaeHzhS5z0GE89gAK18KSXnly7Z6saXXq6L3dVMyK44uoTI03zKxrpmw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -4234,6 +4323,13 @@ packages:
|
||||
os: [linux]
|
||||
libc: [musl]
|
||||
|
||||
'@oxc-parser/binding-linux-arm64-musl@0.132.0':
|
||||
resolution: {integrity: sha512-WozHg3Kc//8Sk756HXXgMbEAvqtG+Lzb9JOojwQzIGDtN78Az2dLttkb71akWYUF/8IgYfDSlfKh4Uot8is5Vw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [linux]
|
||||
libc: [musl]
|
||||
|
||||
'@oxc-parser/binding-linux-ppc64-gnu@0.131.0':
|
||||
resolution: {integrity: sha512-8AW8L7w5cGHSdZPcyZX2yR0+GUODsT15rbRjfdD54rv6DMbtuEB19ysLOpKJlRGfH6UNYNpCHaU1uJWgTWf1/w==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -4241,6 +4337,13 @@ packages:
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-ppc64-gnu@0.132.0':
|
||||
resolution: {integrity: sha512-CmX/ulNBOEwWTyVRmcpYKAcAizW6+OjtLJgo7fXoL9OqQvjF4VER8tPomv44vwzfSCy1BHbsB0ZlZYzYJNj4cA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [ppc64]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-riscv64-gnu@0.131.0':
|
||||
resolution: {integrity: sha512-vvpjkjEOUsPcsYf8evE4MO3aGx9+3wodXEBOicGNnOwTuAik8eBONNkgSdhkGsAblQmfVHJyanRnpxglddTXIA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -4248,6 +4351,13 @@ packages:
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-riscv64-gnu@0.132.0':
|
||||
resolution: {integrity: sha512-j9oQS+hM90SdhviNGWbPgT4+Rlq+ac++q/zjgwPD1mVHgxHzATvoRGtDx0sXGmFOQ9J9YkwAhYGb5MAHL6TAsA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [riscv64]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-riscv64-musl@0.131.0':
|
||||
resolution: {integrity: sha512-AqmcNC3fClXX+fxQ6VGEN1667xVFiRBkY0CZmDMSiaeFUsv1+UkBPYYi48IUKcA9/ivvoKNRzQl2I4//kT9F/w==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -4255,6 +4365,13 @@ packages:
|
||||
os: [linux]
|
||||
libc: [musl]
|
||||
|
||||
'@oxc-parser/binding-linux-riscv64-musl@0.132.0':
|
||||
resolution: {integrity: sha512-bLz+Xi+Agnfmd7kWPEsSVwCn2k4EyIalZkNBcQ0OGIv9rqn8VgCPLNd03tM9mKX/5TdlvDXalz0q71BIrOPNqg==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [riscv64]
|
||||
os: [linux]
|
||||
libc: [musl]
|
||||
|
||||
'@oxc-parser/binding-linux-s390x-gnu@0.131.0':
|
||||
resolution: {integrity: sha512-7d3jOMKy7RSQCcDLIci+ySll2FgsOMl/GiRux4q2JNv0zg4EdhFISa9idvrdN/HEUIQQJNg6dmveUeJl2YErGA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -4262,6 +4379,13 @@ packages:
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-s390x-gnu@0.132.0':
|
||||
resolution: {integrity: sha512-U6t2qbJU0ypTfyj9QV3W1Y6mITDTL8ai/OR6NUn85vyHthOvobKWgXzU4tu0EskSzlpuVFz1g0jFGulDIUKHxQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [s390x]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-x64-gnu@0.131.0':
|
||||
resolution: {integrity: sha512-JHK/h95qVqVQ+ITER837kcTdwBDFpFaNnOTYGCP0zdUSX/mLKC7tXOoyrTb6vG7iRPwGlcgBil3v2IjYw1FqJA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -4269,6 +4393,13 @@ packages:
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-x64-gnu@0.132.0':
|
||||
resolution: {integrity: sha512-WcEaSNHFk8yz5YFlQQAlhq6jOFmZBB/RKE7uzhyCIf+pF1Lmv9gUH4221mle2Gd9iHyWT3ySNph8yZgb1xYdWg==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [x64]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@oxc-parser/binding-linux-x64-musl@0.131.0':
|
||||
resolution: {integrity: sha512-b2BO82O8azXAyf7EUgOPKu145nWypbNyk07HbU09fkzhm9lEA5oPvaN/M8Nlo7tOErVTa2WOgS4QbOnxAPXdDQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -4276,41 +4407,80 @@ packages:
|
||||
os: [linux]
|
||||
libc: [musl]
|
||||
|
||||
'@oxc-parser/binding-linux-x64-musl@0.132.0':
|
||||
resolution: {integrity: sha512-iQrV4iJzQgRwK3BWRmQl1C3C6g3wYpXN2WLdQdyR+efoUnncdShZAVp9OgcojtlD3MDRbuOMGG3SjxF4fL4nlQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [x64]
|
||||
os: [linux]
|
||||
libc: [musl]
|
||||
|
||||
'@oxc-parser/binding-openharmony-arm64@0.131.0':
|
||||
resolution: {integrity: sha512-GHO9glZaX7LkX/OGfluEPf1yjg+ehiFbUdowbX6uNWOQhmwKWU4m4+nZ9FJkrHNKuxyI1KKertMdGjVKCApKWA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [openharmony]
|
||||
|
||||
'@oxc-parser/binding-openharmony-arm64@0.132.0':
|
||||
resolution: {integrity: sha512-FWzmUGrZ6GUby4U7WIwcCtab6tdmlTO3xTRRKyb5kjIJVEiaUAT8animUG/nK8ZCA8gkRkPOTId4rl6uTqUmJQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [openharmony]
|
||||
|
||||
'@oxc-parser/binding-wasm32-wasi@0.131.0':
|
||||
resolution: {integrity: sha512-3SkikPaEFoih1N83qLVEDLRLeY4nYsf6JT9SnWiMCQ5lGQdKup6bEuKCqkRiG9dD1IIaFeYz9RjlciPmYoFIWA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [wasm32]
|
||||
|
||||
'@oxc-parser/binding-wasm32-wasi@0.132.0':
|
||||
resolution: {integrity: sha512-TlbMppxJI5CjWDes0QaP6G3aneVg1yikBu5QYI+DUShF9WDL66ccgKFNNGmi/Wybtszw6hxwAvv76T4DaPKnHw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [wasm32]
|
||||
|
||||
'@oxc-parser/binding-win32-arm64-msvc@0.131.0':
|
||||
resolution: {integrity: sha512-Os5bEhryeA2jkH+ZrnZyAC1EP5gs+X4YB1Fjqml7UPD5kU7ecsK1MPEVMfCrdt/GDNpDbavYXiOXOdyJ5b3OPw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [win32]
|
||||
|
||||
'@oxc-parser/binding-win32-arm64-msvc@0.132.0':
|
||||
resolution: {integrity: sha512-RH/NbFjGKqdUAUi7Oh3LQPxUk2hsWFEEQ38HSnbRQT8QjBZFKqL1fMbmsB3N4jy/KPh9iX94+9dmkEMBBbambw==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [arm64]
|
||||
os: [win32]
|
||||
|
||||
'@oxc-parser/binding-win32-ia32-msvc@0.131.0':
|
||||
resolution: {integrity: sha512-m+jNz9EuF0NXoiptc6B9h5yompZQVW/a5MJeOu5zojfH5yWk82tvF2ccrHkfhgtrS9h9DD5l1Qv8dWlfY7Nz8g==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [ia32]
|
||||
os: [win32]
|
||||
|
||||
'@oxc-parser/binding-win32-ia32-msvc@0.132.0':
|
||||
resolution: {integrity: sha512-JUr4jQY9jxoIB/YTLXr6XofSi5xikj6p5/Ns1h0VOBDT0j1jKU+kMsv2xxv51RwnETcXpA1Yw/9oUAfcqfaqEA==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [ia32]
|
||||
os: [win32]
|
||||
|
||||
'@oxc-parser/binding-win32-x64-msvc@0.131.0':
|
||||
resolution: {integrity: sha512-o14Hk8dAyiEUMFEWEgmAwFZvBt1RzAYLM3xeQ+5315JXgVYhoemivgYcbYVRbsFkS71ShMGlAFE0kPnr460rww==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [x64]
|
||||
os: [win32]
|
||||
|
||||
'@oxc-parser/binding-win32-x64-msvc@0.132.0':
|
||||
resolution: {integrity: sha512-2dapgHpA5X8DSXF4AU36hJWYf6zP0tKjMXFRAZFBD62pkevW/uhFDXoFH9Y/3Fd2EtDrw5ByNnR1wVE9X9y0SQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
cpu: [x64]
|
||||
os: [win32]
|
||||
|
||||
'@oxc-project/types@0.110.0':
|
||||
resolution: {integrity: sha512-6Ct21OIlrEnFEJk5LT4e63pk3btsI6/TusD/GStLi7wYlGJNOl1GI9qvXAnRAxQU9zqA2Oz+UwhfTOU2rPZVow==}
|
||||
|
||||
'@oxc-project/types@0.131.0':
|
||||
resolution: {integrity: sha512-PgnWDfV0h+b16XNKbXU7Daib/BFSt/J2mEzfYIBu6JB/wNdlU+kVYXCkGA1A9fWkTbOgbjh4e6NhPeQOYvFhEA==}
|
||||
|
||||
'@oxc-project/types@0.132.0':
|
||||
resolution: {integrity: sha512-FESMOxil5Se014ui/Eq8fT5uHJo6nIRwH0PfJrZJXs6Gek3ZVFOrpUv3YIZT20m+extU98Hg1Ym72U58rlsxUQ==}
|
||||
|
||||
'@oxc-project/types@0.139.0':
|
||||
resolution: {integrity: sha512-r9gHphtCs+1M7J0pw6Sn/hh/Wpa/iQrOOkrNAlVLF/gHq+/CJmHIWKKUUhdWjcD6CIa8idarspCsASiXCXvFUw==}
|
||||
|
||||
@@ -9673,6 +9843,14 @@ packages:
|
||||
dc-browser@1.0.4:
|
||||
resolution: {integrity: sha512-7oEtnzNlcE+hr4OvO3GR6Gndgw8BhW+wKOEwMqSleyY7N29jbAxzyW5BaJl7qBCw+6OIxfMWtY0T+6dxq8RWLw==}
|
||||
|
||||
dc-polyfill@0.1.11:
|
||||
resolution: {integrity: sha512-TyyeGcjx0YeThAI9fTFtgsvj5qd4R+aGfVmXiUhevbgzWFDr7IK4tv4YjE6jaGzLHQTchk4h7DHdr5q4WGgaZw==}
|
||||
engines: {node: '>=12.17'}
|
||||
|
||||
dd-trace@6.13.0:
|
||||
resolution: {integrity: sha512-/bSj1/IEZ592KyrsdBTB0nVyWcUABmayun/aImoCsJa+hVATumr5d+njqdLPR2WrXSJ7kq4KnZjrnkYgRTH7oA==}
|
||||
engines: {node: '>=22'}
|
||||
|
||||
debounce-fn@4.0.0:
|
||||
resolution: {integrity: sha512-8pYCQiL9Xdcg0UPSD3d+0KMlOjp+KGU5EPwYddgzQ7DATsg4fuUDjQtsYLmWjnk2obnNHgV3vE2Y4jejSOJVBQ==}
|
||||
engines: {node: '>=10'}
|
||||
@@ -11230,8 +11408,8 @@ packages:
|
||||
resolution: {integrity: sha512-TR3KfrTZTYLPB6jUjfx6MF9WcWrHL9su5TObK4ZkYgBdWKPOFoSoQIdEuTuR82pmtxH2spWG9h6etwfr1pLBqQ==}
|
||||
engines: {node: '>=6'}
|
||||
|
||||
import-in-the-middle@3.3.1:
|
||||
resolution: {integrity: sha512-0rymlHSFLwZ0ixx8DaQkoIyZojJPY2a0K2nEYslhKJ6jIYO/m0IcCb7iQsFPmS7WmKwISZiIrv5Icstrw/CmqA==}
|
||||
import-in-the-middle@3.3.3:
|
||||
resolution: {integrity: sha512-AiohS3H80sXO6owEltjGX+glb7qXaDhBoJb9XcQVH4UI207xu/bDLUcadVKp7Qe576reg9yr/PXZjV5qx8gfbA==}
|
||||
engines: {node: '>=18'}
|
||||
|
||||
import-lazy@4.0.0:
|
||||
@@ -11614,6 +11792,10 @@ packages:
|
||||
js-yaml@4.1.1:
|
||||
resolution: {integrity: sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==}
|
||||
|
||||
js-yaml@4.3.2:
|
||||
resolution: {integrity: sha512-SFNOvSJ+Dgf/9An904Yx+CgSlIPCkIpao4qo51lpee25TIRejdH3rhR4EZMGoNx3/TP3O+wzWuiTFl4sqbltzA==}
|
||||
hasBin: true
|
||||
|
||||
jsc-safe-url@0.2.4:
|
||||
resolution: {integrity: sha512-0wM3YBWtYePOjfyXQH5MWQ8H7sdk5EXSwZvmSLKk2RboVQ2Bu239jycHDz5J/8Blf3K0Qnoy2b6xD+z10MFB+Q==}
|
||||
|
||||
@@ -12631,6 +12813,9 @@ packages:
|
||||
xml2js:
|
||||
optional: true
|
||||
|
||||
node-addon-api@6.1.0:
|
||||
resolution: {integrity: sha512-+eawOlIgy680F0kBzPUNFhMZGtJ1YmqM6l4+Crf4IkImjYrO/mqPwRMh352g23uIaQKFItcQ64I7KMaJxHgAVA==}
|
||||
|
||||
node-addon-api@7.1.1:
|
||||
resolution: {integrity: sha512-5m3bsyrjFWE1xf7nz7YXdN4udnVtXK6/Yfgn5qnahL6bCkf2yKt4k3nuTKAtT4r3IG8JNR2ncsIMdZuAzJjHQQ==}
|
||||
|
||||
@@ -12681,6 +12866,10 @@ packages:
|
||||
resolution: {integrity: sha512-LarFH0+6VfriEhqMMcLX2F7SwSXeWwnEAJEsYm5QKWchiVYVvJyV9v7UDvUv+w5HO23ZpQTXDv/GxdDdMyOuoQ==}
|
||||
engines: {node: '>= 6.13.0'}
|
||||
|
||||
node-gyp-build@3.9.0:
|
||||
resolution: {integrity: sha512-zLcTg6P4AbcHPq465ZMFNXx7XpKKJh+7kkN699NiQWisR2uWYOWNWqRHAmbnmKiL4e9aLSlmy5U7rEMUXV59+A==}
|
||||
hasBin: true
|
||||
|
||||
node-gyp-build@4.8.4:
|
||||
resolution: {integrity: sha512-LA4ZjwlnUblHVgq0oBF3Jl/6h/Nvs5fzBLwdEF4nuxnFdsfajde4WfxtJr3CaiH+F6ewcIB/q4jQ4UzPyid+CQ==}
|
||||
|
||||
@@ -12911,6 +13100,10 @@ packages:
|
||||
zod:
|
||||
optional: true
|
||||
|
||||
opentracing@0.14.7:
|
||||
resolution: {integrity: sha512-vz9iS7MJ5+Bp1URw8Khvdyw1H/hGvzHWlKQ7eRrQojSCDL1/SrWfrY9QebLw97n2deyRtzHRC3MkQfVNUCo91Q==}
|
||||
engines: {node: '>=0.10'}
|
||||
|
||||
optionator@0.9.4:
|
||||
resolution: {integrity: sha512-6IpQ7mKUxRcZNLIObR0hz7lxsapSSIYNZJwXPGeF0mTVqGKFIXj1DQcMoT22S3ROcLyY/rz0PWaWZ9ayWmad9g==}
|
||||
engines: {node: '>= 0.8.0'}
|
||||
@@ -12935,6 +13128,10 @@ packages:
|
||||
resolution: {integrity: sha512-SJ3/7ZPbgie8dr5Z9BI/M51zZbpXba+hRSG0MDzVwMW5CRQg2fjYE0jHGlLX4eeiibGgC/mzoDFKSDHwVZEHRQ==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
|
||||
oxc-parser@0.132.0:
|
||||
resolution: {integrity: sha512-+0LAPHaqtfQlvWdpaAa09SmOaZZgP8C552xosEkGJ4+ruEwP1Vgx+sqBgcBCNfR6KDCmagGOZTde8wmAvcI/Hg==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
|
||||
oxc-transform@0.111.0:
|
||||
resolution: {integrity: sha512-oa5KKSDNLHZGaiqIGAbCWXeN9IJUAz9MElWcQX90epDxdKc9Hrt/BsLj3K4gDqfAYa5dwdH+ZCFJG9hR74fiGg==}
|
||||
engines: {node: ^20.19.0 || >=22.12.0}
|
||||
@@ -13499,6 +13696,9 @@ packages:
|
||||
resolution: {integrity: sha512-dM0jVuXJPsDN6DvRpea484tCUaMiXWjuCn++HGTqUWzGDjv5tZkEZldAJ/UMlqRYGFrD/etByo4/xOuC/snX2A==}
|
||||
engines: {node: '>=20'}
|
||||
|
||||
pprof-format@2.3.1:
|
||||
resolution: {integrity: sha512-y51Z83qG2vEQBACPu6lkGFREVkHwQaCaNDdSFEMLIqSo3bmpADsbP6J3F2SSk7tYB741oTQ9Kt5YAdQsmiCRkA==}
|
||||
|
||||
preact-render-to-string@6.5.11:
|
||||
resolution: {integrity: sha512-ubnauqoGczeGISiOh6RjX0/cdaF8v/oDXIjO85XALCQjwQP+SB4RDXXtvZ6yTYSjG+PC1QRP2AhPgCEsM2EvUw==}
|
||||
peerDependencies:
|
||||
@@ -14322,6 +14522,10 @@ packages:
|
||||
resolution: {integrity: sha512-i5uvt8C3ikiWeNZSVZNWcfZPItFQOsYTUAOkcUPGd8DqDy1uOUikjt5dG+uRlwyvR108Fb9DOd4GvXfT0N2/uQ==}
|
||||
engines: {node: '>= 12'}
|
||||
|
||||
source-map@0.8.0:
|
||||
resolution: {integrity: sha512-d8EqvL+k/SOXCreS/SUzg2ciyHqBBLcN/yuRjFsbvVhHTE2pgei7oAhmPM7kWFbkX6OSMQfUq4KbkF3au9lhYQ==}
|
||||
engines: {node: '>= 12'}
|
||||
|
||||
space-separated-tokens@2.0.2:
|
||||
resolution: {integrity: sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q==}
|
||||
|
||||
@@ -16832,6 +17036,40 @@ snapshots:
|
||||
'@colors/colors@1.5.0':
|
||||
optional: true
|
||||
|
||||
'@datadog/libdatadog@0.12.1':
|
||||
optional: true
|
||||
|
||||
'@datadog/native-appsec@11.0.2':
|
||||
dependencies:
|
||||
node-gyp-build: 3.9.0
|
||||
optional: true
|
||||
|
||||
'@datadog/native-iast-taint-tracking@4.2.1':
|
||||
dependencies:
|
||||
node-gyp-build: 3.9.0
|
||||
optional: true
|
||||
|
||||
'@datadog/native-metrics@4.0.0':
|
||||
dependencies:
|
||||
node-addon-api: 6.1.0
|
||||
node-gyp-build: 3.9.0
|
||||
optional: true
|
||||
|
||||
'@datadog/pprof@5.18.1':
|
||||
dependencies:
|
||||
node-gyp-build: 4.8.4
|
||||
pprof-format: 2.3.1
|
||||
source-map: 0.8.0
|
||||
optional: true
|
||||
|
||||
'@datadog/wasm-js-rewriter@5.0.4':
|
||||
dependencies:
|
||||
js-yaml: 4.3.2
|
||||
lru-cache: 7.18.3
|
||||
module-details-from-path: 1.0.4
|
||||
node-gyp-build: 4.8.4
|
||||
optional: true
|
||||
|
||||
'@dimforge/rapier3d-compat@0.12.0': {}
|
||||
|
||||
'@dotenvx/dotenvx@1.75.1':
|
||||
@@ -18797,7 +19035,7 @@ snapshots:
|
||||
dependencies:
|
||||
'@opentelemetry/api': 1.9.1
|
||||
'@opentelemetry/api-logs': 0.219.0
|
||||
import-in-the-middle: 3.3.1
|
||||
import-in-the-middle: 3.3.3
|
||||
require-in-the-middle: 8.0.1(supports-color@10.2.2)
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
@@ -19009,51 +19247,99 @@ snapshots:
|
||||
'@oxc-parser/binding-android-arm-eabi@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-android-arm-eabi@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-android-arm64@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-android-arm64@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-darwin-arm64@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-darwin-arm64@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-darwin-x64@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-darwin-x64@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-freebsd-x64@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-freebsd-x64@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-arm-gnueabihf@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-arm-gnueabihf@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-arm-musleabihf@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-arm-musleabihf@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-arm64-gnu@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-arm64-gnu@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-arm64-musl@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-arm64-musl@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-ppc64-gnu@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-ppc64-gnu@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-riscv64-gnu@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-riscv64-gnu@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-riscv64-musl@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-riscv64-musl@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-s390x-gnu@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-s390x-gnu@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-x64-gnu@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-x64-gnu@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-x64-musl@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-linux-x64-musl@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-openharmony-arm64@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-openharmony-arm64@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-wasm32-wasi@0.131.0':
|
||||
dependencies:
|
||||
'@emnapi/core': 1.10.0
|
||||
@@ -19061,19 +19347,38 @@ snapshots:
|
||||
'@napi-rs/wasm-runtime': 1.1.6(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-wasm32-wasi@0.132.0':
|
||||
dependencies:
|
||||
'@emnapi/core': 1.10.0
|
||||
'@emnapi/runtime': 1.10.0
|
||||
'@napi-rs/wasm-runtime': 1.1.6(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-win32-arm64-msvc@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-win32-arm64-msvc@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-win32-ia32-msvc@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-win32-ia32-msvc@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-win32-x64-msvc@0.131.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-parser/binding-win32-x64-msvc@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-project/types@0.110.0': {}
|
||||
|
||||
'@oxc-project/types@0.131.0': {}
|
||||
|
||||
'@oxc-project/types@0.132.0':
|
||||
optional: true
|
||||
|
||||
'@oxc-project/types@0.139.0': {}
|
||||
|
||||
'@oxc-project/types@0.148.0': {}
|
||||
@@ -24260,7 +24565,7 @@ snapshots:
|
||||
cosmiconfig@8.3.6(typescript@7.0.2):
|
||||
dependencies:
|
||||
import-fresh: 3.3.1
|
||||
js-yaml: 4.1.1
|
||||
js-yaml: 4.3.2
|
||||
parse-json: 5.2.0
|
||||
path-type: 4.0.0
|
||||
optionalDependencies:
|
||||
@@ -24647,6 +24952,24 @@ snapshots:
|
||||
|
||||
dc-browser@1.0.4: {}
|
||||
|
||||
dc-polyfill@0.1.11: {}
|
||||
|
||||
dd-trace@6.13.0:
|
||||
dependencies:
|
||||
dc-polyfill: 0.1.11
|
||||
import-in-the-middle: 3.3.3
|
||||
opentracing: 0.14.7
|
||||
optionalDependencies:
|
||||
'@datadog/libdatadog': 0.12.1
|
||||
'@datadog/native-appsec': 11.0.2
|
||||
'@datadog/native-iast-taint-tracking': 4.2.1
|
||||
'@datadog/native-metrics': 4.0.0
|
||||
'@datadog/pprof': 5.18.1
|
||||
'@datadog/wasm-js-rewriter': 5.0.4
|
||||
'@opentelemetry/api': 1.9.1
|
||||
'@opentelemetry/api-logs': 0.221.0
|
||||
oxc-parser: 0.132.0
|
||||
|
||||
debounce-fn@4.0.0:
|
||||
dependencies:
|
||||
mimic-fn: 3.1.0
|
||||
@@ -26766,7 +27089,7 @@ snapshots:
|
||||
parent-module: 1.0.1
|
||||
resolve-from: 4.0.0
|
||||
|
||||
import-in-the-middle@3.3.1:
|
||||
import-in-the-middle@3.3.3:
|
||||
dependencies:
|
||||
cjs-module-lexer: 2.2.0
|
||||
es-module-lexer: 2.3.1
|
||||
@@ -27150,6 +27473,10 @@ snapshots:
|
||||
dependencies:
|
||||
argparse: 2.0.1
|
||||
|
||||
js-yaml@4.3.2:
|
||||
dependencies:
|
||||
argparse: 2.0.1
|
||||
|
||||
jsc-safe-url@0.2.4: {}
|
||||
|
||||
jsesc@3.1.0: {}
|
||||
@@ -28875,6 +29202,9 @@ snapshots:
|
||||
- vite
|
||||
- webpack
|
||||
|
||||
node-addon-api@6.1.0:
|
||||
optional: true
|
||||
|
||||
node-addon-api@7.1.1: {}
|
||||
|
||||
node-domexception@1.0.0: {}
|
||||
@@ -28908,6 +29238,9 @@ snapshots:
|
||||
|
||||
node-forge@1.4.0: {}
|
||||
|
||||
node-gyp-build@3.9.0:
|
||||
optional: true
|
||||
|
||||
node-gyp-build@4.8.4: {}
|
||||
|
||||
node-html-parser@7.1.0:
|
||||
@@ -29426,6 +29759,8 @@ snapshots:
|
||||
ws: 8.21.3(bufferutil@4.1.0)
|
||||
zod: 4.5.4
|
||||
|
||||
opentracing@0.14.7: {}
|
||||
|
||||
optionator@0.9.4:
|
||||
dependencies:
|
||||
deep-is: 0.1.4
|
||||
@@ -29503,6 +29838,32 @@ snapshots:
|
||||
'@oxc-parser/binding-win32-ia32-msvc': 0.131.0
|
||||
'@oxc-parser/binding-win32-x64-msvc': 0.131.0
|
||||
|
||||
oxc-parser@0.132.0:
|
||||
dependencies:
|
||||
'@oxc-project/types': 0.132.0
|
||||
optionalDependencies:
|
||||
'@oxc-parser/binding-android-arm-eabi': 0.132.0
|
||||
'@oxc-parser/binding-android-arm64': 0.132.0
|
||||
'@oxc-parser/binding-darwin-arm64': 0.132.0
|
||||
'@oxc-parser/binding-darwin-x64': 0.132.0
|
||||
'@oxc-parser/binding-freebsd-x64': 0.132.0
|
||||
'@oxc-parser/binding-linux-arm-gnueabihf': 0.132.0
|
||||
'@oxc-parser/binding-linux-arm-musleabihf': 0.132.0
|
||||
'@oxc-parser/binding-linux-arm64-gnu': 0.132.0
|
||||
'@oxc-parser/binding-linux-arm64-musl': 0.132.0
|
||||
'@oxc-parser/binding-linux-ppc64-gnu': 0.132.0
|
||||
'@oxc-parser/binding-linux-riscv64-gnu': 0.132.0
|
||||
'@oxc-parser/binding-linux-riscv64-musl': 0.132.0
|
||||
'@oxc-parser/binding-linux-s390x-gnu': 0.132.0
|
||||
'@oxc-parser/binding-linux-x64-gnu': 0.132.0
|
||||
'@oxc-parser/binding-linux-x64-musl': 0.132.0
|
||||
'@oxc-parser/binding-openharmony-arm64': 0.132.0
|
||||
'@oxc-parser/binding-wasm32-wasi': 0.132.0
|
||||
'@oxc-parser/binding-win32-arm64-msvc': 0.132.0
|
||||
'@oxc-parser/binding-win32-ia32-msvc': 0.132.0
|
||||
'@oxc-parser/binding-win32-x64-msvc': 0.132.0
|
||||
optional: true
|
||||
|
||||
oxc-transform@0.111.0(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.3):
|
||||
optionalDependencies:
|
||||
'@oxc-transform/binding-android-arm-eabi': 0.111.0
|
||||
@@ -30130,6 +30491,9 @@ snapshots:
|
||||
|
||||
powershell-utils@0.1.0: {}
|
||||
|
||||
pprof-format@2.3.1:
|
||||
optional: true
|
||||
|
||||
preact-render-to-string@6.5.11(preact@10.24.3):
|
||||
dependencies:
|
||||
preact: 10.24.3
|
||||
@@ -31411,6 +31775,9 @@ snapshots:
|
||||
|
||||
source-map@0.7.6: {}
|
||||
|
||||
source-map@0.8.0:
|
||||
optional: true
|
||||
|
||||
space-separated-tokens@2.0.2: {}
|
||||
|
||||
split-ca@1.0.1: {}
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
issue: "TBD (Datadog Experiments reporter partner request)"
|
||||
status: proposed
|
||||
last_updated: "2026-09-01"
|
||||
---
|
||||
|
||||
# Datadog Experiments reporter for eve evals
|
||||
|
||||
Add a reporter that publishes an eve eval run to Datadog LLM Observability
|
||||
Experiments without asking the Datadog SDK to rerun the task.
|
||||
|
||||
## Authoring API
|
||||
|
||||
```ts
|
||||
import { defineEvalConfig } from "eve/evals";
|
||||
import { Datadog } from "eve/evals/reporters";
|
||||
|
||||
export default defineEvalConfig({
|
||||
reporters: [
|
||||
Datadog({
|
||||
projectName: "weather-agent",
|
||||
recordInputs: false,
|
||||
recordOutputs: false,
|
||||
}),
|
||||
],
|
||||
});
|
||||
```
|
||||
|
||||
One shared reporter instance creates one Experiment for the run. Each completed
|
||||
eval becomes one synthetic experiment span, and each assertion becomes an
|
||||
Experiment metric associated with the span returned by Datadog.
|
||||
|
||||
```text
|
||||
evals.config.ts ── Datadog(...) ──► Experiment
|
||||
├─ eval row + assertion metrics
|
||||
├─ eval row + assertion metrics
|
||||
└─ completed/failed status
|
||||
```
|
||||
|
||||
## SDK boundary
|
||||
|
||||
Target the public external Experiment API in `dd-trace@6.13.0`:
|
||||
|
||||
- `tracer.llmobs.experiments.createDataset(...)`
|
||||
- `dataset.push()`
|
||||
- `tracer.llmobs.experiments.startExperiment(...)`
|
||||
- `experiment.submitSpan(...)`
|
||||
- `experiment.submitEvaluationMetrics(...)`
|
||||
- `experiment.close(...)`
|
||||
|
||||
`submitSpan` generates and returns the row's `traceId` and `spanId`. Its input
|
||||
does not accept caller-owned `id`, `traceId`, `spanId`, or `apmTraceId` fields.
|
||||
The reporter must not fabricate those fields or mutate `dd-trace` internals.
|
||||
Keep `dd-trace` optional and load it from the app so the published `eve` package
|
||||
does not gain a runtime dependency.
|
||||
|
||||
Initialize LLM Observability with both `projectName` and `mlApp`. Resolve the
|
||||
project from explicit reporter config, `DD_LLMOBS_PROJECT_NAME`, ml_app config,
|
||||
`DD_LLMOBS_ML_APP`, `DD_SERVICE`, or the first eval id, in that order.
|
||||
|
||||
## Row mapping
|
||||
|
||||
- `name`: the path-derived eve eval id.
|
||||
- `input`, `output`, and expected output: opt-in because they may contain user
|
||||
data. When input recording is enabled, completed evals are buffered until run
|
||||
completion, their inputs are pushed as versioned dataset records, and each
|
||||
experiment span carries its generated dataset record id. Expected output is
|
||||
included in the dataset record only when present and explicitly enabled.
|
||||
- `metadata`: verdict, status, session id, tool/subagent names, authored eval
|
||||
metadata, sanitized target origin, and local git metadata when available.
|
||||
- `tags`: eval id, verdict, status, and authored eval tags.
|
||||
- metrics: assertion scores under normalized descriptive assertion names plus
|
||||
tool, subagent, message, and reasoning counts. Gate labels receive a `gate_`
|
||||
prefix, and authors can set stable names with `.label(...)`. Raw assertion-name
|
||||
tags and failure messages require the explicit `recordAssertionDetails`
|
||||
privacy opt-in.
|
||||
- timestamps: the values already captured by the eval runner.
|
||||
- error: opt-in because exception messages may contain application data.
|
||||
|
||||
All metadata, config, inputs, and outputs are normalized to the JSON value shape
|
||||
accepted by `dd-trace` before submission.
|
||||
|
||||
## Trace-linking boundary
|
||||
|
||||
The external Experiment API creates a synthetic row span and does not support
|
||||
post-hoc association with an already-ingested agent runtime span. Do not add
|
||||
private propagation headers, expose trace identifiers in stream events, force
|
||||
global Datadog environment variables, or mutate private `dd-trace` span fields
|
||||
to simulate that association.
|
||||
|
||||
If Datadog adds a public linking API later, eve can expose an internal,
|
||||
reporter-visible trace link and submit it through that API. Synthetic spans
|
||||
remain the fallback for remote or uninstrumented targets.
|
||||
|
||||
## Validation
|
||||
|
||||
- Unit-test experiment lifecycle, dataset creation and record linkage, row
|
||||
mapping, privacy switches, metric labels, project resolution, summary status,
|
||||
and URL logging with an injected client.
|
||||
- Pin `dd-trace@6.13.0` as an eve development dependency and in the deterministic
|
||||
reporter fixture so TypeScript and runtime API smoke checks exercise the
|
||||
supported release.
|
||||
- Run the targeted reporter and runner unit tests, eve typecheck, dependency
|
||||
checks, invariant guards, docs checks, and `git diff --check`.
|
||||
- Use real Datadog credentials only for manual dogfood verification; CI tests
|
||||
remain deterministic and network-free.
|
||||
Reference in New Issue
Block a user