From 559ec2842b31f1b2364ce1365c082dd60b493404 Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Thu, 20 Aug 2026 11:29:32 -0400 Subject: [PATCH] Add TypeScript OpenTelemetry integration docs (#245) * Add TypeScript OpenTelemetry integration docs Split out from the OpenTelemetry plugins topic (PR #243) so the TypeScript material can be finalized separately. Adds the TS OTel integration reference, the Distributed Tracing section in TS observability, and the TS row in the integrations catalog. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: align python with ts skill --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: Patrik Beqo Co-authored-by: Patrik Beqo --- references/integrations.md | 3 +- .../python/integrations/opentelemetry.md | 63 +++++++++++++++ references/python/observability.md | 47 +---------- .../typescript/integrations/opentelemetry.md | 77 +++++++++++++++++++ references/typescript/observability.md | 9 ++- 5 files changed, 153 insertions(+), 46 deletions(-) create mode 100644 references/python/integrations/opentelemetry.md create mode 100644 references/typescript/integrations/opentelemetry.md diff --git a/references/integrations.md b/references/integrations.md index 6d31bba..af5953b 100644 --- a/references/integrations.md +++ b/references/integrations.md @@ -20,7 +20,8 @@ Temporal ships and supports a growing set of integrations with third-party frame | LangGraph (`temporalio.contrib.langgraph`, Pre-release) | Python | Runs LangGraph Graph-API and Functional-API code as Temporal Workflows - nodes/tasks can execute as either in-workflow or as Activities | `references/python/integrations/langgraph.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Google ADK (`temporalio[google-adk]`) | Python | Durable Google ADK agents: model calls run through `TemporalModel`-wrapped Activities, tools via `activity_tool`, MCP toolsets via `TemporalMcpToolSet` | `references/python/integrations/google-adk.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Pydantic AI (`pydantic-ai[temporal]`) | Python | Durable agents through the `TemporalDurability` capability, with model requests, tool calls, and MCP communication executed as Temporal Activities | `references/python/integrations/pydantic-ai.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | -| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing for Temporal apps with OpenTelemetry | `references/python/observability.md` (Distributed Tracing section) | | +| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing for Temporal apps with OpenTelemetry | `references/python/integrations/opentelemetry.md` | `references/python/observability.md` | +| OpenTelemetry (`@temporalio/interceptors-opentelemetry`) | TypeScript | Distributed tracing for Temporal apps with OpenTelemetry | `references/typescript/integrations/opentelemetry.md` | `references/typescript/observability.md` | | Braintrust (`braintrust[temporal]`, Public Preview) | Python | LLM observability + prompt management: `BraintrustPlugin` traces every Workflow/Activity, `wrap_openai` captures LLM calls, `start_span` adds custom context, `load_prompt` fetches Braintrust-managed prompts | `references/python/integrations/braintrust.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Braintrust (`@braintrust/temporal`) | TypeScript | LLM observability: `BraintrustTemporalPlugin` registers on Client + Worker to trace Workflow/Activity spans; canonical guide hosted by Braintrust | `references/typescript/integrations/braintrust.md` | `references/core/ai-patterns.md` | | Mastra (`@mastra/temporal`, Public Preview) | TypeScript | Build-time transform of Mastra `createWorkflow`/`createStep` definitions into Temporal Workflows and Activities; `MastraPlugin` auto-registers Activities on the Worker | `references/typescript/integrations/mastra.md` | `references/typescript/typescript.md`, `references/core/ai-patterns.md` | diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md new file mode 100644 index 0000000..efdd8c4 --- /dev/null +++ b/references/python/integrations/opentelemetry.md @@ -0,0 +1,63 @@ +# Temporal OpenTelemetry Integration (Python) + +## Overview + +`temporalio.contrib.opentelemetry` wires OpenTelemetry tracing into Temporal through the `OpenTelemetryPlugin`. It propagates W3C TraceContext + Baggage across Client, Workflow, Activity, and Nexus code and supports replay-safe custom Workflow spans. + +For observability beyond OpenTelemetry tracing (metrics, logging, Search Attributes) read `references/python/observability.md`. + +> [!NOTE] +> This feature is Pre-release. It is acceptable to use it on behalf of a user, but inform them that it is Pre-release. + +## Install the plugin + +Install the `temporalio[opentelemetry]` extra plus the OpenTelemetry exporter packages you use. + +## `OpenTelemetryPlugin` + +Create a replay-safe tracer provider, set it globally before creating the Client, and register the plugin on the Client. Workers created from that Client inherit the plugin automatically. Application spans propagate by default; pass `OpenTelemetryPlugin(add_temporal_spans=True)` to also emit Temporal lifecycle spans. + +```python +import opentelemetry.trace +from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor +from temporalio.client import Client +from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider + +provider = create_tracer_provider() +provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) +opentelemetry.trace.set_tracer_provider(provider) + +client = await Client.connect( + "localhost:7233", + plugins=[OpenTelemetryPlugin()], +) +``` + +Inside a Workflow, use standard OpenTelemetry APIs to create custom replay-safe spans: + +```python +from datetime import timedelta +from opentelemetry.trace import get_tracer +from temporalio import workflow + +@workflow.defn +class MyWorkflow: + @workflow.run + async def run(self) -> None: + tracer = get_tracer(__name__) + with tracer.start_as_current_span("workflow-operation"): + await workflow.execute_activity( + my_activity, + start_to_close_timeout=timedelta(seconds=30), + ) +``` + +## Common mistakes + +- **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit it. +- **Creating a Workflow Worker before installing the replay-safe global provider.** Set the provider returned by `create_tracer_provider(...)` globally before constructing a Worker that uses `OpenTelemetryPlugin`. +- **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires a `ReplaySafeTracerProvider`; build it with `create_tracer_provider(...)`. + +## Resources + +- SDK metrics and observability reference: `references/python/observability.md` diff --git a/references/python/observability.md b/references/python/observability.md index 5ad5e18..ab271b8 100644 --- a/references/python/observability.md +++ b/references/python/observability.md @@ -4,7 +4,7 @@ The Python SDK provides comprehensive observability through logging, metrics, tracing (OpenTelemetry), and visibility (Search Attributes). -These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate health, **tracing** stitches a single request across Client/Workflow/Activity boundaries, and **Search Attributes** make executions queryable. +These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate worker health, **tracing** stitches a single request across Client/Workflow/Activity/Nexus boundaries, and **Search Attributes** make executions queryable. ## Logging @@ -98,48 +98,7 @@ Runtime.set_default(runtime, error_if_already_set=True) ## Distributed Tracing (OpenTelemetry) -> [!NOTE] -> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. - -OpenTelemetry is the supported way to add distributed tracing to Temporal applications. The `OpenTelemetryPlugin` (from `temporalio.contrib.opentelemetry`, installed via the `temporalio[opentelemetry]` extra) propagates W3C TraceContext + Baggage through Temporal headers across Client, Workflow, Activity (including Standalone), and Child Workflow boundaries, so one trace follows a request through your whole execution — with replay-safe, accurate span durations. - -```python -import opentelemetry.trace -from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor -from temporalio.client import Client -from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider - -provider = create_tracer_provider() -provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) # attach your span processors as normal for OTel -opentelemetry.trace.set_tracer_provider(provider) - -client = await Client.connect("localhost:7233", plugins=[OpenTelemetryPlugin()]) -``` - -Workers created from this Client inherit the plugin automatically. Inside a Workflow you then use standard OpenTelemetry APIs (`get_tracer(...).start_as_current_span(...)`); pass `OpenTelemetryPlugin(add_temporal_spans=True)` to also emit `StartWorkflow` / `RunWorkflow` / `StartActivity` / `RunActivity` spans automatically alongside the SDK metrics above. - -```python -from datetime import timedelta -from opentelemetry.trace import get_tracer -from temporalio import workflow - -@workflow.defn -class MyWorkflow: - @workflow.run - async def run(self) -> None: - tracer = get_tracer(__name__) - with tracer.start_as_current_span("workflow-operation"): - await workflow.execute_activity( - my_activity, - start_to_close_timeout=timedelta(seconds=30), - ) -``` - -**Common mistakes:** - -- **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. -- **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.** `OpenTelemetryPlugin` raises an exception unless the global tracer provider is already set. -- **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires a `ReplaySafeTracerProvider` — build it via `create_tracer_provider(...)`. +See `references/python/integrations/opentelemetry.md`. ## Search Attributes (Visibility) @@ -151,4 +110,4 @@ See the Search Attributes section of `references/python/data-handling.md` 2. Don't use print() in workflows - it will produce duplicate output on replay 3. Configure metrics for production monitoring 4. Use Search Attributes for business-level visibility -5. Use the `OpenTelemetryPlugin` for distributed tracing across Client/Workflow/Activity boundaries. +5. Use the `OpenTelemetryPlugin` for distributed tracing across Client/Workflow/Activity/Nexus boundaries. diff --git a/references/typescript/integrations/opentelemetry.md b/references/typescript/integrations/opentelemetry.md new file mode 100644 index 0000000..c11afb2 --- /dev/null +++ b/references/typescript/integrations/opentelemetry.md @@ -0,0 +1,77 @@ +# Temporal OpenTelemetry Integration (TypeScript) + +## Overview + +`@temporalio/interceptors-opentelemetry` wires OpenTelemetry tracing into Temporal through the `OpenTelemetryPlugin`. It traces Client, Workflow, Activity, and Nexus code, propagating W3C TraceContext + Baggage across all of them. + +Workflow-side spans are emitted out of the Workflow isolate through an injected Sink that hands serialized spans to a host-side `SpanProcessor`. + +For observability beyond OpenTelemetry tracing (metrics, runtime logger, sinks) read `references/typescript/observability.md`. + +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +## Install the plugin + +Install `@temporalio/interceptors-opentelemetry` plus the OpenTelemetry peer packages you use — typically `@opentelemetry/api`, `@opentelemetry/sdk-trace-base`, and `@opentelemetry/resources` (plus an exporter package such as `@opentelemetry/exporter-trace-otlp-grpc` when you ship spans to a collector). + +## `OpenTelemetryPlugin` + +Construct one `OpenTelemetryPlugin` and pass it to the Client, `bundleWorkflowCode`, and `Worker.create`. It must reach `bundleWorkflowCode` so the Workflow-side interceptors are included in the bundle. Lifecycle spans (workflow / activity / client / nexus) are then created automatically. + +```ts +import { Resource } from '@opentelemetry/resources'; +import { BasicTracerProvider, ConsoleSpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'; +import { NativeConnection, Worker, bundleWorkflowCode } from '@temporalio/worker'; +import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; + +const resource = new Resource({ 'service.name': 'orders-worker' }); +const spanProcessor = new SimpleSpanProcessor(new ConsoleSpanExporter()); // swap in your own exporter + +const provider = new BasicTracerProvider({ resource }); +provider.addSpanProcessor(spanProcessor); +provider.register(); + +// `resource` and `spanProcessor` are required; pass an optional `tracer` to override +// the tracer used by the Client/Activity interceptors. +const plugin = new OpenTelemetryPlugin({ resource, spanProcessor }); + +const bundle = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), + plugins: [plugin], +}); + +const connection = await NativeConnection.connect(); +const worker = await Worker.create({ + connection, + taskQueue: 'orders', + workflowBundle: bundle, + activities: { /* ... */ }, + plugins: [plugin], +}); +await worker.run(); +``` + +Pass the same plugin to the Client so client-side calls are traced: + +```ts +import { Client, Connection } from '@temporalio/client'; + +const client = new Client({ + connection: await Connection.connect(), + plugins: [plugin], +}); +``` + +The SDK uses the global OpenTelemetry propagator (default: W3C TraceContext + Baggage). To use a non-default propagator (e.g. Jaeger), call `propagation.setGlobalPropagator(...)` at the top level of your Workflow code BEFORE the Worker bundles it. + +## Common mistakes + +- **Passing only `resource` or only `spanProcessor`.** Both are required; `new OpenTelemetryPlugin()` with no argument throws. +- **Passing the plugin to `Worker.create` but not `bundleWorkflowCode`.** Workflow-side interceptors must be in the bundle. +- **Installing `@temporalio/opentelemetry`.** The package is `@temporalio/interceptors-opentelemetry`. +- **Expecting a non-default propagator (e.g. Jaeger) to work without setting the global propagator before `bundleWorkflowCode` runs.** + +## Resources + +- SDK metrics / observability reference: `references/typescript/observability.md` diff --git a/references/typescript/observability.md b/references/typescript/observability.md index 211fbc6..d5c8a77 100644 --- a/references/typescript/observability.md +++ b/references/typescript/observability.md @@ -2,7 +2,9 @@ ## Overview -The TypeScript SDK provides replay-aware logging, metrics, and integrations for production observability. +The TypeScript SDK provides replay-aware logging, metrics, and distributed tracing (OpenTelemetry) for production observability. + +These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate worker health, **tracing** stitches a single request across Client/Workflow/Activity/Nexus boundaries, and **Search Attributes** make executions queryable. ## Replay-Aware Logging @@ -100,6 +102,10 @@ Runtime.install({ }); ``` +## Distributed Tracing (OpenTelemetry) + +See `references/typescript/integrations/opentelemetry.md`. + ## Search Attributes (Visibility) See the Search Attributes section of `references/typescript/data-handling.md` @@ -111,3 +117,4 @@ See the Search Attributes section of `references/typescript/data-handling.md` 3. Configure Winston or similar for production log aggregation 4. Monitor Prometheus metrics for worker health 5. Use Event History for debugging workflow issues +6. Use the `OpenTelemetryPlugin` for distributed tracing across Client/Workflow/Activity/Nexus boundaries.