Distributed Tracing Across Agent Chains: A Practical Implementation Guide

Why Tracing Is Different for AI Agents

Distributed tracing was designed for microservices, systems where a request enters at an API gateway and passes through a predictable sequence of services, each making deterministic decisions. AI agent systems break every assumption that makes traditional distributed tracing straightforward. Agent calls are non-deterministic: the same input might cause the planner to spawn two research sub-agents one time and five the next. Control flow is data-driven: which tool gets called depends on what the LLM decides to do, not on code branches you wrote. And the most expensive operations, LLM inference calls that take multiple seconds and cost fractions of a dollar, are invisible to standard infrastructure monitoring.

Despite these differences, distributed tracing remains the most valuable observability primitive for multi-agent systems. When a user reports that their request took 45 seconds and returned a wrong answer, a complete trace is the difference between a two-minute diagnosis and a two-hour investigation. A good trace tells you exactly which agents ran, which tools they called, how long each LLM call took, how many tokens were consumed, and whether any errors occurred along the way.

This covers the full stack: OpenTelemetry primitives, how to wrap LLM SDK calls with proper spans, how to propagate trace context across async agent boundaries, and how to build a sampling strategy that captures the traces you need without drowning in volume.

The Anatomy of an Agent Trace

An agent trace is a tree of spans rooted at the original user request. Every agent invocation, every LLM call, and every tool call becomes a span. Spans nest according to their causal relationships: the planner's LLM call is a child of the planner agent span, which is a child of the root request span. This nesting lets you answer both local questions (how long did this specific LLM call take?) and global questions (how much of the total request time was spent waiting for external tools?).

Essential Span Attributes for AI Workloads

Standard spans carry a span name, start time, duration, status, and arbitrary key-value attributes. For AI workloads, you need a defined set of attributes that every span captures to enable cross-span queries. The OpenTelemetry Semantic Conventions for GenAI define a standard schema for this.

• gen_ai.system: The AI provider (anthropic, openai, google). Enables filtering by provider across all spans.
• gen_ai.request.model: The specific model identifier (claude-sonnet-4-5, gpt-4o). Lets you compare latency and cost by model version.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens: Token counts per LLM call. These are the primary cost drivers and must be on every LLM span.
• gen_ai.agent.id and gen_ai.agent.role: Which agent produced this span. Critical for multi-agent attribution.
• gen_ai.tool.name and gen_ai.tool.status: For tool call spans, what tool ran and whether it succeeded.

OpenTelemetry Primer for LLM Applications

OpenTelemetry (OTel) is the industry standard for distributed observability. It provides language-specific SDKs that instrument your code and export spans to any compatible backend. For Python-based agent systems, the opentelemetry-sdk package provides the core primitives. For Node.js, the @opentelemetry/sdk-node package is the equivalent.

Setting Up the Tracer Provider

The Tracer Provider is the central configuration object that defines how spans are processed and exported. In a production agent system, you configure it once at application startup and inject it into every component that needs to create spans.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME

def setup_tracing(service_name: str, otlp_endpoint: str = "http://localhost:4317"):
    resource = Resource.create({
        SERVICE_NAME: service_name,
        "service.version": "1.2.0",
        "deployment.environment": "production",
    })

    exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
    processor = BatchSpanProcessor(
        exporter,
        max_export_batch_size=512,
        schedule_delay_millis=5000,
    )

    provider = TracerProvider(resource=resource)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)

    return trace.get_tracer(service_name)

# At startup:
tracer = setup_tracing("agentix-research-agent")

Instrumenting LLM Calls as Spans

Every LLM API call should be wrapped in a span. This gives you precise timing, token usage, model metadata, and error information for every inference call in your system. The span should be created before the API call and ended, with status set, after the response is received.

from anthropic import AsyncAnthropic
from opentelemetry import trace
from opentelemetry.trace import StatusCode
from opentelemetry.semconv.ai import SpanAttributes  # opentelemetry-semantic-conventions-ai

client = AsyncAnthropic()
tracer = trace.get_tracer(__name__)

async def traced_llm_call(
    model: str,
    system: str,
    messages: list[dict],
    agent_id: str,
    max_tokens: int = 4096,
) -> str:
    with tracer.start_as_current_span(
        "llm.call",
        attributes={
            SpanAttributes.LLM_SYSTEM: "anthropic",
            SpanAttributes.LLM_REQUEST_MODEL: model,
            "gen_ai.agent.id": agent_id,
            "gen_ai.request.max_tokens": max_tokens,
        },
    ) as span:
        try:
            response = await client.messages.create(
                model=model,
                system=system,
                messages=messages,
                max_tokens=max_tokens,
            )
            span.set_attributes({
                SpanAttributes.LLM_USAGE_PROMPT_TOKENS: response.usage.input_tokens,
                SpanAttributes.LLM_USAGE_COMPLETION_TOKENS: response.usage.output_tokens,
                "gen_ai.response.finish_reason": response.stop_reason,
            })
            span.set_status(StatusCode.OK)
            return response.content[0].text
        except Exception as e:
            span.set_status(StatusCode.ERROR, str(e))
            span.record_exception(e)
            raise

Capturing Token Costs on Every Span

Token counts on every LLM span unlock powerful cost queries. You can calculate the total cost of a user request by summing the token counts across all LLM spans in that trace. You can identify which agent role is consuming the most tokens across all requests. And you can set up alerts when a single span exceeds a token budget, which often indicates a prompt injection attempt or a runaway context accumulation bug.

Propagating Trace Context Across Agent Boundaries

Trace context propagation is the mechanism that links spans from different processes into a single trace tree. When an orchestrator agent spawns a research sub-agent in a separate process, the orchestrator must pass the current trace context to the research agent so all spans appear under the same root. OpenTelemetry provides a W3C TraceContext propagator for this: it serializes the current span's trace ID and span ID into HTTP headers that the receiving agent reads to continue the trace.

import httpx
from opentelemetry.propagate import inject, extract
from opentelemetry import trace

async def call_sub_agent(
    agent_url: str,
    task: dict,
) -> dict:
    # Inject current trace context into outgoing request headers
    headers = {}
    inject(headers)  # Adds traceparent and tracestate headers

    async with httpx.AsyncClient() as client:
        response = await client.post(
            agent_url,
            json=task,
            headers=headers,
        )
        return response.json()

# In the receiving sub-agent's request handler:
async def handle_task(request: Request) -> Response:
    # Extract the parent context from incoming headers
    carrier = dict(request.headers)
    parent_context = extract(carrier)

    with tracer.start_as_current_span(
        "research-agent.run",
        context=parent_context,  # Links to parent trace
    ) as span:
        result = await run_research_task(request.json())
        return Response(json=result)

Context Propagation Across Message Queues

When agent communication goes through a message queue rather than direct HTTP calls, trace context must be serialized into the message envelope and deserialized by the consumer. Kafka, SQS, and similar systems do not automatically propagate OTel context. You must implement this in your message producer and consumer code.

import json
from opentelemetry.propagate import inject, extract

def produce_task_message(task: dict, topic: str):
    # Inject trace context into message headers
    carrier = {}
    inject(carrier)

    message = {
        "task": task,
        "otel_context": carrier,  # traceparent, tracestate
    }
    kafka_producer.send(topic, json.dumps(message).encode())

def consume_task_message(raw_message: bytes):
    message = json.loads(raw_message)
    # Re-extract trace context from message
    parent_context = extract(message.get("otel_context", {}))

    with tracer.start_as_current_span(
        "consumer.process_task",
        context=parent_context,
        kind=trace.SpanKind.CONSUMER,
    ) as span:
        process_task(message["task"])

Capturing Tool Calls as Child Spans

Tool calls are often the slowest operations in an agent pipeline and frequently the source of failures. Each tool call should be a child span of the LLM span that decided to invoke it. The tool span should capture the tool name, the exact input arguments, the output (or error), and the duration. For tools that make external API calls, include the external service name and the HTTP status code.

import functools
from typing import Any, Callable

def traced_tool(tool_fn: Callable) -> Callable:
    @functools.wraps(tool_fn)
    async def wrapper(*args, **kwargs) -> Any:
        tool_name = tool_fn.__name__
        with tracer.start_as_current_span(
            f"tool.{tool_name}",
            attributes={
                "gen_ai.tool.name": tool_name,
                "gen_ai.tool.input": str(kwargs),
            },
        ) as span:
            try:
                result = await tool_fn(*args, **kwargs)
                span.set_attribute("gen_ai.tool.output_size", len(str(result)))
                span.set_status(StatusCode.OK)
                return result
            except Exception as e:
                span.set_status(StatusCode.ERROR, str(e))
                span.record_exception(e)
                raise
    return wrapper

# Usage, decorate any tool function
@traced_tool
async def web_search(query: str, num_results: int = 10) -> list[dict]:
    # Tool implementation
    results = await search_api.search(query, num_results)
    return results

Tracing Async and Parallel Agent Execution

When your orchestrator spawns multiple sub-agents in parallel with asyncio.gather(), each coroutine needs to carry the correct parent context. Python's asyncio does not automatically propagate OTel context across coroutines. You must explicitly copy the context before creating async tasks.

import asyncio
from opentelemetry import context as otel_context

async def run_parallel_agents(tasks: list[dict]) -> list[dict]:
    # Capture current context before spawning async tasks
    current_ctx = otel_context.get_current()

    async def run_with_context(task: dict) -> dict:
        # Each coroutine runs with the parent context attached
        token = otel_context.attach(current_ctx)
        try:
            return await run_sub_agent(task)
        finally:
            otel_context.detach(token)

    # All parallel tasks share the same parent span context
    results = await asyncio.gather(
        *[run_with_context(task) for task in tasks]
    )
    return results

Sampling Strategies for High-Volume Agent Systems

Sampling every trace in a high-throughput agent system is expensive, both in terms of the compute overhead of span processing and the storage cost of the trace backend. A well-designed sampling strategy captures 100% of the traces you need (failures, slow requests, anomalies) while sampling down routine successful requests.

Head-Based vs Tail-Based Sampling

Head-based sampling makes the sampling decision at the start of a trace before you know how it will turn out. It is simple and low-overhead but cannot sample based on trace outcome. Tail-based sampling buffers the complete trace before deciding whether to keep it. This allows error-based and latency-based sampling rules but requires a stateful collector that buffers in-flight traces.

from opentelemetry.sdk.trace.sampling import (
    ParentBased,
    TraceIdRatioBased,
    ALWAYS_ON,
    ALWAYS_OFF,
)

# Tail-based sampling rules (implemented in OTel Collector config):
# - Always keep: any trace with error spans
# - Always keep: any trace longer than 5 seconds
# - Always keep: traces sampled by feature flag header
# - Sample 10%: all other successful traces under 5 seconds

# Head-based sampling for high-volume classification endpoint:
classification_sampler = ParentBased(
    root=TraceIdRatioBased(0.01),  # 1% of root spans (no parent)
    remote_parent_sampled=ALWAYS_ON,  # Always follow parent decision
    remote_parent_not_sampled=ALWAYS_OFF,
)

Choosing a Tracing Backend

The tracing backend is where spans are stored, indexed, and queried. The right choice depends on your team's existing infrastructure, your retention requirements, and your budget. All major backends accept OTel-formatted spans via the OTLP protocol, so the choice does not affect your instrumentation code.

• Jaeger (self-hosted, free): Good for teams that want full control and low cost. Requires operational overhead. Cassandra or Elasticsearch backend needed for production scale.
• Grafana Tempo (self-hosted or cloud): Integrates with the Grafana stack (Loki for logs, Prometheus for metrics). Excellent if you already use Grafana.
• Honeycomb: Exceptional query capabilities for high-cardinality trace data. The best tool for asking 'find all traces where token count > 10,000 AND model = claude-opus'. More expensive at scale.
• Datadog APM: Best choice if you already use Datadog for infrastructure monitoring. Tight integration with Datadog metrics and logs. Premium pricing.
• Langfuse (open source, AI-native): Purpose-built for LLM tracing with token cost tracking, model comparison views, and evaluation scoring built into the UI.

Making Traces Actionable: Queries and Alerts

A trace backend without a query and alert strategy is just an expensive storage system. The most valuable queries for agent systems are latency breakdowns, cost per user session, error rate by agent role, and detection of outlier traces that indicate bugs.

-- Honeycomb-style query: find traces where total token cost exceeded $1
SELECT trace_id, SUM(gen_ai.usage.input_tokens * 3 + gen_ai.usage.output_tokens * 15) / 1e6 AS cost_usd
FROM spans
WHERE service.name = 'agentix-orchestrator'
  AND timestamp > NOW() - INTERVAL '1 hour'
GROUP BY trace_id
HAVING cost_usd > 1.0
ORDER BY cost_usd DESC
LIMIT 20;

-- Find agent roles with highest error rates
SELECT gen_ai.agent.role,
       COUNT(*) AS total_spans,
       SUM(CASE WHEN status = 'ERROR' THEN 1 ELSE 0 END) AS errors,
       errors / total_spans AS error_rate
FROM spans
WHERE timestamp > NOW() - INTERVAL '24 hours'
GROUP BY gen_ai.agent.role
HAVING error_rate > 0.01
ORDER BY error_rate DESC;