Visualizing Agent Execution Graphs for Faster Debugging

Traces vs Graphs: What Each Reveals

Distributed traces and execution graphs answer different questions. A trace is a timeline. It shows you when each operation started, how long it ran, and whether it succeeded or failed. A trace is excellent for answering questions about performance: where is time being spent, what is the latency of each component, and did an error occur in this specific request.

An execution graph is a directed acyclic graph (DAG) that shows the causal structure of the agent's work, which agents were spawned, which decisions led to which tool calls, and which data dependencies caused sequential versus parallel execution. Graphs answer structural questions: why did this request spawn five agents instead of two, why did the synthesis step wait when the research could have been done in parallel, and what is the typical shape of a successful versus failed request.

When to Use Each

Use traces when you are investigating a specific request. A user reported it was slow or returned wrong output. The timeline view of a trace is the fastest way to identify which span was slow or errored. Use execution graphs when you are investigating a pattern. Why are some requests consistently more expensive than others, or why does the agent sometimes spawn more agents than expected. Graphs show structural patterns across many requests that individual traces do not surface.

Anatomy of an Agent Execution Graph

An execution graph for a multi-agent request is a directed acyclic graph where nodes represent agent invocations, LLM calls, and tool executions. Edges represent causal dependencies. An edge from node A to node B means that B was initiated as a result of A's output. Node properties capture the operational data needed for analysis and debugging.

Node Properties

• node_id: Unique identifier for this node in the graph. Correlates to span_id in the trace.
• node_type: 'agent', 'llm_call', 'tool_call', 'decision', or 'output'. Determines rendering style.
• agent_role: For agent and LLM call nodes, the role of the agent that created this node.
• duration_ms: How long this node's operation took. Displayed as node weight in analysis views.
• token_cost: For LLM call nodes, the combined token cost of this specific call.
• status: 'success', 'error', 'timeout', 'cancelled'. Determines node color in the visualization.
• depth: Distance from the root node. Useful for identifying unexpectedly deep agent recursion.

Edge Properties

• edge_type: 'spawns' (agent creates sub-agent), 'calls' (agent invokes tool), 'uses_output' (node uses result of parent).
• data_passed: What data flowed along this edge. Critical for debugging incorrect outputs, trace the data from root to the failing node.
• is_conditional: Whether this edge was created based on a dynamic decision. Conditional edges are key to understanding non-deterministic execution paths.
• wait_time_ms: Time the child node waited before the parent's output was available. Large wait times in parallel graphs indicate suboptimal sequencing.

Building the Execution Graph from Traces

If your agent system already emits proper distributed traces with parent-child span relationships, you can reconstruct the execution graph from trace data. The span hierarchy in a trace directly encodes the DAG structure: a span's parent_span_id tells you which node in the graph spawned it.

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class GraphNode:
    node_id: str
    node_type: str  # agent, llm_call, tool_call, output
    label: str
    duration_ms: float
    status: str
    token_cost: float = 0.0
    depth: int = 0
    children: list["GraphNode"] = field(default_factory=list)

def build_execution_graph(spans: list[dict]) -> GraphNode:
    # Index spans by span_id for fast lookup
    span_map = {s["span_id"]: s for s in spans}
    children_map: dict[str, list[str]] = {}

    for span in spans:
        parent_id = span.get("parent_span_id")
        if parent_id:
            children_map.setdefault(parent_id, []).append(span["span_id"])

    # Find root span (no parent)
    root_span = next(s for s in spans if not s.get("parent_span_id"))

    def build_node(span: dict, depth: int = 0) -> GraphNode:
        node = GraphNode(
            node_id=span["span_id"],
            node_type=classify_span(span),
            label=span.get("gen_ai.tool.name") or span.get("gen_ai.request.model") or span["name"],
            duration_ms=span["duration_ms"],
            status=span["status"],
            token_cost=calculate_token_cost(span),
            depth=depth,
        )
        for child_id in children_map.get(span["span_id"], []):
            child_node = build_node(span_map[child_id], depth + 1)
            node.children.append(child_node)
        return node

    return build_node(root_span)

def classify_span(span: dict) -> str:
    name = span.get("name", "")
    if name.startswith("agent."):
        return "agent"
    if name.startswith("llm."):
        return "llm_call"
    if name.startswith("tool."):
        return "tool_call"
    return "other"

Visualizing Parallel and Conditional Execution

The most valuable patterns to visualize in execution graphs are parallel fan-out (multiple agents running concurrently), sequential bottlenecks (operations that should be parallel but are running sequentially), conditional branches (different execution paths based on agent decisions), and loops (agent re-invocations that indicate retry or multi-step reasoning).

Detecting Unnecessary Sequential Execution

A common performance anti-pattern is sequential execution of operations that have no data dependency on each other. If the orchestrator spawns a research agent and then waits for it to complete before spawning a classification agent, but the classification agent does not need the research agent's output, you have an unnecessary sequential bottleneck. The execution graph makes this immediately visible: nodes at the same depth with no edge between them are parallel candidates.

def find_parallelization_opportunities(graph: GraphNode) -> list[dict]:
    opportunities = []

    def analyze_node(node: GraphNode):
        if len(node.children) < 2:
            for child in node.children:
                analyze_node(child)
            return

        # Check pairs of sibling nodes for data independence
        for i, child_a in enumerate(node.children):
            for child_b in node.children[i+1:]:
                if not has_data_dependency(child_a, child_b):
                    total_sequential_cost = child_a.duration_ms + child_b.duration_ms
                    parallel_savings = min(child_a.duration_ms, child_b.duration_ms)
                    if parallel_savings > 500:  # More than 500ms savings
                        opportunities.append({
                            "parent": node.label,
                            "node_a": child_a.label,
                            "node_b": child_b.label,
                            "savings_ms": parallel_savings,
                        })
        for child in node.children:
            analyze_node(child)

    analyze_node(graph)
    return sorted(opportunities, key=lambda x: x["savings_ms"], reverse=True)

def has_data_dependency(node_a: GraphNode, node_b: GraphNode) -> bool:
    # Check if node_b uses any output from node_a
    # In practice, this requires tracking data flow through edge annotations
    return False  # Simplified, implement with edge data tracking

Graph-Level Metrics for Optimization

Aggregating metrics across many execution graphs reveals system-level patterns that single-trace analysis cannot surface. These graph-level metrics are the most powerful tools for identifying structural optimization opportunities.

• Average graph depth: The average depth of the deepest leaf node across all graphs. Increasing depth over time signals agent delegation chains growing longer than designed.
• Average fan-out at each depth: How many children each node spawns on average at each level. Sudden changes in fan-out indicate behavior drift.
• Critical path length vs. parallelizable path length: What percentage of total latency is on the critical path (unavoidable sequential steps) versus parallelizable work. High critical path ratios indicate architectural bottlenecks.
• Graph shape frequency: How often each distinct graph topology appears. A system with 50 different execution patterns is harder to optimize than one with 5 dominant patterns.
• Node failure position: At what depth and in what node type do failures most commonly occur? This guides where to invest in reliability improvements.

Rendering Execution Graphs in Your UI

For production tools, rendering execution graphs requires a library that handles DAG layout algorithms. The most common choices are D3.js with a custom DAG renderer, React Flow (excellent React integration, handles large graphs), Cytoscape.js (most mature graph visualization library with extensive layout algorithms), and Mermaid (for simple embedded diagrams without interactivity).

// React Flow graph renderer for agent execution graphs
import ReactFlow, { Node, Edge, Background } from "reactflow";

interface ExecutionNode {
  node_id: string;
  label: string;
  node_type: "agent" | "llm_call" | "tool_call" | "output";
  status: "success" | "error" | "timeout";
  duration_ms: number;
}

const NODE_COLORS = {
  agent: "#6366f1",
  llm_call: "#0ea5e9",
  tool_call: "#10b981",
  output: "#f59e0b",
};

function convertToReactFlow(executionNodes: ExecutionNode[], edges: [string, string][]) {
  const nodes: Node[] = executionNodes.map((n, i) => ({
    id: n.node_id,
    position: { x: 0, y: 0 },  // Dagre layout will compute positions
    data: {
      label: (
        <div style={{ textAlign: "center" }}>
          <div style={{ fontWeight: 700, fontSize: 11 }}>{n.label}</div>
          <div style={{ fontSize: 9, color: "#64748b" }}>{n.duration_ms}ms</div>
        </div>
      ),
    },
    style: {
      background: NODE_COLORS[n.node_type],
      color: "#fff",
      border: n.status === "error" ? "2px solid #ef4444" : "none",
      borderRadius: 8,
      padding: "6px 12px",
    },
  }));

  const rfEdges: Edge[] = edges.map(([source, target], i) => ({
    id: `e-${source}-${target}`,
    source,
    target,
    type: "smoothstep",
    animated: false,
  }));

  return { nodes, edges: rfEdges };
}

Debugging Workflows Using Execution Graphs

The most common debugging workflows using execution graphs fall into three categories: root cause analysis for unexpected outputs, performance debugging for slow requests, and anomaly detection for requests that have an unusual structure compared to normal.

Root Cause Analysis: Tracing Wrong Outputs to Source

When an agent produces incorrect output, the execution graph lets you walk backwards from the output node through its parent chain to find where incorrect data entered the pipeline. Start at the output node, check its inputs, trace each input to its source node, and compare the data at each step against expected values. This data flow tracing is much faster through a graph visualization than through raw log records.

Performance Debugging: Finding the Bottleneck

Color-code execution graph nodes by duration. Red for nodes that took longer than their P90 baseline, yellow for nodes between P50 and P90, green for fast nodes. This heat map view immediately reveals which node in the graph is the latency bottleneck without requiring you to mentally parse a flat list of span durations.

Storing and Querying Historical Graphs

Storing execution graphs as structured data rather than just as visual snapshots enables powerful historical queries: find all requests that followed the five-agent topology, compare graph metrics before and after a deployment, or compute the most common failure paths across a week of production traffic.

# Store execution graph as serializable JSON
import json
from dataclasses import asdict

def serialize_graph(root: GraphNode, request_id: str, session_id: str) -> dict:
    def node_to_dict(node: GraphNode) -> dict:
        return {
            "node_id": node.node_id,
            "node_type": node.node_type,
            "label": node.label,
            "duration_ms": node.duration_ms,
            "status": node.status,
            "token_cost": node.token_cost,
            "depth": node.depth,
            "children": [node_to_dict(c) for c in node.children],
        }

    return {
        "request_id": request_id,
        "session_id": session_id,
        "graph": node_to_dict(root),
        "summary": {
            "total_nodes": count_nodes(root),
            "max_depth": max_depth(root),
            "total_duration_ms": root.duration_ms,
            "total_token_cost": sum_token_cost(root),
            "had_errors": any_errors(root),
        }
    }

# Store in ClickHouse for analytical queries
INSERT_QUERY = """
INSERT INTO agent_execution_graphs
(request_id, session_id, graph_json, total_nodes, max_depth,
 total_duration_ms, total_token_cost, had_errors, created_at)
VALUES (%(request_id)s, %(session_id)s, %(graph_json)s, ...)
"""