Agent Metrics That Actually Matter: Latency, Cost, and Success Rate

Why Agent Metrics Are Different from Traditional Application Metrics

Traditional web applications have well-established metric categories: request rate, error rate, and latency (the RED method), or utilization, saturation, and errors (the USE method). These frameworks work because traditional applications make deterministic decisions. The same input reliably produces the same output in the same time.

AI agent systems break the determinism assumption in ways that require additional metric categories. Latency varies by an order of magnitude depending on which model was selected, how many tools were invoked, and how many tokens the model decided to generate. Success is probabilistic. The same request might succeed 95% of the time and silently produce wrong output 5% of the time. Cost is dynamic. Two requests with identical user-visible complexity might have very different token costs depending on what the agent's internal reasoning produced.

This means agent systems need metrics across four dimensions: latency (how fast), cost (how expensive), success rate (how often it works), and quality (how well it works when it does). Each dimension requires different instrumentation and different alerting strategies.

Latency Metrics: Measuring Perceived Performance

Latency for AI agent systems must be measured at multiple granularities. Total end-to-end latency (user request to final response) is the most visible metric and the one users care about. But total latency alone does not tell you where time is being spent, which is what you need to optimize and debug.

Time to First Token (TTFT)

For streaming responses, the time from request receipt to the first token of the response is the primary user-perceived latency metric. Users experience TTFT as 'how long before the AI starts responding'. Optimizing TTFT often means reducing system prompt length, choosing a model with faster first-token latency, or moving expensive pre-processing out of the critical path. P50 TTFT should be under 1 second for a good user experience; P99 under 3 seconds.

Total Generation Time

Total generation time measures the full duration from request to complete response. For non-streaming use cases (structured output, API-to-API), this is the primary latency metric. Total generation time is highly variable in agent systems because it depends on how many tool calls the agent decided to make, how long each tool took, and how many output tokens the model generated. Track P50, P90, P95, and P99. The high percentiles reveal tail latency that your heaviest users experience.

Per-Component Latency Breakdown

Breaking down total latency into components, LLM inference time, tool execution time, orchestration overhead, tells you where to focus optimization. If LLM inference is 80% of latency, model routing and caching are the levers. If tool execution is dominant, parallelizing tool calls or caching tool results are the priorities. If orchestration overhead is significant, it usually signals inefficient context management or unnecessary agent round-trips.

from prometheus_client import Histogram, Counter, Gauge
import time

# Define metrics
llm_call_duration = Histogram(
    "agent_llm_call_duration_seconds",
    "Duration of LLM API calls",
    buckets=[0.1, 0.25, 0.5, 1, 2, 5, 10, 30],
    labelnames=["model", "agent_role", "task_type"],
)

tool_call_duration = Histogram(
    "agent_tool_call_duration_seconds",
    "Duration of tool executions",
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5],
    labelnames=["tool_name", "agent_id", "status"],
)

request_duration = Histogram(
    "agent_request_duration_seconds",
    "Total end-to-end request duration",
    buckets=[0.5, 1, 2, 5, 10, 30, 60],
    labelnames=["agent_type", "task_type"],
)

# Usage in instrumented code
async def traced_llm_call(model: str, agent_role: str, task_type: str, **kwargs):
    start = time.monotonic()
    try:
        result = await llm_client.call(**kwargs)
        return result
    finally:
        duration = time.monotonic() - start
        llm_call_duration.labels(
            model=model,
            agent_role=agent_role,
            task_type=task_type,
        ).observe(duration)

Cost Metrics: Tracking Every Dollar

LLM cost is a first-class operational metric for agent systems. Unlike traditional infrastructure costs that are relatively stable and predictable, LLM costs scale directly with usage patterns that can change dramatically based on traffic spikes, new features, or changes in how the model uses its context. Tracking cost at multiple granularities enables both operational monitoring and product analytics.

Cost per Request

Cost per request is the fundamental cost metric. Track it as a histogram to understand the distribution. A mean of $0.05 per request masks a bimodal distribution where 90% cost $0.01 and 10% cost $0.41 (typical of complex vs. simple queries). Alert on the P99 cost per request exceeding a threshold, which often indicates a runaway context accumulation or an unusually complex query that warrants routing to a cheaper model.

Cost Attribution by Agent Role

In multi-agent pipelines, aggregate cost metrics hide which agent role is the primary cost driver. Tag every LLM call with the agent role that made it, and track cost per request segmented by role. Typically you will find that the orchestrator and planning agents, which make complex, context-heavy calls, consume 60 to 80% of total pipeline cost despite handling simpler tasks.

from prometheus_client import Counter

# Token usage counters for cost calculation
tokens_consumed = Counter(
    "agent_tokens_total",
    "Total tokens consumed",
    labelnames=["model", "agent_role", "token_type"],  # token_type: input or output
)

# Cost counter in microdollars (avoids floating point precision issues)
cost_microdollars = Counter(
    "agent_cost_microdollars_total",
    "Total cost in microdollars",
    labelnames=["model", "agent_role", "task_type"],
)

PRICING = {  # per million tokens
    "claude-opus-4-5":   {"input": 15.0,  "output": 75.0},
    "claude-sonnet-4-5": {"input": 3.0,   "output": 15.0},
    "claude-haiku-4-5":  {"input": 0.25,  "output": 1.25},
    "gpt-4o":            {"input": 5.0,   "output": 15.0},
    "gpt-4o-mini":       {"input": 0.15,  "output": 0.6},
}

def record_llm_cost(
    model: str,
    agent_role: str,
    task_type: str,
    input_tokens: int,
    output_tokens: int,
):
    pricing = PRICING.get(model, {"input": 10.0, "output": 30.0})
    cost_usd = (
        input_tokens / 1_000_000 * pricing["input"] +
        output_tokens / 1_000_000 * pricing["output"]
    )
    tokens_consumed.labels(model=model, agent_role=agent_role, token_type="input").inc(input_tokens)
    tokens_consumed.labels(model=model, agent_role=agent_role, token_type="output").inc(output_tokens)
    cost_microdollars.labels(model=model, agent_role=agent_role, task_type=task_type).inc(cost_usd * 1_000_000)

Defining and Measuring Success Rate

Success rate for agent systems is harder to define than for traditional APIs. An HTTP 200 response does not mean the agent succeeded. It just means it returned something. Defining what constitutes a successful agent response requires making explicit the success criteria that the agent is trying to achieve.

Task Completion as the Primary Success Signal

The most meaningful success metric is task completion: did the agent accomplish what the user requested? This requires defining task completion for your specific use case. For a support agent, task completion might mean the user's issue was resolved without escalating to a human. For a research agent, it might mean all requested sources were found and synthesized. For a code agent, it might mean the generated code passes the test suite.

Proxy Metrics for Automatic Success Measurement

When direct task completion measurement requires human evaluation, proxy metrics that correlate with success are essential for automated monitoring. Common proxies: conversation continuation rate (did the user keep engaging?), explicit feedback signals (thumbs up/down), downstream event completion (did the user complete the purchase after the agent's recommendation?), format compliance rate (did the agent return the expected JSON schema?), and refusal rate (is the agent declining to help with in-scope tasks?).

Quality Metrics: Beyond Binary Success

Binary success rate treats all failures as equal, but not all failures are equal. An agent that confidently gives wrong information is far worse than one that correctly admits it does not know something. Quality metrics capture the spectrum between perfect success and catastrophic failure.

• Confidence calibration: When the agent expresses uncertainty, is it actually uncertain? Track whether stated confidence correlates with actual accuracy.
• Hallucination rate: What percentage of factual claims in agent outputs are verifiably false? Requires a verification pipeline but is the most important quality metric for factual use cases.
• Response coherence: Does the response make sense given the conversation history? Automated coherence scoring using an LLM judge can detect common failure modes.
• Format compliance rate: For structured outputs, what percentage conform to the expected schema without post-processing fixes?
• Appropriate refusal rate: Of requests the agent should decline, what percentage does it actually decline? And of requests it should answer, what percentage does it incorrectly refuse?

SLOs and SLAs for AI Agent Systems

Service Level Objectives (SLOs) define the target reliability of your agent system, expressed as a percentage of requests that meet defined quality thresholds over a time window. SLAs are contractual agreements with customers that typically commit to a subset of your SLOs. For AI agent systems, SLOs should cover latency, availability, and quality, not just uptime.

# SLO definitions for a production AI agent system
slos:
  - name: api_availability
    description: Percentage of requests that receive a non-error response
    target: 99.5%
    window: 30d
    indicator:
      metric: agent_requests_total{status!="error"} / agent_requests_total

  - name: latency_p95
    description: 95th percentile end-to-end response time
    target: 95%  # of requests complete within threshold
    threshold: 10s
    window: 7d

  - name: task_success_rate
    description: Percentage of tasks rated as successfully completed
    target: 94%
    window: 7d
    indicator:
      metric: agent_task_completions_total{outcome="success"} / agent_task_completions_total

  - name: cost_per_request
    description: P99 cost per request stays within budget
    target: 99%  # of requests cost less than threshold
    threshold_usd: 0.25
    window: 1d

Error Budget Management

An error budget is the amount of unreliability your SLO allows over the measurement window. If your SLO is 99% success rate over 30 days, your error budget is 1%, roughly 7.2 hours of complete downtime or a proportional amount of partial degradation. Error budget management is the practice of tracking how fast you are consuming this budget and using it to make decisions about release pace and risk tolerance.

When error budget is healthy (more than 50% remaining), you can deploy aggressively and experiment. When error budget is approaching exhaustion (less than 20% remaining), you should freeze non-critical deployments and focus on reliability work. When error budget is exhausted, you are in SLO breach. Customer-visible impact is occurring at a rate that violates your commitments.

Instrumenting Metrics with Prometheus

Prometheus is the de facto standard for metrics collection in production systems. It uses a pull model where a Prometheus server scrapes metric endpoints exposed by your agent services. The prometheus_client library provides Python bindings for the four core metric types: Counter, Gauge, Histogram, and Summary.

from prometheus_client import start_http_server, REGISTRY
from prometheus_client import Counter, Histogram, Gauge
import asyncio

# Expose metrics endpoint
def start_metrics_server(port: int = 9090):
    start_http_server(port)

# Full set of agent metrics
METRICS = {
    "requests_total": Counter(
        "agent_requests_total", "Total requests processed",
        ["agent_type", "task_type", "status"],
    ),
    "request_duration": Histogram(
        "agent_request_duration_seconds", "Request duration",
        ["agent_type", "task_type"],
        buckets=[0.5, 1, 2, 5, 10, 30, 60],
    ),
    "llm_calls_total": Counter(
        "agent_llm_calls_total", "Total LLM API calls",
        ["model", "agent_role", "status"],
    ),
    "llm_tokens_total": Counter(
        "agent_llm_tokens_total", "Total LLM tokens consumed",
        ["model", "agent_role", "token_type"],
    ),
    "active_sessions": Gauge(
        "agent_active_sessions", "Currently active agent sessions",
        ["agent_type"],
    ),
    "fallback_activations": Counter(
        "agent_fallback_activations_total", "Model fallback activations",
        ["from_model", "to_model", "reason"],
    ),
}

Dashboards Worth Building on Day One

The most valuable dashboards for agent systems support two primary operational activities: detecting problems quickly (operational dashboard) and understanding system behavior over time (analytical dashboard). Start with these two before building anything more specialized.

Operational Dashboard: Is Everything Working Now?

• Request rate and error rate (last 30 minutes) with threshold lines showing normal range.
• P50 and P99 end-to-end latency trend (last 1 hour) with SLO threshold line.
• Active sessions gauge with capacity warning threshold.
• LLM call error rate by model (last 30 minutes), catches provider degradation before it cascades.
• Circuit breaker state for each provider: CLOSED (green), HALF-OPEN (yellow), OPEN (red).
• Current error budget consumption percentage with time remaining in SLO window.

Analytical Dashboard: What Is Happening Over Time?

• Cost per request over 7 days by model tier, shows routing efficiency trends.
• Success rate by task type over 7 days, identifies which task categories are degrading.
• Token consumption by agent role over 30 days, shows context growth trends.
• Fallback activation rate over 30 days, shows provider reliability trends.
• P95 latency by model over 30 days, shows model performance regressions after provider updates.

Alert Configuration and Escalation

Alert fatigue is a real risk in over-monitored systems. Every alert that fires without actionable information is an alert that trains engineers to ignore alerts. Define alert thresholds based on real operational significance. The threshold that, if crossed, genuinely requires someone to take action within the stated urgency window.

# Prometheus alerting rules for agent systems
groups:
  - name: agent_critical
    rules:
      - alert: AgentHighErrorRate
        expr: rate(agent_requests_total{status="error"}[5m]) / rate(agent_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Agent error rate above 5% for 2+ minutes"

      - alert: AgentProviderDown
        expr: agent_circuit_breaker_state == 2  # OPEN state
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "LLM provider circuit breaker open"

  - name: agent_warning
    rules:
      - alert: AgentHighLatency
        expr: histogram_quantile(0.99, rate(agent_request_duration_seconds_bucket[10m])) > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Agent P99 latency above 30s"

      - alert: AgentErrorBudgetLow
        expr: agent_error_budget_remaining_pct < 20
        labels:
          severity: warning
        annotations:
          summary: "Error budget below 20% with {{ $value | humanizeDuration }} remaining in window"