Cost Profiling and Optimization for Production Agent Systems

Anatomy of Agent System Costs

Agent system costs have a fundamentally different structure from traditional software costs. The primary cost driver is not compute or storage, it's API usage. Every LLM call, every tool API call, every vector database query generates a fee. These per-request costs are invisible at low volume and staggering at production scale. An agent that costs $0.05 per task seems cheap until you're running 100,000 tasks per month at $5,000.

Cost Components by Category

• LLM input tokens: Typically 40–50% of total cost. The sum of all tokens sent to the model: system prompt, conversation history, RAG chunks, tool schemas, and user messages.
• LLM output tokens: Typically 35–45% of total cost. Output tokens are priced 3–8x higher than input tokens on most models. Long reasoning chains and verbose responses are expensive.
• Tool and external API calls: Typically 5–15% of total cost. Web search APIs ($0.001–0.01 per search), specialized data APIs, third-party service calls.
• Embedding calls: Typically 1–5% for RAG-heavy agents. Embedding 10,000 chunks at $0.0001/token adds up for high-retrieval workloads.
• Infrastructure (compute, storage, network): Typically 5–10% for well-optimized systems. Often over-estimated relative to API costs.

Cost Attribution and Profiling

Cost attribution answers the question: 'Which agents, tasks, and users are responsible for which spend?' Without attribution, you can see your total bill but not where the money is going. You can't optimize what you can't measure.

Attribution Dimensions

• Per-agent attribution: Track token spend per agent type. A customer support agent and a research agent have very different cost profiles, optimizing the wrong one wastes effort.
• Per-tenant attribution: In multi-tenant systems, attribute costs to individual tenants. This enables cost-based pricing, per-tenant budgets, and identification of unprofitable usage patterns.
• Per-task attribution: Track cost per task type. Enables ROI analysis: is the value of completing a task worth its cost?
• Per-step attribution: Identify which steps in a pipeline are most expensive. Often reveals that one step (e.g., a complex multi-document analysis) accounts for 80% of per-task cost.
• Per-user attribution: For user-facing agents, attribute cost to individual users. Identifies heavy users who may need rate limiting or different pricing.

Token Counting Implementation

from dataclasses import dataclass, field
from datetime import datetime

# Model pricing per million tokens (USD)
PRICING: dict[str, dict[str, float]] = {
    "claude-opus-4-7": {"input": 15.0, "output": 75.0, "cache_read": 1.5},
    "claude-sonnet-4-6": {"input": 3.0, "output": 15.0, "cache_read": 0.3},
    "claude-haiku-4-5-20251001": {"input": 0.8, "output": 4.0, "cache_read": 0.08},
    "gpt-4o": {"input": 5.0, "output": 15.0, "cache_read": 2.5},
    "gpt-4o-mini": {"input": 0.15, "output": 0.6, "cache_read": 0.075},
}

@dataclass
class TokenUsageRecord:
    timestamp: str
    agent_id: str
    task_id: str
    tenant_id: str
    model: str
    step_name: str
    input_tokens: int
    output_tokens: int
    cache_read_tokens: int = 0
    tool_call_count: int = 0

    @property
    def total_cost_usd(self) -> float:
        p = PRICING.get(self.model, {"input": 5.0, "output": 15.0, "cache_read": 1.0})
        return (
            self.input_tokens * p["input"] / 1_000_000
            + self.output_tokens * p["output"] / 1_000_000
            + self.cache_read_tokens * p["cache_read"] / 1_000_000
        )

def record_llm_call(
    usage: dict,
    model: str,
    agent_id: str,
    task_id: str,
    tenant_id: str,
    step_name: str,
) -> TokenUsageRecord:
    record = TokenUsageRecord(
        timestamp=datetime.utcnow().isoformat(),
        agent_id=agent_id, task_id=task_id,
        tenant_id=tenant_id, model=model, step_name=step_name,
        input_tokens=usage.get("input_tokens", 0),
        output_tokens=usage.get("output_tokens", 0),
        cache_read_tokens=usage.get("cache_read_input_tokens", 0),
    )
    # Persist to your cost tracking store
    cost_store.write(record)
    return record

Cost Optimization Levers

Lever 1, Prompt Caching (45% potential savings)

Provider-level prompt caching reduces the cost of repeated system prompt tokens to 10% of standard price. For agents with 2,000-token system prompts running 10,000 requests/day, caching saves $270/day at GPT-4o pricing. Implementation effort: minimal, add cache_control markers to stable prompt sections.

Lever 2, Model Routing (35% potential savings)

Route simple tasks to cheap models and complex tasks to frontier models. A customer support agent that handles 80% simple FAQ queries and 20% complex issues can route simple queries to Haiku ($0.8/M input) and complex ones to Opus ($15/M input). The blended cost is dramatically lower than using Opus for everything. Classification adds ~$0.001 per routing decision, negligible versus the savings.

Lever 3, Context Pruning (28% potential savings)

Each turn of conversation history adds 400–1,000 tokens. At turn 15, that's 6,000–15,000 tokens of history in every LLM call. Context pruning compresses old history, removing it from the billable input token count. A 5-turn sliding window with compression reduces input tokens per call by 40–60% after the initial turns.

Model Cost Routing

Model routing is the practice of dynamically selecting the cheapest model that meets the quality requirements for each specific request. Done well, it's the most impactful single cost optimization, cost differences between models can be 50–100x.

Routing Decision Criteria

• Task complexity: Simple classification, formatting, or extraction → cheap model. Multi-hop reasoning, complex analysis, creative generation → frontier model.
• Output quality requirements: Internal processing steps with no user-facing output → cheap model. User-facing final responses → quality-appropriate model.
• Context length: Very long contexts may require specific models. But they also cost more, justify with quality requirements, not just capability.
• Latency requirements: Interactive user-facing → balance quality and speed. Background batch → optimize for quality and cost over speed.

Building a Cost-Aware Router

from dataclasses import dataclass
from enum import Enum

class TaskTier(Enum):
    SIMPLE = "simple"      # Classification, extraction, formatting
    MEDIUM = "medium"      # Summarization, Q&A with context
    COMPLEX = "complex"    # Multi-hop reasoning, analysis, planning

@dataclass
class ModelTierConfig:
    model: str
    cost_per_million_input: float
    cost_per_million_output: float
    max_context_tokens: int

TIER_MODELS = {
    TaskTier.SIMPLE: ModelTierConfig(
        "claude-haiku-4-5-20251001", 0.8, 4.0, 200_000
    ),
    TaskTier.MEDIUM: ModelTierConfig(
        "claude-sonnet-4-6", 3.0, 15.0, 200_000
    ),
    TaskTier.COMPLEX: ModelTierConfig(
        "claude-opus-4-7", 15.0, 75.0, 200_000
    ),
}

def classify_task_tier(task: dict) -> TaskTier:
    """Route to the cheapest model that can handle this task."""
    if task.get("requires_multi_hop_reasoning"):
        return TaskTier.COMPLEX
    if task.get("output_is_user_facing") and task.get("complexity_score", 0) > 0.7:
        return TaskTier.COMPLEX
    if task.get("requires_summarization") or len(task.get("context", "")) > 5000:
        return TaskTier.MEDIUM
    return TaskTier.SIMPLE

def select_model(task: dict) -> str:
    tier = classify_task_tier(task)
    return TIER_MODELS[tier].model

Prompt Optimization for Cost

The system prompt is the largest fixed cost per request. A 3,000-token system prompt adds $0.015 (GPT-4o) to every request before any user message is processed. Optimizing system prompt token count while preserving capability has compounding savings.

System Prompt Compression Techniques

• Remove redundancy: System prompts typically contain repeated emphases ('always', 'never', 'make sure to') that can be reduced without changing behavior. Audit for repetition.
• Use imperative style: 'Respond in JSON' instead of 'Please make sure that all of your responses are formatted as valid JSON objects'. Savings: 5–15 tokens per instruction.
• Eliminate examples when unnecessary: Few-shot examples consume 100–500 tokens each. Replace with compact format specifications where the model doesn't need examples to follow the format.
• Lazy tool loading: Don't include tool schemas for tools the agent won't need in the current task phase. Each tool schema is 100–400 tokens. Load only relevant tools.

Controlling Output Token Spend

Output tokens are typically 3–8x more expensive than input tokens. Controlling output verbosity has an outsized impact on total cost.

Output Length Control Strategies

• Explicit length instructions: 'Respond in 2–3 sentences' or 'Provide a structured JSON response with exactly these fields: [...]'. Models follow explicit length constraints reliably.
• Structured output reduces verbosity: JSON and structured formats are more token-efficient than prose explanations. A JSON response with 5 fields is typically 50–70% fewer tokens than a prose equivalent.
• Constrain max_tokens per task type: Set appropriate max_tokens for each task type rather than using a high universal default. Simple classification: 100 tokens. Summarization: 500 tokens. Analysis: 1,000 tokens.
• Remove chain-of-thought for simple tasks: CoT reasoning adds 200–500 output tokens per call. Disable it for simple tasks where the quality benefit doesn't justify the cost.

Identifying Cost Anomalies

Cost anomalies, sudden spikes or gradual increases in per-task cost, are often the first sign of a deeper problem: a prompt that grew by accident, a tool that started returning larger results, or a model that started generating longer responses after an update.

Anomaly Detection Signals

• Cost per task by type: Alert when average cost per task type increases by more than 20% week-over-week. A sudden jump usually has a specific cause.
• Input token distribution: Track the distribution of input token counts. Widening variance or rightward shift indicates context bloat.
• Output token distribution: Track output token count distribution. Rightward shift indicates the model started generating longer responses, often caused by prompt changes.
• Tool call frequency per task: Track average tool calls per task. Increasing values indicate the agent is making more calls than necessary, possibly stuck in a loop.
• Cache hit rate: Declining cache hit rate with unchanged query patterns suggests a cache configuration change is forcing unnecessary LLM calls.

Per-Agent Cost Budgets and Guardrails

Hard cost limits prevent runaway spend. A single malfunctioning agent stuck in a loop can generate hundreds of dollars of API costs in minutes without guardrails.

Budget Guardrail Implementation

• Per-task budget: Set a maximum spend per task (e.g., $0.50). If the task reaches this limit before completing, abort with a partial result and alert.
• Per-session token limit: Maximum tokens per agent session. When reached, force context compression and continue, or abort gracefully.
• Daily per-tenant budget: For multi-tenant systems, cap each tenant's daily spend. When a tenant's budget is reached, queue remaining tasks for the next day or require explicit override.
• Real-time spend monitoring: Track cumulative spend in real time and alert when 80% of any budget is consumed. This gives time to investigate before the limit is hit.

Measuring Cost vs Quality ROI

Cost optimization is only valuable if it doesn't degrade quality below the acceptable threshold. Every optimization decision should be accompanied by a quality measurement to verify the ROI.

ROI Measurement Framework

• Baseline quality measurement: Before any optimization, run your eval suite and record quality scores per task type. This is your baseline.
• Post-optimization quality check: After each optimization (model downgrade, context reduction, prompt compression), re-run the eval suite. Compare quality scores to baseline.
• Cost-per-quality-point: Calculate (cost reduction) / (quality reduction) for each optimization. Higher ratios are better, more cost saved per quality point sacrificed.
• Minimum quality floor: Define the minimum acceptable quality score for each task type. Any optimization that drops quality below this floor is rejected regardless of cost savings.

Cost Reporting and Forecasting

Essential Cost Reports

• Daily cost by agent type: Which agents are driving the most spend? Are costs tracking with task volume or growing faster?
• Weekly cost trend: 7-day rolling cost vs 4-week average. Deviations identify anomalies or organic growth.
• Cost per task by type: Absolute cost and week-over-week change. Helps identify tasks that have become unexpectedly expensive.
• Model distribution: Fraction of calls going to each model tier. Are routing decisions working as intended?
• Cache savings: How much is caching saving per day? Is this trending up (cache warming) or down (eviction issues)?
• Monthly forecast: Based on current growth rate, project next month's cost. Flag if projected cost exceeds budget.