Model Routing in Production: When to Use GPT-4o, Claude Sonnet, or Haiku

Every agent system that uses more than one LLM provider eventually faces the routing problem: which model should handle which task? The naive answer is to always use the most capable model available. That answer is wrong at scale, and wrong in ways that matter. A frontier model that costs fifty times more than a fast model, takes six times longer to respond, and achieves marginally better results on tasks that didn't need frontier-level capability is a tax on every request, paid in dollars and in milliseconds.

Good model routing is one of the highest-ROI engineering investments you can make in a production agent system. Teams that implement intelligent routing consistently see 60 to 80 percent reductions in LLM costs without measurable quality degradation, because most tasks in most systems simply don't need the most powerful model available.

The Routing Problem

The routing problem has three components that need to be solved simultaneously: which model produces adequate quality for this specific task, which of those adequate models has the best cost-latency tradeoff for this context, and how do you make that determination at the speed required to not add meaningful latency to your pipeline.

The key word is adequate, not optimal. Routing is not about finding the absolute best model for every task. It's about finding the cheapest and fastest model that clears the quality bar for each specific task. A customer support chat agent that routes all requests to Claude Opus because it theoretically produces the best responses is over-engineering. Haiku or Sonnet handles 80 percent of support tasks with no perceptible quality difference, at a fraction of the cost.

The Current Model Landscape

Understanding the practical differences between model tiers is a prerequisite for routing decisions. The differences are not just about benchmark scores. They manifest differently for different task types in real production conditions.

The practical takeaway from this matrix is that the cost differences are dramatic and the capability differences are task-dependent. For classification, extraction, and simple generation tasks, the performance gap between Haiku and Sonnet is negligible. For multi-step reasoning, complex coding, and tasks requiring deep contextual understanding, the gap between Sonnet and Opus is meaningful.

Task Classification: The Foundation of Routing

Every routing decision starts with task classification. Before you can route to the right model, you need to understand what kind of task you're routing. The classification happens before the main LLM call, either through a lightweight model, a heuristic classifier, or a combination of both.

The fastest classification uses heuristics: task length, presence of specific keywords, API endpoint called, user tier, and historical task outcomes for similar requests. Heuristic classification adds sub-millisecond latency and covers the majority of routing decisions correctly. A lightweight LLM classifier (using a cheap model to classify before routing) is appropriate for ambiguous cases.

The Six Routing Dimensions

A complete routing framework evaluates tasks across six dimensions simultaneously. Each dimension produces a signal, and those signals combine to determine the appropriate model tier.

• Complexity score (0.0 to 1.0): a composite measure of reasoning depth required, number of steps involved, ambiguity of the request, and specificity of domain knowledge needed. High complexity routes to frontier models.
• Task type: reasoning, generation, classification, extraction, code, summarization, and translation each have different model strengths. Code tasks may route to a specialized code model even if complexity is moderate.
• Context length: tasks with very long contexts may be forced to specific models based on context window availability, regardless of other signals.
• Latency SLO: if a user-facing interaction requires a response in under 500ms, frontier models are disqualified regardless of task complexity. Fast models with streaming win on latency-critical paths.
• Cost budget: per-request cost budgets set by user tier or product feature can hard-cap model selection. Enterprise users may have frontier model budgets; free-tier users route to fast models by default.
• Quality requirement: some tasks have explicitly set quality requirements by the application layer. A document review for a legal firm routes to the highest tier regardless of complexity score.

Routing Architecture Design

The routing layer sits between the task submission and the model call. It's a fast, stateless component that takes a task descriptor and returns a model selection. The architecture should be designed so that the routing decision itself adds no more than 5 to 10 milliseconds to the request path.

Building the Task Classifier

The classifier is the heart of the routing system. It takes a raw task and produces the signals the routing logic needs. A well-designed classifier is fast, composable, and produces consistent scores across similar tasks.

from dataclasses import dataclass
from typing import Optional
import re

@dataclass
class TaskSignals:
    complexity_score: float       # 0.0 = trivial, 1.0 = frontier required
    task_type: str                 # "reason" | "gen" | "code" | "extract" | "classify"
    estimated_tokens: int          # input + expected output
    latency_budget_ms: int         # from request context
    cost_budget_usd: Optional[float]
    quality_tier: str              # "standard" | "high" | "critical"

class TaskClassifier:
    # Patterns that increase complexity score
    HIGH_COMPLEXITY_PATTERNS = [
        r"compare|analyze|evaluate|critique|synthesize",
        r"explain why|reasoning|because|therefore",
        r"multi-step|plan|strategy|approach",
        r"nuanced|subtle|edge case|exception",
    ]
    CODE_INDICATORS = [r"code|function|class|debug|refactor|implement|algorithm"]
    SIMPLE_PATTERNS = [
        r"summarize in (one|1|a) sentence",
        r"^what is the|^define|^list d+",
        r"yes or no|true or false",
        r"translate to|convert from",
    ]

    def classify(self, task: str, context: dict) -> TaskSignals:
        task_lower = task.lower()
        complexity = self._score_complexity(task_lower)
        task_type = self._detect_type(task_lower)
        tokens = self._estimate_tokens(task, context.get("system_prompt", ""))

        return TaskSignals(
            complexity_score=complexity,
            task_type=task_type,
            estimated_tokens=tokens,
            latency_budget_ms=context.get("latency_budget_ms", 5000),
            cost_budget_usd=context.get("cost_budget_usd"),
            quality_tier=context.get("quality_tier", "standard"),
        )

    def _score_complexity(self, task: str) -> float:
        score = 0.3  # base score
        for pattern in self.HIGH_COMPLEXITY_PATTERNS:
            if re.search(pattern, task):
                score += 0.15
        for pattern in self.SIMPLE_PATTERNS:
            if re.search(pattern, task):
                score -= 0.15
        # Long tasks tend to be more complex
        word_count = len(task.split())
        if word_count > 200:
            score += 0.2
        elif word_count < 20:
            score -= 0.1
        return max(0.0, min(1.0, score))

    def _detect_type(self, task: str) -> str:
        for pattern in self.CODE_INDICATORS:
            if re.search(pattern, task):
                return "code"
        if re.search(r"classify|categorize|label|tag", task):
            return "classify"
        if re.search(r"extract|find|identify|list", task):
            return "extract"
        if re.search(r"summarize|summarise|tldr|brief", task):
            return "summarize"
        return "gen"  # default: text generation

    def _estimate_tokens(self, task: str, system_prompt: str) -> int:
        # Rough estimate: 4 chars per token
        input_tokens = (len(task) + len(system_prompt)) // 4
        # Estimated output: 2x input for generation, 0.3x for extraction
        return int(input_tokens * 1.5)

Routing Rules in Practice

Once you have task signals, routing rules map those signals to model selections. Start with simple rules and add complexity only where measurement shows it is needed. Premature routing sophistication creates maintenance debt without proportional benefit.

from dataclasses import dataclass

@dataclass
class ModelSelection:
    provider: str
    model: str
    max_tokens: int
    routing_reason: str

class ModelRouter:
    def route(self, signals: TaskSignals) -> ModelSelection:
        # Hard constraints first, these override all other logic
        if signals.latency_budget_ms < 500:
            return ModelSelection("anthropic", "claude-haiku-4-5-20251001", 1024,
                "latency_budget_hard_constraint")
        if signals.quality_tier == "critical":
            return ModelSelection("anthropic", "claude-opus-4-7", 8096,
                "quality_tier_critical")

        # Code tasks get specialized routing
        if signals.task_type == "code" and signals.complexity_score > 0.5:
            return ModelSelection("anthropic", "claude-sonnet-4-6", 8096,
                "code_task_medium_plus")

        # Complexity-based routing for general tasks
        if signals.complexity_score >= 0.75:
            return ModelSelection("anthropic", "claude-opus-4-7", 8096,
                f"complexity_{signals.complexity_score:.2f}_high")

        if signals.complexity_score >= 0.45:
            # Check if cost budget allows Sonnet
            estimated_cost = (signals.estimated_tokens / 1_000_000) * 3.0
            if signals.cost_budget_usd and estimated_cost > signals.cost_budget_usd:
                return ModelSelection("anthropic", "claude-haiku-4-5-20251001", 2048,
                    "cost_budget_exceeded_downgrade")
            return ModelSelection("anthropic", "claude-sonnet-4-6", 4096,
                f"complexity_{signals.complexity_score:.2f}_medium")

        # Simple tasks: use cheapest capable model
        if signals.task_type in ("classify", "extract"):
            return ModelSelection("anthropic", "claude-haiku-4-5-20251001", 1024,
                "simple_classification_extraction")

        return ModelSelection("anthropic", "claude-haiku-4-5-20251001", 2048,
            f"complexity_{signals.complexity_score:.2f}_low")

Measuring Routing Quality

A routing system is only as good as your ability to measure whether it's working. Three metrics matter most: routing accuracy (are you selecting the right tier?), quality delta (is routed-down quality actually acceptable?), and cost efficiency (how much are you saving versus a single-model baseline?).

Routing accuracy is measured by comparing routing decisions to outcome quality. When a task is routed to Haiku and produces a poor result, that's a routing miss. Collect these misses and use them to tune your complexity thresholds. A 5 percent miss rate is acceptable. Above 10 percent means your classifier needs work.

Setting Up Quality Measurement

For automated quality measurement, run a parallel evaluation pipeline. For a sample of tasks, route to the selected model AND to the next tier up. Compare the outputs using an LLM judge. When the cheaper model's output passes the quality bar, the routing decision was correct. When it fails, you have a classifier miss to learn from.

Common Routing Mistakes

• Routing by task type alone without complexity scoring. A simple question about a technical topic may be trivial. A complex question about a simple topic may require deep reasoning. Type alone is insufficient.
• Setting complexity thresholds too conservatively. Defaulting to high thresholds to avoid quality misses defeats the purpose of routing. Set thresholds based on measured acceptable miss rates, not fear.
• Not logging routing decisions. Without a log of what was routed where and why, you cannot improve your classifier. Every routing decision should be logged with the full signal set.
• Ignoring latency in routing decisions. A complex task with a 200ms latency budget cannot go to a frontier model even if it needs one. Design your application to surface these constraints to the router.
• Treating routing as a one-time configuration. User behavior, model capabilities, and pricing all change. Routing rules need quarterly reviews against current data.

Iterating on Your Routing Logic

The first version of your routing logic should be deliberately simple: one or two thresholds based on complexity score. The complexity score is your primary signal. Start there, measure extensively for two to four weeks, and use the data to identify where the simple rules fail. Only then add dimension-specific rules.

Conclusion

Model routing is one of the few engineering investments in LLM systems where the ROI is both significant and immediate. A well-implemented router that correctly classifies 90 percent of tasks can reduce your LLM spend by 50 to 70 percent while maintaining output quality that users cannot distinguish from the always-frontier approach. The implementation complexity is real but manageable, and the payoff compounds with every request at scale.