Retry, Timeout, and Circuit Breaker Patterns for Agentic Pipelines

Failure Taxonomy for Agents

The first step in building reliable agents is classifying failures correctly. The wrong classification leads to the wrong response, retrying a permanent error wastes time and money; not retrying a transient error creates unnecessary failures. Every agent failure belongs to one of three categories, and the classification determines everything that follows.

Transient Failures

Transient failures are temporary conditions that will resolve on their own. The service is up but momentarily overloaded (HTTP 429 rate limit), briefly unavailable (HTTP 503), or the network is congested (timeout). The correct response is to wait and retry. The key insight: the exact same request, sent a few seconds later, will likely succeed.

Client Errors

Client errors indicate that the request itself is malformed. HTTP 400 Bad Request, 422 Unprocessable Entity, invalid JSON schema, expired authentication tokens, these failures will recur on every retry until the request is fixed. Retrying client errors is a bug, not a feature. The correct response is to fix the request and alert the developer.

Permanent Failures

Permanent failures indicate that the requested operation is not possible. HTTP 404 Not Found, 410 Gone, quota permanently exhausted. No amount of waiting or request modification will make these succeed. The correct response is to abort, use a fallback path, and notify stakeholders.

Retry Fundamentals and Backoff Strategies

Retrying without backoff creates retry storms: when a service is overloaded and all clients retry immediately, the service receives a thundering herd of requests that overwhelms it further. Backoff, increasing the wait between retries, gives the service time to recover. The most common strategy is exponential backoff: after the first failure, wait 1 second; after the second, 2 seconds; after the third, 4 seconds; and so on.

Retry Configuration Parameters

• max_retries: Maximum number of attempts before giving up. For LLM calls, 3 retries (4 total attempts) is a sensible default. More than 4 retries suggests a structural problem, not a transient condition.
• base_delay: The initial wait time before the first retry. Typically 1 second for LLM APIs, shorter for fast internal services.
• max_delay: Cap on the maximum wait time regardless of exponential growth. Typically 32–60 seconds. Without a cap, exponential growth eventually produces minute-long waits.
• backoff_multiplier: The factor by which the delay grows. 2.0 (doubling) is standard. Values above 3.0 produce very long waits quickly.
• retry_on: The set of error codes that trigger a retry. Must be carefully defined, only include genuinely transient errors.

Jitter and Thundering Herd Prevention

In a system where many agent instances retry simultaneously, a common scenario in multi-agent pipelines, synchronized retries can cause retry storms as severe as the original failure. Jitter, randomizing the wait time, desynchronizes retries and prevents thundering herds.

Jitter Strategies Compared

• Full jitter: Wait = random(0, base × 2^attempt). Completely randomizes the wait within the exponential window. Best for preventing thundering herds across many clients.
• Equal jitter: Wait = (base × 2^attempt / 2) + random(0, base × 2^attempt / 2). Guarantees minimum wait while still adding randomness. Good balance of predictability and storm prevention.
• Decorrelated jitter: Wait = min(cap, random(base, previous_wait × 3)). Each wait is derived from the previous, not the base formula. Most effective at keeping the aggregate retry load smooth.
• No jitter: Wait = base × 2^attempt exactly. Simple and predictable but creates synchronized retries across multiple clients. Only acceptable for single-agent scenarios with no peer retriers.

Timeout Hierarchy for Agent Pipelines

Timeouts prevent operations from hanging indefinitely. In agentic pipelines, operations are nested: the agent session contains LLM calls, which contain tool calls, which contain network requests. Each level needs its own timeout, and inner timeouts must be shorter than their parent to prevent the parent from being decided by a child operation.

Timeout Design Principles

• Set timeouts based on p99 latency, not average: If your LLM call averages 2 seconds but has a p99 of 15 seconds, set the timeout at 20–25 seconds. Setting it at 3 seconds (close to average) generates false timeouts on legitimate slow calls.
• Agent session timeout > sum of component timeouts: The session timeout must be larger than the total time all components could take, including retries. Otherwise the session times out before components can complete their retry budget.
• Always time out, never wait forever: An agent with no timeout will eventually hang indefinitely on a hung API call, blocking the session and leaking resources. Every external call must have a timeout.
• Distinguish connection timeout from read timeout: Connection timeout (how long to wait for TCP handshake) should be 2–5 seconds. Read timeout (how long to wait for response data) should match the p99 of the operation.

The Circuit Breaker Pattern

The circuit breaker prevents a failing service from being called repeatedly when it's clearly down. Rather than every agent instance waiting for timeouts on a service that's been down for 10 minutes, the circuit breaker detects the pattern of failures and 'opens', immediately returning a failure response for all subsequent calls, without attempting the request.

Why Circuit Breakers Matter for Agents

LLM providers have occasional partial outages where the API remains reachable but responds slowly and unreliably. Without a circuit breaker, every agent call will wait for the full timeout (say, 60 seconds) before failing. In a multi-agent pipeline running 50 concurrent tasks, that means 3,000 seconds of wasted wait time per minute of downtime. With a circuit breaker open, those same calls fail in milliseconds and the agent can switch to its fallback model immediately.

Circuit Breaker State Transitions

• CLOSED → OPEN: Triggered when the failure count within a rolling time window exceeds the threshold. Example: 5 failures in the last 60 seconds.
• OPEN → HALF-OPEN: Triggered when a configurable recovery timeout expires. Example: after 30 seconds open, allow one test request.
• HALF-OPEN → CLOSED: Triggered when the probe request succeeds. The service is recovering, resume normal operation.
• HALF-OPEN → OPEN: Triggered when the probe request fails. The service is still down, stay open and wait another recovery timeout.

Circuit Breaker Configuration

Circuit breaker configuration parameters must be calibrated to your specific service's failure patterns. Misconfigured thresholds either open too eagerly (causing false outages) or too slowly (providing no protection during genuine outages).

Configuration Parameters

• failure_threshold: Number of failures required to open the circuit. Typical values: 5–10 failures in the monitoring window. Lower values are more sensitive but more prone to false trips.
• success_threshold: Number of successful calls in HALF-OPEN before fully closing. Typically 2–3. Prevents premature closure on a single lucky probe.
• timeout (recovery): Duration to stay OPEN before allowing a probe. Must be long enough for the service to recover. Typical: 30–120 seconds.
• monitoring_window: The rolling time window for counting failures. Typically 60 seconds. Shorter windows are more reactive; longer windows prevent triggering on brief spikes.
• expected_exception: The exception types that count as failures. Don't count business errors (validation failures), only count service errors (timeouts, 5xx responses).

Resilience Pattern Library

The Bulkhead Pattern

The bulkhead pattern isolates components into separate resource pools so that a failure in one doesn't exhaust resources for others. Named after the watertight compartments in a ship's hull, if one compartment floods, the ship stays afloat. In agent systems: separate thread pools for LLM calls vs tool calls, separate connection pools per external service, separate rate limit budgets per tenant.

When to Apply Each Pattern

• Retry: Transient failures, rate limits, brief unavailability, timeouts. Always combine with backoff and jitter.
• Timeout: Every external call, without exception. Prevents indefinite blocking.
• Circuit Breaker: Services with persistent outages or cascading failure risk. LLM providers, external APIs, databases.
• Bulkhead: Shared infrastructure where one tenant or component could starve others. Critical for multi-tenant agent platforms.
• Checkpoint: Long-running tasks where restart-from-scratch would be expensive. Any task taking more than 30 seconds.
• Fallback: Services with known alternatives. Always pair with Circuit Breaker to trigger cleanly.
• Idempotency: Any action with side effects that might be retried, emails, payments, record creation.

Combining Patterns Safely

Reliability patterns interact with each other, and combining them incorrectly creates new failure modes. The most common dangerous interaction is between retry and circuit breaker: if retries are configured with a timeout shorter than the circuit breaker monitoring window, individual requests time out before the circuit breaker can accumulate enough failures to open.

Safe Combination Rules

• Timeout before retry: Always set the timeout to fire before the retry exhausts its budget. If max_retries=3 and base_delay=1s, total retry time is ~7 seconds. The parent timeout should be at least 10 seconds.
• Circuit breaker wraps the retry: The circuit breaker should sit outside the retry logic. A retry loop that hits an open circuit breaker should immediately fail without attempting all retries.
• Fallback outside circuit breaker: The fallback is triggered when the circuit breaker is OPEN. It must not be subject to the same circuit breaker, use a separate circuit breaker instance per service.
• Idempotency before retry: Ensure all retriable operations are idempotent before adding retry logic. Adding retry to a non-idempotent operation creates duplicate side effects.

LLM-Specific Reliability Considerations

LLM APIs have unique failure characteristics that differ from typical REST APIs. Standard reliability patterns need adaptation for the specific failure modes of language model providers.

Context Window Exhaustion

When a request fails because the context is too long (context_length_exceeded), this is a permanent client error, retrying with the same context will always fail. The correct response is to compress the context and retry, not to simply retry. Build this as a special case in your retry logic: catch context_length_exceeded, trigger compression, then retry once with the compressed context.

Partial Response Streaming Failures

Streaming responses can be interrupted mid-generation. The first 500 tokens arrive successfully, then the connection drops. This creates a partial response that may look like a complete response if your parser doesn't track the stream's termination signal. Always track whether a streaming response completed normally, and treat partial responses as errors requiring retry.

Rate Limit Strategy by Provider

• Anthropic: Separate rate limits for tokens-per-minute (TPM) and requests-per-minute (RPM). Rate limit headers in responses indicate current usage. Implement proactive throttling when approaching limits rather than waiting for 429 errors.
• OpenAI: Similar TPM/RPM limits. Tier-based limits that increase with usage history. The Retry-After header specifies exact wait time on 429 responses.
• Cross-provider: Implement a global token budget that spans all providers. When Anthropic is rate limited, shift traffic to OpenAI rather than waiting.

Monitoring Reliability Metrics

Reliability patterns only work if you can observe their behavior. The metrics you track determine whether you can identify misconfigured patterns, detect degrading reliability before users notice, and understand the cost of your resilience infrastructure.

Key Reliability Metrics

• retry_rate: Fraction of requests requiring at least one retry. High values (> 5%) indicate persistent service instability. Spike detection on retry_rate is an early warning of outages.
• retry_success_rate: Of retried requests, the fraction that eventually succeeded. Low values indicate the underlying condition is not transient and retry configuration needs adjustment.
• circuit_breaker_open_duration: Total time each circuit breaker spent in OPEN state. Long durations indicate provider reliability issues worth escalating.
• timeout_rate: Fraction of requests that hit the timeout before completing. Helps calibrate timeout values, if timeout_rate is high, the timeout is too short for the actual p99 latency.
• fallback_activation_rate: How often fallback paths are used. High rates indicate primary paths are unreliable. Track fallback quality separately from primary path quality.