Horizontal Scaling of Agent Fleets: Infrastructure and Architecture

Why Scaling Agents Differs from Scaling APIs

Stateless REST API scaling is well-understood: add more pods, put them behind a load balancer, done. Agent scaling is fundamentally different because agents are stateful: they maintain conversation history, tool session state, checkpoints, and in-flight reasoning chains. You can't route the second turn of a conversation to a different pod without coordinating the state transfer. The pod that handled turn 1 holds the session state that turn 2 needs.

Additionally, agents make external API calls (LLM providers, tool services) that have their own rate limits. Scaling from 10 agents to 100 agents doesn't just increase your compute, it increases your LLM API usage 10x, potentially hitting provider rate limits. Horizontal scaling must account for these external dependencies, not just internal compute resources.

The Three Scaling Dimensions for Agents

• Compute scaling: More pods to handle more concurrent agent sessions. Standard horizontal scaling, but requires state externalization.
• I/O scaling: More LLM API capacity, more tool API quota, more database connection pool capacity. Often the actual bottleneck at scale.
• State scaling: More capacity for the state layer (Redis, database) that holds session state, checkpoints, and shared context. Under-scaled state infrastructure causes cascading failures at high agent concurrency.

Agent Fleet Architecture

Stateless Worker Design

Worker pods must be designed as stateless compute units. All session state, conversation history, checkpoint data, working memory, is stored in the external state store (Redis or a database), not in the worker's memory. When a worker picks up a task from the queue, it loads the relevant session state, executes the next agent step, saves updated state, and releases the task. Any worker can handle any task for any session.

Task Queue as the Scaling Lever

The task queue decouples task submission from task execution, enabling independent scaling of producers and consumers. When load spikes, the queue absorbs the burst and workers drain it at their capacity. This prevents the fleet from being overwhelmed by sudden load and provides natural backpressure, clients that submit tasks at a rate exceeding processing capacity see queue depth increase, signaling that they should slow submission.

Autoscaling Patterns for Agent Workloads

Queue-Depth Based Autoscaling

For agent fleets, queue depth is a better autoscaling signal than CPU or memory utilization. Agents are often I/O-bound, CPU may be low while the agent waits for an LLM response. Scale based on messages per worker: when average queue depth per active worker exceeds your target (typically 5–10 messages per worker), add workers. When it drops below 1, remove workers.

Scale-Up vs Scale-Down Asymmetry

Scale up aggressively (add 50% capacity immediately when threshold is crossed) and scale down conservatively (wait 10–15 minutes below threshold before removing capacity). Aggressive scale-up prevents user-facing latency spikes during load increases. Conservative scale-down prevents the oscillation where scale-down causes another spike which triggers scale-up again.

Pre-Scaling for Predictable Patterns

If your agent workload has predictable daily or weekly patterns (e.g., peak traffic at 9am when users start work), use predictive pre-scaling rather than reactive scaling. Scale up 5–10 minutes before expected peak and scale down after the peak. This provides full capacity exactly when needed without the scale-up latency.

Stateful Agent Scaling Challenges

The hardest scaling challenge for agents is state. When you add a new worker, it needs access to all existing session states. When you remove a worker, its in-flight sessions need to complete or be handed off. When you update worker code, sessions in progress need to migrate gracefully.

State Externalization Patterns

• Redis for hot state: Active session context (last N turns, current step, working variables). Fast access, volatile. TTL of 24 hours. Workers read/write per-turn.
• PostgreSQL for durable state: Task definitions, completion records, user data, billing. Durable, ACID-compliant. Workers read on task start, write on task completion.
• S3/Blob for large artifacts: Checkpoints, large tool outputs, conversation archives. Cheaper than Redis at scale. Workers write checkpoints, read on recovery.
• No local state: Workers never store session state in memory beyond the current request. Pod restarts are seamless, the next request loads state from the external store.

Session Affinity, When and When Not to Use

Session affinity routes subsequent requests from the same session to the same pod. This is tempting because it avoids loading state on every request. But it creates scaling problems: hot pods with many active sessions become overloaded while cool pods with few sessions are underutilized. The right architecture externalizes all state and routes requests to the least-loaded available pod.

Queue-Based Load Leveling

Priority Queue Design

Implement separate queues per priority tier: HIGH (interactive user requests), MEDIUM (background processing), LOW (batch jobs). Workers poll queues in priority order. When the HIGH queue is empty, workers drain MEDIUM. This ensures user-facing interactive requests are always processed promptly even when the system is under heavy batch load.

Multi-Tenant Isolation at Scale

When serving multiple tenants from the same fleet, isolation becomes critical at scale. A noisy tenant consuming excessive resources must not degrade service for other tenants.

Isolation Mechanisms

• Per-tenant rate limiting: Each tenant has its own token bucket at the queue level. A tenant that submits 1,000 tasks in one second is throttled to their allocated rate rather than flooding the shared queue.
• Per-tenant resource quotas: Maximum concurrent tasks, maximum tokens per hour, maximum tool calls per minute. Enforced at both the queue admission level and the worker execution level.
• Dedicated worker pools: For enterprise tenants, provision dedicated worker pods isolated from the shared pool. Higher cost but guarantees resource isolation and compliance.
• Tenant-scoped state namespacing: All state store keys are prefixed with tenant ID. Workers never query cross-tenant state. Data isolation is enforced at the key level.

Kubernetes Deployment Patterns

Recommended Kubernetes Configuration

• Horizontal Pod Autoscaler (HPA): Scale based on custom metrics (queue depth) using the Kubernetes External Metrics API or KEDA (Kubernetes Event-Driven Autoscaling).
• Pod Disruption Budgets: Set minAvailable=50% to ensure scale-down operations never remove too many pods simultaneously, preventing capacity gaps.
• Graceful shutdown: Configure terminationGracePeriodSeconds=120 to allow in-flight agent tasks to complete before pod termination. Signal the pod to stop accepting new tasks when SIGTERM is received.
• Resource requests vs limits: Set CPU requests to expected average usage. Set CPU limits to 2x requests. Never set memory limits lower than peak usage, OOM kills during peak load are disruptive.
• Node affinity: Run agent workers on nodes with fast local SSD for checkpoint I/O. Separate from API/web server nodes to prevent resource contention.

Rate Limit Coordination Across Fleet

With 20 worker pods each making LLM calls, rate limit management becomes a distributed coordination problem. A per-worker rate limiter is insufficient, 20 workers each allowed 50 RPM would attempt 1,000 RPM against a 100 RPM provider limit.

Centralized Rate Limit Implementation

import redis.asyncio as aioredis
import time

class DistributedRateLimiter:
    """Fleet-wide rate limiter backed by Redis sliding window."""

    def __init__(self, redis_url: str, provider: str, limit_per_minute: int):
        self.redis = aioredis.from_url(redis_url)
        self.key = f"ratelimit:{provider}"
        self.limit = limit_per_minute
        self.window = 60  # seconds

    async def acquire(self) -> bool:
        """Returns True if request is allowed, False if rate limited."""
        now = time.time()
        window_start = now - self.window
        async with self.redis.pipeline() as pipe:
            # Sliding window counter
            pipe.zremrangebyscore(self.key, 0, window_start)
            pipe.zadd(self.key, {str(now): now})
            pipe.zcard(self.key)
            pipe.expire(self.key, self.window * 2)
            results = await pipe.execute()
        current_count = results[2]
        return current_count <= self.limit

    async def wait_and_acquire(self, max_wait: float = 60.0) -> None:
        start = time.monotonic()
        while True:
            if await self.acquire():
                return
            if time.monotonic() - start > max_wait:
                raise TimeoutError("Rate limit wait exceeded maximum")
            await asyncio.sleep(1.0)

Cost-Efficient Scaling

Horizontal scaling adds cost. Efficient scaling designs minimize the cost per processed task through smart resource allocation and utilization optimization.

Spot/Preemptible Instances for Batch Workloads

Batch and background agent tasks are excellent candidates for spot/preemptible instances (60–80% cheaper than on-demand). The checkpointing infrastructure built for reliability doubles as a spot instance resilience mechanism, when a spot instance is preempted, the task resumes from the last checkpoint on a new instance. Interactive tasks should always run on on-demand instances to prevent user-facing latency spikes from preemptions.

Capacity Planning for Agent Fleets

Capacity Model

• Average task duration: Profile your tasks to get p50 and p99 task completion times. Use p99 for capacity planning to handle tail latency.
• Peak concurrency estimation: tasks_per_second × average_task_duration_seconds = required concurrent workers. Add 50% headroom for bursts.
• LLM token budget: tasks_per_day × average_tokens_per_task = daily token requirement. Verify your provider tier supports this before scaling.
• State store sizing: active_sessions × average_session_state_size_KB = Redis memory requirement. Add 3x headroom for checkpoints and metadata.
• Cost projection: (compute_cost + llm_api_cost + tool_api_cost + state_store_cost) / tasks_per_day = cost per task. Track this metric as you scale to identify efficiency regressions.