Context Window Budgeting Strategies for Long-Running Agents

Why Token Budgeting Matters

Long-running agents are context hungry. Each tool call appends its full input and output. Every retrieved document lands verbatim in the prompt. System prompts grow as capability instructions accumulate. Without a deliberate budget, you hit the model's context limit at the worst possible moment, mid-task, mid-reasoning, in the middle of a complex multi-step workflow.

The consequences range from hard failures (the API returns a 400 context_length_exceeded) to soft failures (the model silently ignores the earliest messages because its attention degrades on extremely long sequences). Neither is acceptable in production. A systematic budget prevents both.

The Cost Dimension

Token budgeting is not just an availability concern, it's a cost concern. LLM providers charge per token on both input and output. A 128K context window at $15/M input tokens costs $1.92 per request before a single output token is generated. If your agent loops 20 times with a growing context, you're paying $38+ per task in input costs alone. Budget discipline directly controls cost.

The Latency Dimension

Time-to-first-token (TTFT) scales with context length. At 4K tokens, TTFT is typically 300–500ms. At 64K tokens, it's 2–4 seconds. At 128K tokens, it's 8–15 seconds on most providers. Keeping contexts lean directly translates to a faster, more responsive agent.

Anatomy of a Context Window

Before you can budget, you need to understand what occupies context. Every agentic context window consists of five logical regions, each with different management characteristics.

Fixed vs Variable Cost Regions

• System prompt (FIXED): Instructions, persona, constraints, output format. Ranges from 500 to 4,000 tokens. Rarely changes per request. Optimize once, compress aggressively.
• Tool definitions (FIXED): JSON schema for every registered tool. Each tool adds 100–400 tokens. Use lazy registration, only include tools relevant to the current task phase.
• Memory / RAG chunks (VARIABLE): Retrieved documents, past summaries, knowledge injections. The most variable region. Budget 20–40% of total window here.
• Conversation history (VARIABLE): The growing accumulation of turns. Grows by 400–1,000 tokens per round trip. Needs active pruning after 5–10 turns.
• User input + assistant response reserve (VARIABLE): Space for the current turn. Always reserve at minimum 1,000 tokens for user input and 2,000 for model output.

Hidden Token Consumers

Beyond the obvious categories, several hidden consumers quietly eat tokens. Special tokens (BOS, EOS, role delimiters) add 4–20 tokens per message. Structured output format instructions (JSON schemas, XML constraints) can add 200–500 tokens. Chain-of-thought prompting instructions inflate the assistant prefix. Count all of these in your budget model.

Static vs Dynamic Allocation

The simplest budgeting strategy is static allocation: reserve fixed token counts for each region up front. This is easy to implement but wastes capacity. If your memory region is budgeted at 3,000 tokens but the current task requires no retrieval, those 3,000 tokens go unused while your history region might be starved.

Static Allocation Implementation

from dataclasses import dataclass
from typing import Optional

@dataclass
class ContextBudget:
    total: int = 10_000
    system_prompt: int = 2_000
    tool_definitions: int = 1_500
    memory_rag: int = 3_000
    conversation_history: int = 2_500
    user_input_reserve: int = 500
    assistant_output_reserve: int = 1_500

    def __post_init__(self):
        allocated = (
            self.system_prompt + self.tool_definitions +
            self.memory_rag + self.conversation_history +
            self.user_input_reserve + self.assistant_output_reserve
        )
        assert allocated <= self.total, (
            f"Budget overallocated: {allocated} > {self.total}"
        )

    @property
    def unallocated(self) -> int:
        return self.total - (
            self.system_prompt + self.tool_definitions +
            self.memory_rag + self.conversation_history +
            self.user_input_reserve + self.assistant_output_reserve
        )

Dynamic Allocation Implementation

Dynamic allocation computes region sizes at runtime based on actual content. It requires a BudgetManager that tracks live usage and adjusts limits as the task progresses.

import tiktoken

class DynamicBudgetManager:
    def __init__(self, model: str, total_tokens: int):
        self.enc = tiktoken.encoding_for_model(model)
        self.total = total_tokens
        self._usage: dict[str, int] = {}

    def count(self, text: str) -> int:
        return len(self.enc.encode(text))

    def measure_region(self, region: str, content: str) -> int:
        tokens = self.count(content)
        self._usage[region] = tokens
        return tokens

    @property
    def used(self) -> int:
        return sum(self._usage.values())

    @property
    def remaining(self) -> int:
        return self.total - self.used

    def can_fit(self, content: str, reserve: int = 1000) -> bool:
        return self.count(content) <= (self.remaining - reserve)

    def budget_for_region(
        self, region: str, base_pct: float, min_tokens: int = 500
    ) -> int:
        # Give each region a dynamic slice of what's left
        # after fixed costs are accounted for
        fixed_used = sum(
            v for k, v in self._usage.items()
            if k in ("system_prompt", "tool_definitions")
        )
        variable_budget = self.total - fixed_used - 2000  # output reserve
        return max(min_tokens, int(variable_budget * base_pct))

Token Counting Methods and Tools

Accurate token counting is a prerequisite for any budgeting system. Different models use different tokenizers, and approximate counting leads to budget overruns or overly conservative pruning.

Tokenizer Libraries by Provider

• OpenAI (GPT-4o, GPT-4): Use tiktoken library, cl100k_base encoding. Install: pip install tiktoken.
• Anthropic (Claude): Use anthropic.count_tokens() API or the official anthropic Python SDK's count_tokens method.
• Google (Gemini): Use google.generativeai.count_tokens() or the Vertex AI API countTokens endpoint.
• Local models (Llama, Mistral): Use transformers.AutoTokenizer from HuggingFace. Load the exact checkpoint tokenizer for accuracy.
• Estimation fallback: 1 token ≈ 4 characters for English text. Useful for quick estimates but never for production budget enforcement.

Counting Overhead Beyond Raw Text

import tiktoken

def count_chat_tokens(messages: list[dict], model: str = "gpt-4o") -> int:
    """Count tokens including per-message overhead for chat format."""
    enc = tiktoken.encoding_for_model(model)
    # Each message adds overhead for role + delimiters
    # gpt-4o: 3 tokens per message, 1 for reply priming
    tokens_per_message = 3
    tokens_per_name = 1
    num_tokens = 3  # reply priming tokens
    for msg in messages:
        num_tokens += tokens_per_message
        for key, value in msg.items():
            num_tokens += len(enc.encode(str(value)))
            if key == "name":
                num_tokens += tokens_per_name
    return num_tokens

def count_tool_schema_tokens(tools: list[dict], model: str = "gpt-4o") -> int:
    """Tool schemas are serialized to JSON before tokenization."""
    import json
    enc = tiktoken.encoding_for_model(model)
    schema_json = json.dumps(tools, indent=2)
    return len(enc.encode(schema_json))

Implementing Budget Enforcement

Counting tokens is only useful if you enforce the budget. Enforcement means intercepting context assembly before the API call and triggering compression or pruning when limits are exceeded.

Three-Zone Enforcement Model

Model your budget as three zones rather than a binary fit/overflow check. The green zone (0–70% utilization) operates normally. The yellow zone (70–85%) activates light compression: summarize the oldest 20% of history, drop low-priority tool results. The red zone (85–95%) triggers aggressive pruning: keep only the system prompt, last 3 turns, and the most recent tool result.

from enum import Enum
from dataclasses import dataclass, field

class BudgetZone(Enum):
    GREEN = "green"
    YELLOW = "yellow"
    RED = "red"
    CRITICAL = "critical"

@dataclass
class ContextAssembler:
    budget: DynamicBudgetManager
    compressor: "ContextCompressor"  # forward ref

    def get_zone(self) -> BudgetZone:
        pct = self.budget.used / self.budget.total
        if pct < 0.70:
            return BudgetZone.GREEN
        elif pct < 0.85:
            return BudgetZone.YELLOW
        elif pct < 0.95:
            return BudgetZone.RED
        return BudgetZone.CRITICAL

    async def assemble(
        self,
        system_prompt: str,
        tools: list[dict],
        history: list[dict],
        memory_chunks: list[str],
        user_message: str,
    ) -> list[dict]:
        # Always include fixed regions first
        self.budget.measure_region("system_prompt", system_prompt)
        self.budget.measure_region("tool_definitions", str(tools))

        zone = self.get_zone()

        if zone == BudgetZone.GREEN:
            trimmed_history = history
            trimmed_memory = memory_chunks
        elif zone == BudgetZone.YELLOW:
            trimmed_history = await self.compressor.light_compress(history)
            trimmed_memory = memory_chunks[:3]  # top-3 chunks only
        else:
            trimmed_history = history[-3:]  # last 3 turns only
            trimmed_memory = memory_chunks[:1]  # single most relevant

        # Measure and assemble
        self.budget.measure_region("history", str(trimmed_history))
        self.budget.measure_region("memory", " ".join(trimmed_memory))

        return self._build_messages(
            system_prompt, trimmed_history, trimmed_memory, user_message
        )

Priority Queuing Strategies

Not all context items are equal. Priority queuing assigns a retention score to each item and evicts the lowest-scored items first when the budget is under pressure.

Priority Score Formula

A good priority score combines recency (newer items score higher), relevance (semantic similarity to current task), and type (tool results generally more valuable than conversational filler). A practical formula: score = (recency_weight × normalized_position) + (relevance_weight × cosine_similarity) + type_bias.

Type-Based Priority Tiers

• TIER 0, Protected (never evicted): system prompt, current task specification, hard constraints from user, active in-progress reasoning.
• TIER 1, High priority: last 3 conversation turns, most recent tool results, cited sources, explicit user decisions.
• TIER 2, Medium priority: tool results from earlier steps, retrieved knowledge chunks, background context documents.
• TIER 3, Low priority (first to evict): verbose explanations, intermediate reasoning steps, duplicate information, metadata blocks.
• TIER 4, Purgeable: filler text, pleasantries, auto-generated commentary without substance.

Implementing Scored Eviction

from dataclasses import dataclass, field
from typing import Literal
import numpy as np

ContextTier = Literal[0, 1, 2, 3, 4]

@dataclass
class ContextItem:
    content: str
    tier: ContextTier
    position: int          # index in conversation, 0 = oldest
    token_count: int
    relevance: float = 0.5  # cosine similarity to current task

    @property
    def priority_score(self) -> float:
        tier_bias = {0: 100.0, 1: 10.0, 2: 5.0, 3: 1.0, 4: 0.1}
        recency = self.position / 100  # normalized 0–1
        return tier_bias[self.tier] + recency * 3.0 + self.relevance * 2.0

def evict_to_fit(
    items: list[ContextItem],
    token_budget: int,
) -> list[ContextItem]:
    """Remove lowest-priority items until total fits within budget."""
    total = sum(i.token_count for i in items)
    if total <= token_budget:
        return items

    # Sort by priority ascending, lowest priority evicted first
    sortable = sorted(
        [i for i in items if i.tier > 0],  # never touch tier 0
        key=lambda x: x.priority_score
    )
    protected = [i for i in items if i.tier == 0]

    result = list(sortable)
    while result and total > token_budget:
        evicted = result.pop(0)
        total -= evicted.token_count

    return protected + sorted(result, key=lambda x: x.position)

Context Overflow Handling

Even with good budgeting, overflow happens, especially in long-running autonomous agents where tasks expand unpredictably. A robust overflow handler must not lose information; it must transform it into a more compact form.

The Four-Step Overflow Pipeline

• Step 1, Prune: Apply priority-based eviction to remove purgeable and low-priority items. Fast operation, no LLM required.
• Step 2, Compress: Run extractive summarization on the oldest history window. Group turns into 2-turn batches and summarize each with a cheap model (haiku, mini).
• Step 3, Checkpoint: Persist the full state to external storage before continuing. This creates a recovery point if the agent needs to restart.
• Step 4, Rebuild: Reconstruct the context from protected items + summaries + recent raw turns. Verify the rebuilt context fits within budget before proceeding.

Checkpoint Storage Strategy

Checkpoints should be stored in fast key-value storage (Redis, DynamoDB) keyed by session ID and turn number. Include: full conversation history up to this point, all tool call results, the current task state, and the agent's working memory. Checkpoints enable resumption after crashes, budget-triggered restarts, and post-hoc debugging.

Multi-Agent Budget Sharing

In multi-agent systems, multiple agents may share a token budget for a single task. The orchestrator must coordinate budget allocation across workers, ensure no single agent starves others, and collect results efficiently.

Budget Allocation Patterns

• Equal split: Divide the task budget evenly across all worker agents. Simple but inefficient, some tasks require more tokens than others.
• Capability-weighted split: Allocate more budget to agents running complex reasoning tasks and less to simple extraction or formatting agents.
• Demand-based allocation: Each worker requests the budget it needs before starting. The orchestrator approves, denies, or negotiates a smaller allocation.
• Pool-based allocation: Maintain a shared token pool. Workers draw from the pool as needed. Risk: a runaway agent can exhaust the pool.
• Hierarchical budgets: Each parent agent has a budget for itself and a child pool budget. Children get allocated from the pool, never from the parent's own budget.

Monitoring Budget Utilization

Budget monitoring closes the feedback loop. Without observability into how tokens are being spent, you're flying blind and can't identify which regions are bloated or which tasks are unexpectedly expensive.

Key Metrics to Track

• context_utilization_pct: Percentage of total budget used at time of API call. Alert when > 85% consistently.
• region_tokens_by_type: Token counts broken down by system_prompt, tools, memory, history, reserve. Identifies which region is growing fastest.
• overflow_events_per_session: Count of times overflow handling was triggered per agent session. High values indicate structural budget misalignment.
• compression_ratio: Tokens before vs after compression. Tracks compression efficiency over time.
• tokens_wasted_on_pruned_items: Tokens spent computing content that was ultimately pruned. Indicates upstream inefficiency.

Prometheus Instrumentation

from prometheus_client import Histogram, Counter, Gauge

context_utilization = Histogram(
    "agent_context_utilization_pct",
    "Context window utilization percentage at API call time",
    ["agent_id", "task_type"],
    buckets=[0.3, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 1.0],
)

region_tokens = Gauge(
    "agent_context_region_tokens",
    "Token count by region in current context",
    ["agent_id", "region"],
)

overflow_events = Counter(
    "agent_context_overflow_total",
    "Number of context overflow events triggering compression",
    ["agent_id", "zone"],  # zone: yellow, red, critical
)

def record_context_snapshot(
    agent_id: str, task_type: str, budget: DynamicBudgetManager
) -> None:
    pct = budget.used / budget.total
    context_utilization.labels(agent_id, task_type).observe(pct)
    for region, tokens in budget._usage.items():
        region_tokens.labels(agent_id, region).set(tokens)

Common Budgeting Mistakes

Token budgeting is conceptually simple but practically tricky. Here are the mistakes that cause the most production incidents.

Mistake 1: Counting Tokens Too Late

The most common mistake is counting tokens after assembling the full context, by which point there's nothing to do except truncate crudely. Count tokens incrementally as each region is added, and make go/no-go decisions at each stage.

Mistake 2: Not Reserving Output Space

Many teams allocate the full context window to input and forget that the model needs room to generate output. If your context is 127,500 tokens in a 128K window, the model can only generate 500 tokens of response. Always reserve at least 1,500–2,500 tokens for output, more for tasks that require long structured responses.

Mistake 3: Ignoring Tool Definition Token Cost

Tool schemas are surprisingly expensive. A complex tool with nested parameters can cost 300–500 tokens. An agent with 20 registered tools might spend 6,000–8,000 tokens just on definitions. Use lazy tool registration: only include tools relevant to the current phase of the task.

Mistake 4: Different Tokenizers for Different Models

If you switch models mid-deployment (e.g., from GPT-4o to Claude), the tokenizer changes. GPT-4 and Claude have meaningfully different token counts for the same text. Always use the tokenizer that matches your current model, or you'll consistently misestimate your budget.