Prompt Compression and Summarization Techniques That Preserve Meaning

Why Naive Compression Fails

The simplest compression strategy, truncate the oldest messages, destroys critical information. Many agentic tasks require referencing decisions made in early turns: the original task specification, constraints the user set, a key data point retrieved in turn 2. Truncation-based compression systematically removes exactly the context that was established earliest and is therefore most likely to be foundational.

A second failure mode is lossy re-phrasing that changes factual content. If a tool returned the number 47,392 and your summarizer compresses this to 'approximately 47,000', any subsequent calculation is wrong. Compression must preserve exact values, named entities, numerical data, and stated constraints without alteration.

The Compression Quality Spectrum

Think of compression quality as a spectrum from lossless to lossy. At the lossless end: deduplication removes exact-duplicate sentences while preserving all unique content. At the lossy end: abstractive summarization rewrites content in fewer words, potentially introducing errors. Most production systems use a cascade that starts lossless and progressively applies lossier techniques only when needed.

Extractive vs Abstractive Summarization

The two broad families of summarization differ fundamentally in how they reduce content. Extractive methods select and retain original sentences unchanged. Abstractive methods generate new text that captures meaning. Each has a distinct role in a production compression pipeline.

Extractive Summarization for Agents

• Sentence scoring: Score each sentence by TF-IDF weight, position in document, and overlap with the task query. Keep the top-k scoring sentences.
• Lead bias extraction: In conversational contexts, the opening sentences of each turn tend to carry the most information. Extract the first 1–2 sentences of each turn as the default compression.
• Keyword anchoring: Identify entity names, numerical values, and domain terms. Any sentence containing these anchors is automatically kept regardless of score.
• TextRank graph method: Build a similarity graph where sentences are nodes. Apply PageRank to identify the most 'central' sentences. Works well for long-form documents.

Abstractive Summarization for Agents

• LLM-based compression: Send a batch of old turns to a cheap model (claude-haiku-4-5, gpt-4o-mini) with a compression prompt. Effective compression ratios of 4:1 to 8:1 are achievable.
• Chain-of-density prompting: Instruct the LLM to iteratively densify a summary, adding missed entities without increasing length. Produces more information-dense summaries than single-pass.
• Constraint-aware compression: Include a list of critical entities in the prompt and instruct the model to preserve them verbatim. Prevents hallucination of numerical values.
• Structured compression: Rather than free-text summaries, compress turns into structured JSON with fields for decisions, constraints, entities, and tool results. Easier to reference programmatically.

What Information Must Be Preserved

Before implementing compression, build a non-negotiables list, information that must survive compression exactly. Compressing anything on this list is a bug, not a feature.

The Preservation Checklist

• Named entities: Person names, company names, product names, location names. Never summarize 'John Smith, CEO of Acme Corp' to 'the executive'.
• Numerical values: Prices, percentages, dates, counts, thresholds. Never round or approximate. Preserve exact values verbatim.
• Explicit constraints: Statements like 'the budget must not exceed $50,000' or 'avoid using library X'. These are binding requirements that must be preserved word-for-word.
• Task objectives: The original goal stated by the user. This is the most important item in the entire context. Never compress the task specification.
• Tool call outputs that were acted on: If a previous step used a tool result to make a decision, that result must be preserved. Removing it makes the decision appear ungrounded.
• Error states and failure information: If a step failed and the agent worked around it, the failure context must be retained to prevent re-attempting the same failing path.

Embedding-Based Deduplication

Agents frequently receive similar information multiple times: the same data point from different tools, repetitions in the user's messages, RAG chunks that overlap semantically. Embedding-based deduplication removes this redundancy without losing unique information.

Cosine Similarity Deduplication

import numpy as np
from typing import Optional

def deduplicate_chunks(
    chunks: list[str],
    embeddings: list[list[float]],
    similarity_threshold: float = 0.92,
) -> list[str]:
    """Remove chunks that are too similar to an already-kept chunk."""
    if not chunks:
        return []

    kept_indices = [0]
    kept_embeddings = [np.array(embeddings[0])]

    for i in range(1, len(chunks)):
        emb = np.array(embeddings[i])
        # Check against all kept embeddings
        similarities = [
            np.dot(emb, k) / (np.linalg.norm(emb) * np.linalg.norm(k))
            for k in kept_embeddings
        ]
        if max(similarities) < similarity_threshold:
            kept_indices.append(i)
            kept_embeddings.append(emb)

    return [chunks[i] for i in kept_indices]

async def embed_and_deduplicate(
    chunks: list[str],
    embed_fn,  # async function returning list[list[float]]
    threshold: float = 0.92,
) -> list[str]:
    embeddings = await embed_fn(chunks)
    return deduplicate_chunks(chunks, embeddings, threshold)

Deduplication Threshold Calibration

The similarity threshold determines aggressiveness. At 0.98, only near-identical chunks are removed. At 0.85, semantically similar but differently phrased content is merged. For factual domains, use 0.95+ to avoid removing distinct numerical values. For conversational history, 0.88–0.92 works well to remove repetitive messages.

LLM Compression Pass Techniques

Using an LLM for compression is effective but introduces latency and cost. Structure the compression pass carefully to maximize quality and minimize token spend.

Compression Prompt Design

COMPRESSION_PROMPT = """You are a context compression specialist for AI agents.

Compress the following conversation excerpt to {target_tokens} tokens or fewer.

PRESERVATION RULES (mandatory):
- Keep all named entities exactly as written (names, companies, products)
- Keep all numerical values exactly (prices, percentages, dates, IDs)
- Keep all explicit constraints word-for-word
- Keep all task objectives and stated goals
- Keep all error descriptions and failure information

COMPRESSION RULES:
- Remove pleasantries and conversational filler
- Merge repeated information (keep the most recent version)
- Shorten explanations to their core conclusion
- Use telegraphic style: omit articles and conjunctions where meaning is preserved

Original text ({actual_tokens} tokens):
{content}

Compressed version ({target_tokens} token budget):"""

async def compress_with_llm(
    content: str,
    target_tokens: int,
    actual_tokens: int,
    model: str = "claude-haiku-4-5-20251001",
) -> str:
    prompt = COMPRESSION_PROMPT.format(
        target_tokens=target_tokens,
        actual_tokens=actual_tokens,
        content=content,
    )
    # Use cheap fast model for compression
    response = await llm_call(model=model, prompt=prompt, max_tokens=target_tokens + 200)
    return response.content

Batch Compression for Efficiency

Don't compress each turn individually. Batch 2–4 consecutive turns and compress them together. Context across consecutive turns gives the summarizer enough information to identify what is repetitive vs novel. Single-turn compression can't identify cross-turn redundancy.

Hierarchical Compression Architecture

The most robust compression architecture for long-running agents is hierarchical. Recent turns are kept verbatim. Older turns are grouped and compressed into summaries. Even older summaries are merged into a session summary. This creates a temporal hierarchy where detail decreases with age.

Implementing the Three-Level Hierarchy

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class TurnSummary:
    turn_range: tuple[int, int]   # (start_turn, end_turn) inclusive
    summary_text: str
    token_count: int
    preserved_entities: list[str] = field(default_factory=list)
    original_token_count: int = 0

    @property
    def compression_ratio(self) -> float:
        if self.original_token_count == 0:
            return 0.0
        return 1 - (self.token_count / self.original_token_count)

class HierarchicalCompressor:
    def __init__(self, group_size: int = 2, llm_fn):
        self.group_size = group_size
        self.llm = llm_fn
        self._group_summaries: list[TurnSummary] = []
        self._session_summary: Optional[TurnSummary] = None

    async def compress_group(
        self, turns: list[dict], start_idx: int
    ) -> TurnSummary:
        """Compress 2-4 consecutive turns into a group summary."""
        raw_text = "\n".join(
            f"{t['role'].upper()}: {t['content']}" for t in turns
        )
        original_tokens = count_tokens(raw_text)
        target = max(120, original_tokens // 5)  # 5:1 compression target
        compressed = await compress_with_llm(raw_text, target, original_tokens)
        return TurnSummary(
            turn_range=(start_idx, start_idx + len(turns) - 1),
            summary_text=compressed,
            token_count=count_tokens(compressed),
            original_token_count=original_tokens,
        )

    async def merge_group_summaries(self) -> TurnSummary:
        """Merge all group summaries into a single session summary."""
        combined = "\n\n".join(s.summary_text for s in self._group_summaries)
        total_tokens = sum(s.token_count for s in self._group_summaries)
        target = max(300, total_tokens // 4)
        merged = await compress_with_llm(combined, target, total_tokens)
        return TurnSummary(
            turn_range=(self._group_summaries[0].turn_range[0],
                        self._group_summaries[-1].turn_range[1]),
            summary_text=merged,
            token_count=count_tokens(merged),
            original_token_count=total_tokens,
        )

Measuring Compression Quality

Compression quality is easy to misunderstand. Token reduction is not quality, a one-sentence summary achieves 99% token reduction but captures almost nothing. Quality means preserving the information that actually matters for task completion.

Quality Metrics Suite

• Entity recall: Fraction of named entities from the original that appear in the compressed version. Target: > 0.95 for strict domains.
• Numerical preservation: Fraction of numerical values that appear in the compressed version unchanged. Target: 1.0, zero tolerance for altered numbers.
• Semantic similarity: Cosine similarity between embeddings of original and compressed text. Target: > 0.85.
• Downstream task accuracy: Run a set of question-answering probes against the original and compressed context. Compare answer accuracy. This is the gold standard quality metric.
• Constraint recall: Fraction of explicit constraints (budget limits, prohibitions, requirements) preserved in compressed form.

Selective Compression Strategies

Not all content compresses equally well. Selective compression applies different techniques to different content types rather than running everything through the same pipeline.

Content-Type Compression Map

• Factual data (numbers, tables, dates): Use extractive only. Never abstractive. Preserve verbatim or remove entirely.
• Procedural steps (instructions, plans): Keep the action verbs and objects; remove explanatory filler. 'Navigate to Settings > Privacy > Clear all data' compresses well to the action sequence.
• Error messages and stack traces: Keep the error type and key lines; remove redundant frames. Preserve error codes exactly.
• User explanations and context: Abstractive compression works well here. Rephrase in compact form; flag preserved constraints.
• Tool API responses: JSON responses can be compressed by removing null fields, empty arrays, and metadata fields. Keep only the fields the agent acted on.

Compressing Tool Call Outputs

Tool outputs are often the largest token consumers in an agentic context. A single web search result or database query can return 2,000–10,000 tokens. The agent used maybe 5% of that content in its reasoning.

Tool Output Compression Strategies

import json
from typing import Any

def compress_json_output(
    data: Any, keys_to_keep: list[str] | None = None
) -> dict:
    """Remove null values, empty collections, and non-essential keys from JSON."""
    if isinstance(data, dict):
        compressed = {}
        for k, v in data.items():
            if v is None or v == [] or v == {}:
                continue
            if keys_to_keep and k not in keys_to_keep:
                continue
            compressed[k] = compress_json_output(v, keys_to_keep)
        return compressed
    elif isinstance(data, list):
        return [compress_json_output(item, keys_to_keep) for item in data]
    return data

def compress_tool_result(
    tool_name: str, result: str, max_tokens: int, count_fn
) -> str:
    """Intelligently compress tool output based on tool type."""
    current_tokens = count_fn(result)
    if current_tokens <= max_tokens:
        return result

    # Try JSON compression first (lossless)
    try:
        parsed = json.loads(result)
        compressed_json = json.dumps(compress_json_output(parsed), separators=(",", ":"))
        if count_fn(compressed_json) <= max_tokens:
            return compressed_json
    except (json.JSONDecodeError, TypeError):
        pass

    # Truncate with summary marker for non-JSON
    truncated = result[:max_tokens * 4]  # rough char estimate
    return truncated + f"\n[TRUNCATED: original was {current_tokens} tokens]"

Building a Production Compression Pipeline

A production pipeline needs to be fast (not blocking the agent on every turn), reliable (compression failure should fall back gracefully), and auditable (log what was compressed so you can debug unexpected agent behavior).

Pipeline Architecture Decisions

• Async compression: Run compression asynchronously after each turn. Don't block the agent while compression runs. Apply the result in the next turn if ready.
• Compression caching: Cache compressed summaries keyed by content hash. If the same tool output appears again, use the cached compression rather than re-running.
• Compression queue: Maintain a queue of items pending compression. Process in order of age, oldest items get compressed first.
• Fallback to truncation: If the LLM compressor fails or times out, fall back to extractive truncation rather than failing the agent.
• Compression audit log: Log every compression event with original content hash, compressed content hash, token delta, and compression method used. Essential for debugging.

Compression Latency Budget

LLM-based compression calls add latency. A haiku compression call for 2,000 tokens typically takes 800ms–1.5s. If run synchronously, this blocks every turn once compression triggers. Use the async pipeline pattern: compress in the background while the agent continues, and inject the compressed version at the next context assembly step.