When to Summarize vs Retain Raw History in Agent Memory

The Core Tradeoff: Fidelity vs Efficiency

Every token of conversation history kept in context costs money and increases latency. Every token of history compressed to a summary risks losing information that the agent needs. The design challenge is identifying which information is worth the token cost to retain verbatim and which is safe to compress without meaningful quality loss.

The answer is not a single policy, 'always summarize after 10 turns' or 'always keep the last 20 turns'. It's a content-type analysis. The same conversation can contain a hard constraint that must be preserved verbatim, an emotional context that can be summarized to its core sentiment, and a lengthy explanation that can be discarded entirely after its conclusion is retained.

The Cost of Getting It Wrong in Each Direction

• Over-summarizing: The agent loses a critical constraint the user stated in turn 3. In turn 20, it makes a recommendation that violates that constraint. The user is frustrated that the agent 'forgot' something explicitly stated. Trust is damaged.
• Under-summarizing: The agent carries 30,000 tokens of raw history into turn 50. Input costs are $0.45 per call. Latency is 4+ seconds per turn. The system is too expensive to run profitably.
• Inconsistent policy: Sometimes raw, sometimes summarized with no clear rule. Different agent instances handle the same conversation differently. Users experience unpredictable behavior.

The Retain vs Summarize Decision Tree

The Three Key Questions

• Question 1, Exact values? Prices, quantities, dates, IDs, usernames, product codes. These must be retained verbatim. A summary that rounds $47,392 to '$47K' introduces an error that propagates to any calculation using that value.
• Question 2, Near-term reference? If the item will likely be referenced in the next 5 turns (currently active in the task), keep it raw in working memory. If it's older and less likely to be directly referenced, it becomes a candidate for summarization.
• Question 3, Verbatim wording matters? Explicit user decisions, direct quotes, contractually-significant language, these need verbatim preservation. The agent saying 'approximately what you said' is not sufficient for these cases.

Content That Must Always Be Retained Raw

Some categories of information should never be summarized. Summarization introduces paraphrasing, and paraphrasing introduces errors. For these categories, the token cost is always justified.

Absolute Retention Categories

• Hard constraints and prohibitions: 'Never recommend this approach', 'Budget must not exceed $50,000', 'This user is allergic to X'. These are behavioral constraints that must be preserved exactly. A paraphrase might soften the constraint unintentionally.
• Numerical values that drive computation: Any number that subsequent calculations depend on, quantities, prices, rates, percentages, dates. A compressed '~50K' is wrong for any downstream arithmetic.
• Explicit user decisions and approvals: When the user explicitly approved an approach, the approval itself is the important thing, not a summary of it. The verbatim 'Yes, proceed with Option B' is stronger than 'User approved an option'.
• Error states and failures: Exact error messages, failure descriptions, and retry counts matter for debugging. Summarizing 'the API returned an authentication error' to 'there was a problem' loses diagnostic information.
• User quotes used as references: If the user said something precise that the agent needs to reference accurately, preserve it verbatim. Paraphrasing user language in a response can feel presumptuous.

Content Safe to Summarize

These content types preserve their essential value through summarization because the meaning, not the wording, is what matters.

Summarization-Eligible Categories

• Explanatory context: Background information the user provided to explain their situation. The conclusion matters; the phrasing of the explanation doesn't. 'User is a startup founder managing a 10-person team' is a valid summary of a 3-paragraph company description.
• Conversational back-and-forth: 'Did you mean X or Y?', 'Yes, I meant X', 'Got it, I'll do X'. This exchange can be summarized to 'Confirmed: scope is X' without losing anything actionable.
• Completed reasoning chains: When the agent worked through a multi-step analysis and reached a conclusion, the conclusion is what matters. The intermediate steps can be compressed to 'Agent analyzed options A/B/C and selected A because it met the cost requirement'.
• Redundant restatements: Users often restate their needs multiple times in different ways as the conversation evolves. Keep the most recent, most precise restatement; discard the earlier variants.
• Small talk and pleasantries: Entirely removable without any meaningful loss.

Hybrid Memory Tier Architecture

Tier Transition Rules

• Tier 1 → Tier 2: When a turn exits the working memory window (older than N turns), apply summarization. Extract key entities, decisions, and constraints. Store the summary with the original turn's metadata.
• Tier 2 → Tier 3: After session end, consolidate the Tier 2 summaries into a session-level summary. Embed it and write to the vector store. This represents the permanent long-term memory of the session.
• Cross-tier retrieval: When assembling context for a turn, the agent uses all three tiers. Tier 1 is always included. Tier 2 summaries of recent prior context are included selectively. Tier 3 is retrieved on demand based on semantic relevance to the current query.

Ensuring Summarization Quality

Summarization quality is mission-critical for memory systems. A bad summarizer silently corrupts the agent's memory, causing subtle failures that are hard to trace back to the summarization step.

Quality Assurance Prompts

SUMMARIZATION_PROMPT = """Summarize the following conversation excerpt.

MANDATORY PRESERVATION RULES:
1. Keep ALL numerical values exactly as stated (prices, dates, quantities, IDs)
2. Keep ALL explicit constraints verbatim ("never", "must", "always", "not allowed")
3. Keep ALL explicit user decisions and approvals verbatim
4. Keep ALL named entities exactly (person names, product names, company names)
5. Preserve CAUSALITY: if A led to B, the summary must show this relationship

COMPRESSION RULES:
- Combine repeated information (keep only the most recent/precise version)
- Replace explanatory background with its conclusion
- Remove pleasantries and conversational filler
- Use telegraphic style where meaning is unambiguous

After writing the summary, check it against each rule above.
If any rule is violated, revise.

Conversation:
{conversation_text}

Summary:"""

Automated Quality Validation

After generating a summary, run automated checks: extract all numerical values from the original, verify they appear in the summary unchanged; extract named entities, verify they appear in the summary; check that no sentence in the summary is contradicted by the original. These checks catch the most common summarization errors without requiring human review.

Progressive Compression Over Time

Memory compression should become progressively more aggressive with age. Recent context (last 5 turns) is kept verbatim. Context from last session is summarized at moderate detail. Context from 3 months ago is compressed to high-level insights. The compression ratio increases with time because the specifics of old interactions matter less than their conclusions.

Compression Ratio by Age

• 0–5 turns (current session working memory): 1:1, raw, no compression. Keep everything.
• 6–50 turns (recent history): 4:1 to 6:1, preserve key facts, decisions, constraints, named entities.
• 1–4 sessions ago: 8:1 to 12:1, preserve high-level decisions and significant facts only.
• 5+ sessions ago: 20:1 to 50:1, session-level single-paragraph summary covering: what the session accomplished, key decisions, and any constraints established.
• 1+ year ago: Distill to a few key user preference facts if still relevant; discard session-specific context entirely.

Retrieval Strategy for Hybrid Systems

In a hybrid tier system, context assembly for each turn must intelligently combine content from all three tiers without exceeding the token budget.

Context Assembly Algorithm

• Step 1, Fixed allocation: Reserve budget for system prompt and user message. These are non-negotiable.
• Step 2, Tier 1 injection: Always include the most recent K turns (working memory). Budget: 2,000–5,000 tokens.
• Step 3, Tier 2 injection: Include the Tier 2 summary of the current session's older history. Budget: 500–1,000 tokens.
• Step 4, Tier 3 retrieval: Semantically retrieve 3–6 relevant long-term memories. Budget: 500–1,500 tokens.
• Step 5, Budget reconciliation: If total exceeds budget, reduce Tier 3 first (fewer memories), then reduce Tier 2 detail, then reduce Tier 1 window.

Session-End Memory Consolidation

What to Extract During Consolidation

• User preferences revealed: Communication style, format preferences, level of detail wanted, topics of interest or disinterest.
• Decisions made: Every explicit decision the user or agent made during the session, with the reasoning captured.
• Constraints established: Any limitations, restrictions, or 'never do this' statements made during the session.
• Domain knowledge acquired: Facts about the user's domain, organization, or situation that weren't in the initial agent context.
• Task outcomes: What was accomplished, what wasn't, and why. This shapes how the agent approaches follow-up sessions.

Failure Modes and Mitigations

Failure Mode 1, Constraint Compression

The summarizer compresses 'NEVER recommend product X to this user, they had a severe reaction' to 'user has had issues with some products'. The severity and specificity are lost. Mitigation: extract constraints before summarization, store them verbatim in a separate constraint store, never include them in the content that goes through the summarizer.

Failure Mode 2, Hallucinated Summary

The LLM summarizer introduces a fact that wasn't in the original, a confabulation. The summarized memory now contains false information that the agent will use as if it were true. Mitigation: automated fact-checking against the original content before accepting a summary, human-in-the-loop for high-importance memories.

Failure Mode 3, Over-Aggressive Compression

The compression policy is too aggressive, too short a working window, too high a compression ratio. The agent regularly asks users to re-state information they already provided. This is more damaging to user trust than slightly higher token costs. Calibrate compression aggressiveness by measuring re-stated information rate as a quality signal.