Short-Term, Long-Term, and Episodic Memory Architecture for AI Agents

The Human Memory Analogy

Cognitive psychology identifies multiple distinct human memory systems, each optimized for different tasks and timescales. Working memory holds the last few seconds of information in active processing, the conversation you're currently having, the calculation you're mid-way through. Semantic memory stores general facts and knowledge, what you know about the world, language, concepts. Episodic memory stores autobiographical events, what happened to you, when, and in what context.

AI agents benefit from analogous distinctions. An agent that uses a single undifferentiated memory blob, retrieving anything from anywhere for any query, provides a much worse experience than one that correctly distinguishes between what's happening right now (working memory), what it knows about this user in general (semantic/long-term memory), and what happened in a specific past interaction (episodic memory). The architecture mirrors cognitive reality because the underlying retrieval problems are the same.

Why the Distinction Matters Practically

• Working memory determines: What the agent is attending to right now, which tool results are in scope, what the last user message said. Needs instant access, exact content.
• Semantic memory determines: What the agent knows about the user's preferences, constraints, and domain. Needs semantic retrieval, approximate matches acceptable.
• Episodic memory determines: What happened in a specific past session, what was discussed on a specific date, how a past task was handled. Needs temporal and contextual retrieval.

Three-Memory Architecture Overview

The three-memory architecture maps each human memory type to a specific technical implementation optimized for its access patterns. Short-term/working memory maps to in-context storage (the LLM's context window itself). Long-term semantic memory maps to a vector database with semantic retrieval. Episodic memory maps to a time-indexed event store with temporal querying.

Memory System Comparison

• Short-term (working): Content: last N turns raw. Access: always in context, zero retrieval latency. Capacity: 2K–8K tokens. Persistence: session only. Use when: anything in the current conversation.
• Long-term (semantic): Content: facts, preferences, constraints, domain knowledge. Access: semantic vector search. Capacity: millions of items. Persistence: indefinite. Use when: 'what do I know about this user?'
• Episodic: Content: session summaries, events, decisions in context. Access: temporal + semantic search. Capacity: hundreds to thousands of sessions. Persistence: indefinite. Use when: 'what did we discuss on March 3rd?'

Short-Term Working Memory Design

Working memory is the simplest of the three systems because it's inherent to the LLM architecture, the context window is working memory. The design challenge is managing what goes into working memory, not how to store it.

Working Memory Contents

• Current turn: The user's current message. Always present.
• Recent conversation: The last 5–10 turns of raw exchange. Provides immediate conversation context. This is the primary source of continuity within a session.
• Active tool results: Results from tools called in the last 2–3 turns. Still being reasoned about. Should not be summarized until they're acted on.
• Current task state: The active sub-task description, current step in the plan, and any variables being tracked. Small, structured, always fresh.
• Retrieved memories: The 3–6 memories pulled from long-term and episodic stores for this turn. Injected as a labeled block, not mixed with conversation.

Working Memory Management

As the conversation grows, the working memory window slides forward. Turns that fall off the end of the window are handled by the summarization/consolidation pipeline, their key content is extracted and written to long-term memory before they leave the window. The transition from working memory to long-term memory should be imperceptible to the user.

Long-Term Semantic Memory Design

Long-term semantic memory stores persistent knowledge about the user and their domain that should influence the agent's behavior across all sessions. It answers the question: 'What do I know about this user that's relevant to what they're asking right now?'

Semantic Memory Item Types

• Preferences: Communication style, output format, level of technical detail, topics of interest/disinterest. 'User prefers bullet points over prose.' 'User wants concise responses under 200 words for most queries.'
• Constraints: Hard limits and prohibitions. 'User's budget cap is $10,000 for this project.' 'Never suggest vendor Y, previous bad experience.' These are the most important items to retrieve reliably.
• Domain knowledge: Facts about the user's organization, domain, and context. 'User works on a team of 8, uses Python/FastAPI stack, deploys to AWS.' Prevents the agent from repeatedly asking for background information.
• Historical decisions: Patterns from past decisions. 'User consistently chooses approach A over B for this type of problem.' Enables the agent to make better recommendations aligned with established patterns.

Semantic Memory Indexing Strategy

Semantic memory items should be chunked at the single-fact level, one memory item per discrete piece of information. 'User prefers Python' and 'User dislikes Java' are two separate memory items, not one combined item. Single-fact chunking maximizes retrieval precision: you retrieve exactly what's relevant, not a block that happens to contain what you need alongside irrelevant information.

Episodic Memory, What Happened When

Episodic memory stores records of what happened in specific past interactions. Unlike semantic memory (which stores extracted facts), episodic memory stores the events themselves, what was discussed, what decisions were made, what tasks were completed, in the context of when they occurred.

Episodic Record Structure

• session_id: Unique identifier for the session.
• timestamp_start / timestamp_end: When the session occurred and how long it ran.
• session_summary: A 200–500 word summary of what occurred in the session.
• key_events: Ordered list of significant events within the session (decisions made, tasks completed, problems encountered).
• entities_mentioned: Named entities that appeared in the session, people, projects, products, organizations.
• outcomes: What was accomplished and what was not, and why.
• embedding: Dense vector of the session summary for semantic retrieval.

Temporal Indexing for Episodic Memory

Episodic memory requires temporal querying in addition to semantic search. 'What did we discuss last week?' requires a time-range filter that pure vector search can't handle. Implement a time-indexed secondary index (B-tree on timestamp in PostgreSQL) alongside the vector embedding. Hybrid queries that combine time-range filtering with semantic similarity provide the most precise episodic retrieval.

Memory Consolidation at Session End

What Gets Written to Which Memory System

• → Long-term semantic memory: Extracted preferences, constraints, domain facts, and decision patterns. Written as individual memory items with importance scores.
• → Episodic memory: The full session record (summary, key events, outcomes). Written as a single episodic entry with temporal metadata.
• → Cold storage archive: The compressed full conversation. Written for compliance and potential future retrieval of raw detail.
• NOT written: Intermediate reasoning, tool call logs, conversational filler. These have no long-term retrieval value.

Cross-Memory Retrieval and Synthesis

The most sophisticated agents synthesize information across all three memory systems for each turn. This requires a retrieval orchestrator that queries each memory system, combines the results, resolves conflicts, and assembles a coherent context block.

Multi-Memory Context Assembly

from dataclasses import dataclass

@dataclass
class MemoryContext:
    working_memory: list[dict]       # recent turns (always included)
    semantic_memories: list[dict]    # retrieved long-term facts
    episodic_memories: list[dict]    # retrieved past sessions
    total_tokens: int

async def assemble_memory_context(
    session_id: str,
    user_id: str,
    current_query: str,
    working_window: list[dict],
    semantic_store,
    episodic_store,
    token_budget: int = 4000,
) -> MemoryContext:
    # Always include working memory (highest priority)
    working_tokens = count_tokens(str(working_window))
    remaining = token_budget - working_tokens

    # Retrieve semantic memories (user facts, preferences)
    semantic = await semantic_store.search(
        query=current_query,
        namespace=f"user:{user_id}",
        top_k=8,
        min_score=0.75,
    )
    semantic_tokens = count_tokens(str(semantic[:5]))  # cap at 5
    remaining -= semantic_tokens

    # Retrieve episodic memories if budget allows
    episodic = []
    if remaining > 500:
        episodic = await episodic_store.search(
            query=current_query,
            user_id=user_id,
            top_k=3,
            min_score=0.70,
        )

    return MemoryContext(
        working_memory=working_window,
        semantic_memories=semantic[:5],
        episodic_memories=episodic[:2],
        total_tokens=working_tokens + semantic_tokens + count_tokens(str(episodic)),
    )

Memory Decay and Relevance Over Time

Not all memories remain equally relevant forever. A preference stated by the user 2 years ago may no longer reflect their current preference. A constraint that was critical for a project that ended 6 months ago may no longer be relevant. Memory decay, reducing the retrieval priority of older memories, models this reality.

Decay Mechanisms

• TTL-based expiry: Assign a TTL to memories based on type. Situational preferences (project-specific): 90 days. General preferences: indefinite. Specific constraints: indefinite until explicitly removed.
• Score decay over time: Reduce the importance_score of memories that haven't been retrieved or referenced recently. A memory that hasn't been relevant in 6 months gets a 30% score reduction each month.
• Supersession: When a new memory explicitly contradicts an old one, mark the old one as superseded. It's no longer returned in retrieval but is preserved in the audit trail.
• User-triggered refresh: When a user explicitly updates a preference ('actually, I've changed my mind about format X'), trigger an explicit memory update rather than relying on decay.

Handling Memory Conflicts and Updates

Memories can conflict, a user said A last week and B this week. Determining which is correct (B is more recent, so probably more accurate) and updating the memory store accordingly is one of the more complex memory management tasks.

Conflict Resolution Protocol

• Detect contradiction at write time: Before writing a new memory, search for semantically similar existing memories. If a high-similarity match is found (> 0.90) with contradicting content, flag as a conflict.
• Surface the conflict to the agent: In the next turn, inject both memories with a conflict flag. Let the agent acknowledge the conflict and ask the user to confirm the current state.
• User resolution: After user clarification, write the correct memory and mark the incorrect one as superseded.
• Automatic resolution by recency: For time-sensitive information (like preferences that naturally evolve), automatically treat the more recent memory as authoritative without explicit user confirmation.

Implementation Guide

Implementation Priority Order

• Phase 1, Working memory (0–2 weeks): Sliding window for conversation turns, basic context assembly. Immediate quality improvement for multi-turn conversations.
• Phase 2, Long-term semantic memory (2–6 weeks): Vector store integration, preference/constraint extraction at session end, basic retrieval injection.
• Phase 3, Episodic memory (6–12 weeks): Session record creation at session end, temporal+semantic retrieval, cross-session continuity.
• Phase 4, Consolidation pipeline (3–4 months): Automated session-end processing, importance scoring, duplicate detection, memory decay.
• Phase 5, Conflict resolution and updates (4–6 months): Contradiction detection, user-triggered memory updates, supersession tracking.

Minimum Viable Memory System

If you're starting from scratch, the highest-impact minimum viable memory system is: (1) a Redis-backed sliding window of the last 8 turns injected into every call, and (2) a simple PostgreSQL table of user preferences extracted at session end with basic CRUD retrieval. This provides 80% of the user experience benefit of a full three-memory architecture at 20% of the implementation complexity.