Vector Store Design for Agent Memory: Beyond the Basic Similarity Search

Why Basic Similarity Search Falls Short

The standard agent memory implementation embeds a query, searches a vector store for the top-k similar memories, and injects them into context. This works for simple cases but fails systematically in production. Semantic similarity alone cannot distinguish between memories that are conceptually related but temporally irrelevant, between facts that were true last month and preferences that changed last week, or between memories belonging to different users in a multi-tenant system.

Production agent memory requires five capabilities beyond similarity search: metadata filtering for precision, hybrid search for exact entity recall, temporal awareness for freshness, importance scoring for relevance weighting, and namespace isolation for multi-tenant safety. Each of these is a design decision, not a default.

The Top-3 Failures of Basic Memory Implementations

• Stale memory surfacing: The agent retrieves memories from a previous session where the user had different preferences or constraints. The memories are semantically relevant but temporally outdated. Without TTL or recency scoring, stale memories pollute current context.
• Entity confusion: 'Tell me about Project Alpha' retrieves memories about any project with similar embedding patterns, Project Alpha from 2 months ago and Project Aleph from last week both rank highly. Semantic similarity can't distinguish exact entity names. You need keyword search for this.
• Cross-user contamination: In multi-tenant systems, a memory from User A about 'my preferred report format' may match a query from User B looking for 'report format preferences'. Without namespace filtering, the similarity search returns the wrong user's memories.

Vector Store Architecture for Agents

Dual-Pipeline Design

The indexing pipeline runs offline or asynchronously: it processes raw memory items, chunks them appropriately, embeds them, and writes them to the vector index with metadata. The query pipeline runs online on every memory retrieval: it embeds the query, applies metadata filters, performs ANN search, and reranks the results. These two pipelines have completely different latency requirements and should be optimized independently.

Memory Item Schema Design

• content: The raw text content to be embedded and retrieved. Should be a self-contained, semantically complete unit, a fact, preference, decision, or event description.
• embedding: The dense vector representation. Stored alongside the content in the vector index.
• namespace: Hierarchical isolation key: tenant_id.user_id.memory_type. Never allow cross-namespace retrieval.
• memory_type: Enum, preference, fact, decision, episode, constraint. Used for type-based filtering and different retrieval strategies per type.
• importance_score: Float 0–1. Explicitly set by the consolidation process based on how significant this memory is. Used for score weighting during retrieval.
• created_at / updated_at: Timestamps. Critical for temporal filtering and recency scoring.
• source_session: Which session this memory came from. Enables session-scoped retrieval.
• ttl_expires: Optional expiry timestamp. Memories with expiry dates are automatically excluded from retrieval after expiry.

Embedding Model Selection

The embedding model is the foundation of your vector memory system. A mismatched model, one trained on general text when your agent handles technical documents, or one that's too small for the semantic complexity of your memory items, degrades retrieval quality across the entire system.

Model Selection Criteria

• Domain alignment: text-embedding-3-large and voyage-3 perform well on general text. For code, use code-specific models (voyage-code-2). For medical text, use clinical embedding models. Domain mismatch causes systematic retrieval failures.
• Dimensionality: Higher-dimensional models (3072D) capture more semantic nuance but cost more to store and search. 1536D is a good balance for most agent memory use cases. 768D is sufficient for simple preference memories.
• Consistency: Always use the same model for indexing and querying. Mixing models causes embedding space incompatibility, similar concepts will have low similarity scores because they're encoded in different spaces.
• Query vs document asymmetry: Some models (E5-family, GTE-family) are trained with query: and passage: prefixes. Using the wrong prefix degrades performance significantly. Read the model documentation.
• Cost at scale: At 10,000 memory writes per day × 500 tokens per memory, you're embedding 5M tokens daily. text-embedding-3-large at $0.13/M tokens costs $0.65/day, reasonable. More expensive models compound quickly.

Hybrid Search: Vector + Keyword

When Keyword Search Is Essential

Keyword search is essential for any memory retrieval involving exact entity names, IDs, product names, people's names, or technical identifiers. If a user says 'what did we decide about Project Avalanche?', the word 'Avalanche' must be matched exactly, vector search may retrieve memories about any project with a similar description. BM25 retrieves exact matches regardless of semantic distance.

RRF Fusion Implementation

def reciprocal_rank_fusion(
    vector_results: list[tuple[str, float]],  # (id, score)
    keyword_results: list[tuple[str, float]],
    k: int = 60,
    vector_weight: float = 0.7,
    keyword_weight: float = 0.3,
) -> list[tuple[str, float]]:
    """Fuse vector and keyword search results using RRF scoring."""
    scores: dict[str, float] = {}

    # Vector scores, weighted by position rank
    for rank, (doc_id, _) in enumerate(vector_results):
        scores[doc_id] = scores.get(doc_id, 0) + vector_weight / (k + rank + 1)

    # Keyword scores, weighted by position rank
    for rank, (doc_id, _) in enumerate(keyword_results):
        scores[doc_id] = scores.get(doc_id, 0) + keyword_weight / (k + rank + 1)

    # Sort by fused score descending
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Metadata Filtering for Precision Retrieval

Metadata filtering applies hard constraints that vector similarity cannot enforce. It's the mechanism that prevents cross-user contamination, stale memory surfacing, and irrelevant memory type retrieval. Every production memory retrieval should include at minimum a namespace filter and a temporal filter.

Essential Filters for Every Query

• Namespace filter: WHERE namespace = '{tenant_id}.{user_id}'. This is non-negotiable. Never retrieve without this filter.
• TTL filter: WHERE (ttl_expires IS NULL OR ttl_expires > NOW()). Automatically excludes expired memories without manual cleanup.
• Importance threshold: WHERE importance_score >= 0.3. Filters out low-quality, incidental memories that add noise without value.
• Memory type filter (optional): WHERE memory_type IN ('preference', 'constraint'), use when only specific memory types are relevant to the current task.
• Temporal recency (optional): WHERE created_at > NOW() - INTERVAL '30 days', use for frequently-changing information like preferences and constraints that may be superseded.

Indexing Strategies for Agent Memory

The indexing strategy, how memory items are chunked and processed before embedding, determines the granularity of retrieval. Too coarse and you retrieve large memory blocks that contain irrelevant context. Too fine and individual sentences lack enough context to be useful on their own.

Memory-Specific Chunking

Unlike document RAG, agent memory items should be chunked at the semantic unit level rather than the token-count level. A user preference is one memory item. A decision made in a session is one memory item. A fact learned about the user's domain is one memory item. Each item should be self-contained and independently retrievable without needing adjacent items for context.

HNSW Index Configuration

• M parameter (connections per node): Higher M = better recall, more memory. Default 16 is good for < 1M items. Use M=32 for > 1M items where recall quality matters more than index size.
• ef_construction: Controls index build quality. Higher = better recall at query time, slower build. Default 200 is good. Don't go below 100.
• ef_search: Controls recall vs speed tradeoff at query time. Higher = better recall, slower queries. Set to 2–3x your top-k value.
• Distance metric: Use cosine similarity for normalized embeddings (most text embeddings). Use L2/Euclidean for raw embeddings that aren't normalized.

Reranking for Better Recall Quality

ANN search trades recall precision for speed. The top-k results from a vector search are approximately the most similar, not exactly. Reranking applies a more precise scoring model to these candidates to identify the truly most relevant memories before injecting them into context.

Two-Stage Retrieval Pattern

Stage 1: ANN search retrieves 20–50 candidate memories at high speed. Stage 2: A cross-encoder reranker scores each candidate against the query text with full attention, it sees both the query and memory item together, producing much more precise relevance scores. Keep only the top 3–8 after reranking. This two-stage approach gives you the speed of ANN search with near-perfect precision at the final retrieval stage.

Memory Updates and Deletions

Memory items change over time: user preferences shift, facts become outdated, decisions are reversed. A memory system that can only add items becomes unreliable as it ages. Updates and deletions are essential maintenance operations.

Update Strategies

• Versioned append: Never modify existing memory items. Create a new version with an updated timestamp. During retrieval, return only the most recent version of each memory key. Older versions are available for audit but not returned.
• Hard delete with re-embed: Delete the old item and insert a new one. Simpler but loses history. Appropriate for preferences where the old value is never useful.
• Soft delete with TTL: Mark old items as superseded and set a short TTL. They expire automatically without a delete operation. Good for high-write memory systems.
• Contradiction detection: Before writing a new memory, check for existing memories that contradict it. Flag contradictions for the agent to resolve rather than silently overwriting.

Right to Erasure (GDPR)

When a user requests deletion of all their data, every memory item in their namespace must be deleted, from the vector index, from any summaries, from cold storage archives, and from audit logs (after the legally required retention period). Design your namespace schema so a single namespace prefix covers all of a user's data. Deletion by namespace prefix is a first-class operation, not an afterthought.

Vector Store Technology Selection

Technology Comparison Matrix

• Pinecone: Fully managed, simple API, auto-scaling. Best for teams that want zero infrastructure management. Expensive at scale. Good metadata filtering support.
• Weaviate: Self-hosted or managed. Excellent hybrid search support built-in. Schema-based with rich metadata. Strong multi-tenancy support. Best for complex memory schemas.
• Qdrant: Self-hosted, high performance, excellent filtering. Native payload indexing for metadata. Best for high-write workloads with complex metadata queries.
• pgvector: PostgreSQL extension. Best for teams already on PostgreSQL who want to add vector search without new infrastructure. Limited ANN algorithm choice but excellent SQL integration for metadata.
• Chroma: Open-source, easy setup. Best for development and small deployments. Not recommended for production > 100K items, lacks enterprise features.

Production Patterns and Anti-Patterns

Anti-Pattern 1, Embedding Everything

Not every piece of text needs to be in the vector store. Tool call logs, intermediate reasoning steps, and verbose API responses add noise to memory retrieval without providing retrievable value. Only embed semantically meaningful, self-contained items that represent something the agent would benefit from recalling in a future session.

Anti-Pattern 2, No Importance Scoring

Without importance scoring, all memories are treated equally. A trivial mention ('the user said hello') ranks the same as a hard constraint ('never recommend product X to this user'). Importance scoring ensures that high-value memories surface reliably while low-value ones don't pollute context.

Anti-Pattern 3, Injecting Too Many Memories

More memories injected into context is not better. 10 loosely-relevant memories add more noise than signal. The sweet spot is 3–6 high-relevance memories. After reranking, filter aggressively, only inject memories with reranked score > 0.75. The model reasons better with a few high-quality memories than with many mediocre ones.

Pattern, Structured Memory Injection

Don't inject raw memory text, inject structured memory blocks. Format retrieved memories as a clearly labeled section in the system prompt: '[RELEVANT MEMORIES] Preference (created 3d ago): User prefers concise bullet-point answers. Constraint (created 1w ago): Never recommend vendor Y.' The structured format helps the model distinguish memories from other context.