Designing Agent Memory for Multi-User Systems Without Cross-Contamination

The Cross-Contamination Risk

In a multi-tenant agent platform, cross-contamination, one user's memories influencing another user's agent responses, is both a privacy violation and a correctness failure. It's a privacy violation because it exposes one user's data to another. It's a correctness failure because the agent is using the wrong user's context, producing responses that are wrong for the actual user.

Cross-contamination is surprisingly easy to introduce accidentally. Any vector store query without an explicit namespace filter will retrieve memories from any user whose content is semantically similar to the current query. 'What's the status of Project Alpha?' from User A could retrieve memories about 'Project Aleph' from User B if namespace filtering isn't enforced at every query.

How Cross-Contamination Happens in Practice

• Missing namespace filter: A developer adds a new memory retrieval path and forgets to include the tenant_id filter. All queries on this path retrieve from the global pool.
• Incorrect namespace construction: A bug causes the namespace to be constructed incorrectly, wrong user_id, missing tenant prefix, or namespace collision between tenants.
• Shared embedding space without filtering: Two users with similar queries get each other's memories because the embedding similarity is high and the namespace wasn't filtered.
• Cached cross-user response: A response generated for User A is cached without proper cache key scoping and served to User B whose query is semantically similar.
• Migration error: During a data migration, memory items are written without proper namespace tags, making them unscoped and retrievable by any user.

Namespace Design for Multi-Tenant Memory

Namespace Hierarchy Levels

• Level 1, Tenant namespace: The outermost isolation boundary. All data for a tenant shares a prefix. Tenant isolation is the most critical, cross-tenant contamination is a severe security incident.
• Level 2, User namespace: Within a tenant, each user's memories are isolated. Cross-user contamination within a tenant is a privacy violation but less severe than cross-tenant.
• Level 3, Memory type namespace: Preferences, facts, episodes, and constraints are sub-namespaced. Enables type-specific retrieval without mixing types unnecessarily.
• Level 4, Item ID: Each individual memory item has a unique ID within its namespace. Enables precise operations (update, delete, retrieve) on specific items.

Namespace Key Construction

Namespace keys should be validated at construction time, not assumed to be correct. Build a NamespaceBuilder class that validates each component before assembling the key. Reject invalid component values (empty strings, null values, values containing the namespace separator). This prevents a class of bugs where malformed namespace keys bypass isolation.

Isolation Architecture Options

There are three levels of memory isolation, ranging from namespace-level filtering in a shared store to fully dedicated infrastructure per tenant. The right choice depends on your compliance requirements, scale, and operational capacity.

Option 1, Namespace Isolation (Shared Store)

All tenants share the same vector store and relational database. Isolation is enforced entirely through namespace filtering on every query. Cheapest to operate, simplest to scale. Appropriate for most multi-tenant products. The risk: a query without the namespace filter leaks data. Mitigate with mandatory filter enforcement in your query layer.

Option 2, Separate Collections per Tenant

Each tenant has their own collection (or index) in the shared vector store, but shares the database infrastructure. Isolation is enforced at the collection level, queries against Tenant A's collection physically cannot return Tenant B's data. Higher operational complexity (manage N collections), but stronger isolation guarantees. Best for regulated industries where row-level security isn't sufficient.

Option 3, Dedicated Infrastructure per Tenant

Each enterprise tenant gets their own dedicated vector store, database, and storage. Complete data isolation by infrastructure boundary. Highest cost and operational complexity. Required for tenants with strict data residency requirements or who cannot share infrastructure for contractual reasons. Typically reserved for large enterprise customers.

Query Scoping and Enforcement

Query scoping ensures that every memory retrieval includes the appropriate namespace filter. The most reliable approach is to enforce scoping at the query layer, not at the application layer, the database or vector store enforces the filter, not the calling code.

Mandatory Filter Enforcement

from dataclasses import dataclass
from typing import Any

@dataclass
class ScopedMemoryStore:
    """
    Wrapper around a vector store that enforces namespace scoping
    on every query. Prevents accidental unscoped queries.
    """
    _store: Any  # underlying vector store client
    _tenant_id: str
    _user_id: str

    @property
    def namespace(self) -> str:
        return f"{self._tenant_id}.{self._user_id}"

    async def search(
        self,
        query: str,
        memory_types: list[str] | None = None,
        top_k: int = 10,
        min_score: float = 0.75,
    ) -> list[dict]:
        # Namespace filter is ALWAYS applied, not optional
        filters = {"namespace": {"$eq": self.namespace}}
        if memory_types:
            filters["memory_type"] = {"$in": memory_types}

        results = await self._store.search(
            query=query,
            filter=filters,
            top_k=top_k,
        )
        return [r for r in results if r["score"] >= min_score]

    async def write(self, content: str, memory_type: str, **metadata) -> str:
        # Namespace is ALWAYS included in the write, not optional
        return await self._store.upsert(
            content=content,
            namespace=self.namespace,
            memory_type=memory_type,
            **metadata,
        )

def get_scoped_store(tenant_id: str, user_id: str) -> ScopedMemoryStore:
    """Factory: always create a scoped store, never expose the raw store."""
    raw_store = get_raw_vector_store()  # internal only
    return ScopedMemoryStore(_store=raw_store, _tenant_id=tenant_id, _user_id=user_id)

Defense in Depth

Namespace filtering should be enforced at multiple layers, not just one. Layer 1: application code uses ScopedMemoryStore, making unscoped queries impossible in normal code paths. Layer 2: database/vector store row-level security based on the authenticated tenant claim in the request. Layer 3: network isolation separating tenant data at the infrastructure level for enterprise tiers.

Memory Access Audit Trail

What the Audit Trail Enables

• Forensic debugging: 'Why did the agent say X in session Y?' → trace all memory reads that session to understand what context was injected.
• GDPR compliance verification: 'Prove that user Z's data is not being read by any other user's queries.' → audit log shows every read with the requesting user_id.
• Anomaly detection: Alert when a single agent reads more than N memories per minute (possible data scraping), reads memories outside its assigned user's namespace (isolation failure), or reads memories at unusual hours.
• Right-to-erasure audit: After deleting a user's data, the audit log can verify that all reads of their data stopped after the deletion timestamp.

Shared vs Private Memory Partitions

Not all memory needs to be private. Some memory should be shared within a tenant, domain knowledge that's relevant to all users, shared project context, team preferences. Designing shared and private memory partitions allows knowledge sharing within appropriate boundaries.

Shared Memory Use Cases

• Tenant-wide knowledge: Product documentation, organizational structure, shared policies. All users in a tenant benefit from this. Stored at the tenant namespace level, not user level.
• Team-shared context: A project that multiple users are collaborating on. Context about the project is shared across team members. Stored at a team namespace level.
• Anonymized preference insights: Aggregate preference patterns (not individual user data) that can improve recommendations for all users in the tenant. Stored at the tenant level with aggregation to prevent individual identification.

Shared Memory Safety Rules

• Shared memories must never contain PII from individual users. Shared context about projects is fine; shared context containing specific user statements is not.
• Write permission to shared memory should be restricted. Not every user should be able to modify shared tenant knowledge. Implement role-based write permissions.
• Shared memory changes should be audited with special scrutiny. A change to shared memory affects all users, the impact is wider than a private memory change.

GDPR and Privacy Compliance

Data Minimization

GDPR requires collecting only the minimum data necessary. Apply this principle to memory: only store memories that meaningfully improve agent quality. Preferences that the agent won't use, observations that won't be retrieved, and ephemeral conversational content have no value in long-term memory and should not be stored. Build explicit criteria for what qualifies as worth storing.

Privacy by Design Checklist

• Memory disclosure: Users can see all memories stored about them via a 'what do you know about me?' query or a memory dashboard.
• Memory correction: Users can correct inaccurate memories without losing their other memories.
• Memory deletion: Users can delete specific memories or all memories via a clear process.
• Memory portability: Users can export their memories in a machine-readable format.
• Processing lawfulness: Document the legal basis for storing each category of memory (consent, legitimate interest). This documentation is required for GDPR compliance.
• Data residency: Know where each memory is stored. For EU users, memories may need to stay in the EU. Your namespace design should support geo-partitioned storage.

Testing Memory Isolation

Isolation Test Suite

• Cross-user isolation test: Write a unique memory for User A. Query as User B with a semantically similar query. Verify User B receives zero results from User A's namespace.
• Cross-tenant isolation test: Write a unique memory for Tenant A, User 1. Query as Tenant B, User 1 with an identical query. Verify complete isolation between tenants.
• Namespace construction test: Test namespace key construction with edge cases, empty strings, SQL injection attempts, extremely long values, unicode characters.
• Unscoped query prevention test: Attempt to call the underlying vector store directly without the ScopedMemoryStore wrapper. Verify that this code path either doesn't exist or requires special authentication.
• Deletion completeness test: Delete all memories for User A. Verify that subsequent queries as User A return empty results across all memory systems (Redis, PostgreSQL, vector DB, S3 archive).

Incident Response for Memory Leaks

Despite best practices, memory isolation incidents can occur. Having a defined incident response process ensures that a leak is contained quickly and users are notified appropriately.

Incident Response Steps

• Detection: Automated anomaly detection alerts on cross-namespace reads. Manual detection through user reports or security audits.
• Containment: Immediately disable the affected query path. If the leak is widespread, temporarily disable memory retrieval entirely until the fix is deployed.
• Scope assessment: Use the audit log to determine exactly which users' data was exposed to which other users, for what time period.
• Remediation: Fix the isolation bug. Delete any cross-contaminated memories from affected user namespaces. Verify the fix is complete with isolation tests.
• Notification: Notify affected users of the breach within the regulatory-required timeframe (72 hours under GDPR). Provide a clear description of what was exposed.
• Post-incident: Document root cause and what controls failed. Update isolation tests to prevent recurrence. Brief engineering team on lessons learned.

Isolation Performance Considerations

Namespace filtering adds a constraint to every query. In some vector store implementations, filtered queries are significantly slower than unfiltered queries, the filter may prevent the use of the most efficient ANN index path. Benchmark your filtered query performance before deploying at scale.

Performance Optimization for Filtered Queries

• Pre-filter with metadata index: Create a database index on the namespace field. Pre-filter candidate IDs using the database index, then run ANN search only on those IDs. This is typically faster than post-filtering.
• Per-tenant collections: For very large tenants, a dedicated collection eliminates the namespace filter overhead by providing physical isolation.
• Benchmark with realistic data distribution: Performance characteristics depend heavily on how many items each namespace contains and how those namespaces are distributed. Test with realistic production data distributions, not toy datasets.