Caching LLM Responses Safely: Speed Without Stale Answers

Why Cache LLM Responses

LLM API calls are the most expensive and slowest operation in most agent pipelines. A typical GPT-4o call costs $0.005–0.025 and takes 1–8 seconds. For high-traffic agents handling thousands of requests per day, this cost compounds quickly. Caching addresses both problems simultaneously: a cache hit costs a fraction of a cent in Redis lookup time and returns in under 10 milliseconds.

The challenge is correctness. A cached answer that was accurate last week may be wrong today if the underlying data has changed. A cached response generated for one user's context may be subtly wrong for another user's context. Caching LLM responses safely requires understanding what makes an answer stable enough to cache and what makes it volatile enough to require freshness guarantees.

Cacheable vs Non-Cacheable Response Types

• HIGHLY CACHEABLE: FAQ answers, product feature explanations, general knowledge responses, static policy descriptions. These have high query repetition and low volatility.
• MODERATELY CACHEABLE: Analysis of static documents, code explanations, summarizations of unchanged content. Cache with medium TTL and event-based invalidation on document updates.
• CACHE WITH CAUTION: Personalized responses (requires user-scoped keys), responses incorporating real-time data (must invalidate on data changes), responses that depend on model version (invalidate on model updates).
• NEVER CACHE: Responses involving live market data, current news, real-time inventory, or session-specific information. These require freshness guarantees that caching cannot provide.

Multi-Tier Cache Architecture

L1, Exact Match Cache (Redis)

The L1 cache handles exact repeats: identical prompt, identical model, identical parameters. Key: SHA-256 hash of (system_prompt + user_message + model + temperature). Value: the full response JSON with metadata. TTL: 1–24 hours depending on content type. Expected hit rate: 20–35% for typical agent workloads. Response time: 1–5ms.

L2, Semantic Cache (Vector DB)

The L2 cache handles semantic equivalents: different phrasing, same meaning. Embed the user query, search a vector index for similar past queries, return the cached response if cosine similarity exceeds the threshold (typically 0.92). This catches cases like 'how do I reset my password?' and 'password reset steps please', different tokens, same intent, same correct answer.

L3, Provider Prompt Cache

Anthropic and OpenAI both offer server-side prompt caching for repeated system prompts. When the same system prompt prefix is sent repeatedly, the provider caches the internal representation and charges reduced input token rates (Anthropic: 10% of standard input price for cache hits). This is the cheapest form of caching, it's automatic and requires only that your system prompt is stable across requests.

Semantic Caching, Beyond Exact Match

Similarity Threshold Calibration

The similarity threshold determines the precision-recall tradeoff. A threshold of 0.98 produces very few false positives (cached responses that are wrong for the new query) but misses most semantic equivalents. A threshold of 0.85 catches many more equivalents but risks serving incorrect responses for queries that are similar but not identical in meaning. For production agents, start at 0.92 and adjust based on false positive analysis.

Domain-Specific Threshold Adjustment

Different domains require different thresholds. For factual queries in narrow domains (product specs, policy lookups), 0.95+ is appropriate, small semantic differences may change the correct answer. For conversational queries and general-knowledge questions, 0.88–0.92 provides good cache efficiency without meaningful quality loss.

Cache Key Design

Cache key design determines cache effectiveness. A key that's too coarse serves wrong cached responses. A key that's too fine never gets cache hits. The right key captures everything that affects the response and nothing that doesn't.

Key Component Matrix

• ALWAYS include: User message content (the query itself), model identifier, temperature setting.
• INCLUDE when relevant: System prompt hash (if variable per request), tenant/user ID (if responses are personalized), language setting, response format requirement.
• NEVER include: Request timestamp, request ID, tracing headers, session ID (unless responses are session-specific). These prevent any cache hits for repeated queries.
• CONDITIONAL: Tool call results, include the hash of tool results that were injected into the context, since different tool results should produce different cached responses.

Key Construction Pattern

import hashlib, json

def build_cache_key(
    user_message: str,
    model: str,
    temperature: float,
    system_prompt: str | None = None,
    tenant_id: str | None = None,
    tool_context_hash: str | None = None,
) -> str:
    """Build a deterministic, complete cache key."""
    components = {
        "user_message": user_message.strip(),
        "model": model,
        "temperature": round(temperature, 2),
    }
    if system_prompt:
        components["system_hash"] = hashlib.sha256(
            system_prompt.encode()
        ).hexdigest()[:16]
    if tenant_id:
        components["tenant"] = tenant_id
    if tool_context_hash:
        components["tools"] = tool_context_hash

    canonical = json.dumps(components, sort_keys=True)
    return "llm:v1:" + hashlib.sha256(canonical.encode()).hexdigest()

TTL Strategy by Content Type

Time-to-live (TTL) is the first line of defense against stale cache responses. The right TTL balances cache hit rate (lower TTL = lower hit rate) against staleness risk (higher TTL = higher staleness risk). Different content types warrant dramatically different TTLs.

TTL Recommendations by Content Type

• Static knowledge (historical facts, stable documentation): 24–72 hours. Very low volatility. Long TTL maximizes cache efficiency.
• Product information (specs, features, policies): 4–12 hours. Changes occasionally. Balance freshness with hit rate.
• Pricing and inventory: 15–60 minutes. Changes frequently. Short TTL required. Consider event-based invalidation instead.
• Current events and news: 5–15 minutes or no cache. Changes constantly. Caching provides minimal benefit.
• User-generated content summaries: Cache with the content's last-modified timestamp as part of the key. TTL becomes irrelevant, the key changes when the content changes.

Cache Invalidation Strategies

Event-Based Invalidation Implementation

Event-based invalidation invalidates cache entries when the underlying data changes. When your product catalog is updated, publish a 'catalog_updated' event. Cache listeners receive the event and delete all cache entries that depend on catalog data. This requires maintaining a dependency map: which cache keys are affected by which data sources. More complex to implement but provides the most accurate cache freshness.

Semantic Drift Detection

After a model update, cached responses generated by the old model may no longer represent the new model's behavior. Semantic drift detection runs periodically: sample a random subset of cached queries, regenerate their responses with the current model, and compare to the cached responses. If the average semantic similarity drops below a threshold (0.85), bulk-invalidate the cache for that model version.

Provider-Level Prompt Caching

Anthropic's prompt caching feature reduces the cost of repeated system prompt processing. When you mark a portion of your prompt with cache_control: ephemeral, Anthropic caches the internal representation for 5 minutes. Subsequent requests with the same cached prefix pay only 10% of the standard input token price for the cached portion.

Optimizing for Provider Cache Hits

def build_anthropic_messages_with_caching(
    system_prompt: str,
    tools: list[dict],
    conversation: list[dict],
    user_message: str,
) -> dict:
    """Structure messages to maximize Anthropic prompt cache hits."""
    # Cache the stable system prompt (changes infrequently)
    system = [
        {
            "type": "text",
            "text": system_prompt,
            "cache_control": {"type": "ephemeral"},  # Cache for 5 min
        }
    ]

    # Cache tool definitions (stable across requests for same agent)
    if tools:
        system.append({
            "type": "text",
            "text": f"Available tools: {json.dumps(tools)}",
            "cache_control": {"type": "ephemeral"},
        })

    # Don't cache conversation history (changes every turn)
    messages = conversation + [{"role": "user", "content": user_message}]

    return {"system": system, "messages": messages}
    # Result: system prompt + tools cached = 60-80% of input tokens
    # at 10% of standard price on cache hit

Cache Safety, What Never to Cache

Some response types are unsafe to cache regardless of efficiency gains. Serving a stale cached response for these types causes more harm than the cost and latency of a fresh response.

Absolute No-Cache List

• Security-relevant decisions: 'Is this request authorized?', 'Does this user have access to X?' These must always be fresh, a cached authorization decision from when the user had different permissions is a security vulnerability.
• Real-time financial data: Prices, exchange rates, account balances, portfolio values. Stale financial data can cause harmful user decisions.
• Medical or safety information: Current drug interactions, emergency procedures, safety alerts. The risk of stale information in these domains is too high.
• Personalized recommendations based on live user state: Recommendations driven by a user's current activity, shopping cart, or location require live context.
• One-time operations: Any response that initiates a transaction, sends a communication, or creates a record. Replaying a cached 'I sent your email' is wrong.

Cache Metrics and Monitoring

Essential Cache Metrics

• cache_hit_rate by tier: L1 hit rate, L2 hit rate, L3 hit rate, miss rate. Track separately, L1 misses that hit L2 are still wins.
• cost_savings_per_day: (hits × average_llm_cost) - (hits × cache_cost). Quantifies the financial value of your cache infrastructure.
• freshness_violations: Rate of cache responses that were later found to be incorrect due to staleness. The key quality metric for cache safety.
• cache_latency_p99: Distribution of cache lookup latency. Should be < 10ms for L1. Alert if p99 > 50ms, indicates Redis performance issues.
• eviction_rate: Rate at which entries are evicted before their TTL expires. High eviction indicates insufficient cache memory.

Production Cache Implementation

Implementation Checklist

• Key versioning: Include a cache version in every key (e.g., 'llm:v2:...'). When cache semantics change, bump the version to invalidate all existing entries without touching Redis directly.
• Negative caching: Cache known-bad queries (queries that always fail) with a short TTL. Prevents repeated expensive calls for queries with permanent failures.
• Cache warming: Pre-populate the cache with responses to your most frequent queries before peak traffic. A warm cache handles traffic spikes without the latency spike of cold-start misses.
• Thundering herd prevention: When a popular cache entry expires, multiple simultaneous requests may all miss and trigger concurrent LLM calls. Use probabilistic early expiration or distributed locking to prevent this.
• Cache bypass flag: Implement a no-cache header or query parameter for testing and debugging. Allows cache-free requests without infrastructure changes.