Idempotency in Agent-Triggered Actions: Preventing Duplicate Operations

What Is Idempotency and Why Agents Need It

An operation is idempotent if performing it multiple times has the same effect as performing it once. GET requests are idempotent, querying a database repeatedly returns the same result without changing anything. POST requests that create records are not idempotent, each POST creates a new record. In agent systems, the distinction is critical because retries are a first-class feature: when a network timeout occurs after an agent sends a payment request but before it receives the confirmation, the agent will retry the request. Without idempotency, the customer is charged twice.

The fundamental challenge is timing. The failure window that creates duplicate operations is the interval between the action being executed and the agent receiving confirmation. During this window, if anything interrupts the agent, network failure, process crash, timeout, the agent has no way to know whether the action completed or not. Idempotency is the mechanism that makes 'try again' safe regardless of what happened in that window.

The Exactly-Once Illusion

True exactly-once execution is impossible in distributed systems. Any message-passing system over a network can experience the 'two generals problem', you can never be certain your message arrived until you receive an acknowledgment, but the acknowledgment itself can be lost. The practical solution is at-least-once execution with idempotency: retry until you get confirmation, but ensure that duplicate executions are harmless.

Non-Idempotent Failure Modes

Understanding the specific failure modes caused by non-idempotent operations helps prioritize which operations to make idempotent first.

High-Stakes Non-Idempotent Operations

• Payment processing: A duplicate charge is a direct financial harm to the user and a legal liability for your company. The single highest priority for idempotency in any agent with payment capabilities.
• Email and notification sending: Duplicate emails erode user trust immediately. 'You sent me 3 order confirmations' is a support ticket and a trust problem.
• Database record creation: Duplicate records corrupt data integrity and require manual cleanup. A duplicate customer account can cause authentication confusion and data duplication.
• Webhook dispatch: Downstream systems receive the same event twice, triggering their processing twice. Cascading duplicates can propagate through an entire integration chain.
• External API mutations: Creating resources in external systems (tickets, calendar events, CRM records) creates duplicates that are visible to end users.
• File uploads: Uploading the same file twice creates storage duplicates and may trigger duplicate processing pipelines.

Idempotency Key Design

An idempotency key is a unique identifier associated with a specific operation intent. If the same operation is attempted multiple times with the same key, only the first execution proceeds; subsequent attempts return the cached result from the first execution.

Key Design Principles

• Deterministic: The same operation must always produce the same key. If the key generation has any randomness, retries will generate different keys and bypass idempotency.
• Unique per intent: Two different operations must never share a key. If 'charge user A $10' and 'charge user A $20' share a key, one of them will be silently skipped.
• Stable across retries: The key must survive the failure scenario being protected against. If the key is stored only in memory, a process crash loses it.
• Appropriately scoped: A key that's too broad (just 'send_email') prevents all emails of that type from being sent again, even legitimate new sends. A key that's too narrow provides no deduplication.

Key Construction Examples

• Payment: sha256(user_id + amount_cents + currency + order_id). Deterministic, unique per payment intent, retry-safe.
• Email: sha256(recipient_email + email_template_id + session_id + step_number). Prevents duplicate sends from the same agent run while allowing new sends from different runs.
• Record creation: sha256(entity_type + canonical_fields_sorted + session_id). Prevents duplicate records from the same operation while allowing genuinely new records.
• Webhook: sha256(event_type + event_timestamp + source_id). Deduplicates repeated webhook dispatches for the same event.

Deduplication Architecture

The deduplication store is the central component of idempotency infrastructure. It records which operations have been executed and stores their results so that duplicates can return the original result without re-executing.

Redis as Deduplication Store

• Atomic SET NX (set if not exists): Redis SET NX provides atomic 'create if not exists' semantics. The first writer wins; subsequent writes for the same key return nil, indicating a duplicate.
• TTL management: Every idempotency key needs a TTL. Too short and retries after the TTL expire lose deduplication protection. Too long and the key store grows without bound. Typical TTL: 24 hours for short-lived operations, 7 days for high-stakes operations.
• Store results alongside keys: The Redis value should include the operation result, not just the key's existence. When a duplicate is detected, return the stored result from the first execution.
• Handle Redis unavailability: If the dedup store is unavailable, fail safe. Don't execute the operation without deduplication, instead, return an error and ask the agent to retry when the store is available.

Provider-Native Idempotency Support

Many API providers offer native idempotency support through idempotency keys passed as request headers. Using provider-native idempotency is always preferable to building your own, it deduplicates at the provider level, meaning even if your dedup store fails, the provider prevents duplicate execution.

Provider Idempotency Headers

• Stripe: Pass Idempotency-Key header. Stripe stores the result for 24 hours and returns the same response for all requests with the same key.
• Anthropic API: Not needed for inference (inference is naturally idempotent). Use for any management API calls.
• SendGrid: Pass X-Message-ID header. SendGrid deduplicates sends with the same message ID within a configurable window.
• Twilio: Pass X-Twilio-Idempotency-Token header. Deduplicates SMS and call requests.
• Custom internal APIs: Implement Idempotency-Key support in all internal APIs that have side effects. This is a non-negotiable engineering standard for any internal service called by agents.

At-Least-Once vs Exactly-Once Delivery

Distributed systems offer at-least-once message delivery: a message may arrive multiple times, but it will arrive at least once. Exactly-once delivery is impossible to guarantee across network boundaries. Understanding this forces a choice: either design operations to be idempotent (making at-least-once delivery equivalent to exactly-once), or accept that some operations may execute twice and design compensating mechanisms.

Designing Compensating Mechanisms

For operations that cannot be made idempotent, implement compensating mechanisms: automatic duplicate detection and cancellation (detect a double charge and automatically refund the second), human review queues for suspected duplicates, customer-visible duplicate activity logs. These mechanisms treat duplicates as recoverable errors rather than fatal failures.

Classifying Actions by Idempotency Requirement

Not all agent actions require idempotency. Classifying actions by their idempotency requirement helps focus implementation effort where it matters most.

Idempotency Requirement Tiers

• CRITICAL (must be idempotent): Payments, charges, refunds. Email/SMS sends. Record creation. Webhook dispatch. File uploads. Data deletion.
• IMPORTANT (should be idempotent): Database updates that change state. External API calls that create or modify resources. Notification sends.
• OPTIONAL (idempotency is a nice-to-have): Read-only queries (naturally idempotent). Cache updates. Analytics events (duplicates are acceptable).
• NEVER (idempotency not applicable): Truly stateless computations with no side effects. These cannot produce duplicates by definition.

Database-Level Idempotency Patterns

Database operations offer several patterns for achieving idempotency at the persistence layer. These complement application-level deduplication and provide a safety net even when the application-level dedup store fails.

Unique Constraint Pattern

Add a unique constraint on the idempotency key column. INSERT with the idempotency key will succeed on the first attempt and fail with a unique constraint violation on duplicates. Catch the constraint violation and return the existing record instead of the error. Simple, reliable, and works without a separate dedup store.

Upsert Pattern

Use INSERT ... ON CONFLICT (idempotency_key) DO NOTHING or DO UPDATE. The DO NOTHING variant makes the insert idempotent, subsequent inserts with the same key are silently ignored. The DO UPDATE variant updates the existing record (useful when the operation should overwrite rather than skip). Both patterns are atomic and require no additional dedup infrastructure.

Event Sourcing Pattern

Store all agent actions as immutable events with idempotency keys. Never mutate existing records, always append new events. Idempotency is enforced by the event store's unique constraint on (event_type, idempotency_key). Current state is derived by replaying the event log. This pattern is particularly well-suited for financial systems where audit trails are required.

Testing Idempotent Agent Actions

Idempotency must be tested explicitly, it's not covered by standard happy-path tests. Idempotency test cases deliberately send the same operation multiple times and verify the expected behavior.

Idempotency Test Patterns

• Duplicate detection test: Send the same action twice with the same idempotency key. Verify the side effect occurred exactly once (one email sent, one record created, one charge processed).
• Result consistency test: Send the same action twice with the same idempotency key. Verify the returned result is identical on both calls.
• Concurrent duplicate test: Send the same action from two concurrent agent instances simultaneously. Verify exactly one execution and no race condition errors.
• Post-TTL test: Wait until the idempotency key's TTL expires, then send the action again. Verify the action executes a second time (TTL expiry is intended, not a duplicate).
• Dedup store failure test: Disable the dedup store, send an action, verify the system fails safely rather than executing without deduplication.

Monitoring for Duplicate Executions

Even with idempotency in place, monitoring for duplicate execution attempts is valuable. High duplicate rates indicate upstream retry logic that's too aggressive, bugs in agent state management, or operational incidents where agents are restarting repeatedly.

Duplicate Detection Metrics

• duplicate_request_rate: Fraction of requests to idempotency-protected operations that are identified as duplicates. Normal values: < 5%. High values: investigate retry configuration and agent restart loops.
• idempotency_key_collision_rate: Rate at which the key generation function produces collisions (two different operations generating the same key). Should be effectively zero, any collision indicates a key design bug.
• dedup_store_hit_rate: Fraction of requests that hit the dedup store and return a cached result. Higher rates indicate more aggressive retries but also more effective idempotency.
• missed_idempotency_incidents: Manual tracking of production incidents where duplicate operations occurred despite idempotency infrastructure. Each incident identifies a gap in coverage.