Handling Partial Failures in Agent Pipelines Without Losing Work

The Partial Failure Problem

A seven-step agent pipeline that fails at step four has a fundamentally different recovery challenge than a stateless API endpoint that fails. Steps one through three completed successfully, they may have written database records, sent API requests, consumed tokens, and made decisions that the remaining steps depend on. A naive restart from step one replays those completed steps unnecessarily, wastes compute, and risks duplicate side effects.

The engineering challenge of partial failure recovery is threefold: accurately determining which steps completed successfully, preserving their outputs in a form that later steps can consume after a restart, and ensuring that the recovery path correctly distinguishes between what needs to be redone vs what needs to be resumed from.

Why Partial Failures Are More Common Than You Expect

• Long-running pipelines cross multiple network boundaries, each with its own failure probability. A 10-step pipeline with 99.9% reliability per step has a 99% chance of completing, meaning 1% of runs will experience at least one step failure.
• External services degrade independently. A database that was healthy at step 2 may have a transient failure at step 6. The pipeline must handle this without losing the step 1–5 results.
• Infrastructure events (deploys, autoscaling, spot instance preemption) interrupt agents mid-run. Cloud-native deployments must treat mid-run interruptions as a first-class case, not an edge case.
• LLM context overflow can occur mid-pipeline. A pipeline that was building context step by step may hit the limit at step N, requiring recovery that includes context compression rather than a full restart.

Recovery Decision Flow

The recovery decision starts with failure classification. The same pipeline failure at step 4 requires three entirely different responses depending on the failure type: a transient timeout means retry from the checkpoint; a permanent 404 means abort the pipeline and return whatever was completed; a degradable tool failure means use an alternative tool and continue.

Decision Matrix by Failure Type

• Transient + low cost steps: Retry from last checkpoint. If the cost to redo completed steps is low, always checkpoint before the failure point.
• Transient + high cost steps: Retry from last checkpoint ONLY. Never restart from the beginning when expensive steps (long LLM calls, external API queries, database operations) have already completed.
• Permanent at a non-critical step: Skip the failed step, mark as degraded, continue with remaining steps. Return a partial result flagged as incomplete.
• Permanent at a critical step: Abort the pipeline. Save all completed step outputs to storage. Return a structured partial result that includes everything completed so far.
• Degradable: Use the fallback path for the failed step (alternative tool, cached result, simplified logic) and continue. The output quality may be lower, flag this in the result.

Checkpoint Architecture

A checkpoint is a persistent snapshot of the agent's state at a known-good point in the pipeline. Effective checkpointing requires: saving the right content (complete, resumable state), at the right frequency (after each significant step), in the right storage (fast write, reliable read), with the right metadata (session ID, step number, timestamp).

Checkpoint Frequency Strategy

• Checkpoint after every step that has significant side effects: If a step sent an email, wrote to a database, or made an API call with real-world consequences, always checkpoint after it succeeds.
• Checkpoint after expensive compute steps: If a step consumed thousands of tokens on a complex reasoning task, checkpoint the result. Never redo expensive computation if a cheap checkpoint prevents it.
• Checkpoint before risky steps: Save state before any step that has high failure probability or irreversible consequences.
• Checkpoint at natural task phase boundaries: When the pipeline transitions between phases (research, planning, execution), always checkpoint at the transition point.

What to Include in a Checkpoint

A checkpoint must contain everything needed to resume the pipeline from that point, nothing more, nothing less. Over-capturing wastes storage. Under-capturing requires rerunning steps to reconstruct missing state.

Required Checkpoint Contents

• Step completion map: A record of which steps have been completed and whether they succeeded. Binary per step, with a timestamp for each completion.
• Step output data: The actual output from each completed step, serialized to JSON. This is what subsequent steps will consume. Must be complete and deterministically serializable.
• Context summary: A compressed representation of the conversation context up to this point. Not the full context (too large), a structured summary preserving facts, decisions, and constraints.
• Tool state: The state of any stateful tools (auth tokens, session IDs, connection parameters, cursor positions in paginated APIs). These enable tools to resume without re-authentication.
• Working variables: Any variables the agent has accumulated (search results, extracted entities, computed values) that later steps depend on.
• Task specification: The original task objective and constraints. This must be preserved verbatim, it's the ground truth the entire pipeline serves.

Checkpoint Storage Strategies

Checkpoint storage must be fast to write (not blocking the pipeline), reliable to read (available when recovery is needed), and cost-effective (checkpoints are transient artifacts, not permanent records).

Two-Tier Checkpoint Storage

• Hot tier (Redis): Active session checkpoints. Fast writes (< 5ms), fast reads (< 5ms). TTL of 24 hours. Size limit: keep checkpoints small (< 100KB) by compressing context summaries.
• Cold tier (S3/Blob): Long-running task archives. Checkpoints older than the hot tier TTL, or checkpoints for tasks that may resume days later. Higher latency acceptable for cold recovery scenarios.
• Key schema: checkpoint:{session_id}:{step_number}:{timestamp}. Enables querying the latest checkpoint, a specific step's checkpoint, or all checkpoints for a session.
• Consistency: Use atomic writes (Redis SETNX for idempotent checkpoint creation). Ensure the checkpoint is fully written before the step is marked complete, never mark a step complete without a valid checkpoint.

Resuming Without Replaying Harmful Actions

The most dangerous failure mode in checkpoint recovery is replaying side-effectful steps. If step 3 sent an email and the checkpoint was taken after the email was sent, recovering to step 3's checkpoint and re-running the step would send the email twice. Recovery logic must distinguish between 'redo this step' and 'resume from after this step.'

Side Effect Classification

• Idempotent actions (safe to replay): Read-only database queries, non-destructive API calls, LLM inference, text parsing and transformation. These can be safely replayed if their outputs aren't checkpointed.
• Non-idempotent actions (must NOT replay): Sending emails, charging payment methods, creating database records, posting to social media, sending webhooks. These must be checkpointed and never replayed.
• Conditionally idempotent: Actions that become idempotent with the right design, creating a database record with a unique constraint on an idempotency key, sending a request with an idempotency header.

Recovery Skip Logic

When loading a checkpoint and resuming a pipeline, the recovery logic must iterate through each step in order and check the step completion map. Steps marked as complete must be skipped, their outputs are loaded from the checkpoint rather than recomputed. Only the failed step (and all subsequent steps) are executed. This is the critical invariant: completed steps are never re-executed during recovery.

Returning Partial Results Gracefully

When a pipeline cannot complete all steps, due to a permanent failure that can't be worked around, the most user-friendly response is a structured partial result rather than a generic error. Partial results communicate what was accomplished, what was not, and what the user can do next.

Partial Result Schema

• status: 'partial_complete', distinguishes from 'success' and 'error'. Enables clients to handle partial results differently.
• completed_steps: List of steps that succeeded, with brief output summaries. Shows what was accomplished.
• failed_step: The step that failed, with the failure reason and error code. Enables debugging and manual remediation.
• partial_output: Whatever useful output was produced before failure. May be incomplete but is still valuable to the user.
• recovery_suggestions: What the user or system can do to complete the task. 'Retry with a corrected parameter', 'Contact support', or 'The following steps can be completed manually'.

Idempotency and Partial Failure

Partial failure recovery is only safe when the recovery path is idempotent. If recovering and re-running a failed step can produce duplicate side effects, recovery is worse than no recovery at all. Idempotency must be designed into every step that has external side effects before partial failure recovery is enabled.

Making Recovery Idempotent

• Assign idempotency keys before execution: Generate a unique key for each step execution and pass it to all external calls. If recovery retries the step, the same key prevents duplicate operations at the external service.
• Check completion before acting: At the start of each step, check whether it has already completed (by looking up the step's idempotency key in the dedup store). If it has, load the previous result and continue.
• Store results atomically with the idempotency key: When a step completes, write the idempotency key and the result atomically. A step is not 'done' until both the result AND the idempotency key are persisted.

Testing Recovery Paths

Recovery paths that aren't tested aren't reliable. The most common failure in production is discovering that the recovery logic itself has a bug, typically at the worst possible time, during an actual production incident.

Recovery Test Cases

• Fail-at-each-step tests: For each step N in your pipeline, write a test that injects a failure at step N and verifies that recovery correctly resumes from the last checkpoint without re-executing completed steps.
• Fail-during-checkpoint-write tests: Simulate a failure during the checkpoint write itself. Verify that the agent correctly handles a corrupt or missing checkpoint (by re-executing the step cleanly).
• Double-recovery test: Simulate two consecutive failures at different steps. Verify that the second recovery correctly picks up from the step-2 checkpoint, not the step-1 checkpoint.
• Side-effect replay test: After recovery, verify that all non-idempotent actions (emails, database writes) happened exactly once, not twice.

Observability for Partial Failures

Partial failures must be visible in your observability stack. A pipeline that returns a partial result is not a success, it's a degraded outcome that should be tracked, alerted on, and analyzed.

Key Metrics

• partial_completion_rate: Fraction of pipeline runs that ended in a partial result. Alert when this rises above your baseline, it indicates an upstream reliability problem.
• checkpoint_save_success_rate: Fraction of checkpoint writes that succeeded. A low rate here means recovery is unreliable even if the checkpointing logic is correct.
• recovery_success_rate: Of recovery attempts (pipelines that resumed from a checkpoint), the fraction that successfully completed. Low values indicate problems in the recovery logic itself.
• steps_redone_per_recovery: Average number of steps re-executed during recovery. Should be close to 1 (only the failed step). High values indicate checkpointing is too infrequent.