Agent Restart and Failure Recovery Logic That Actually Works

Why Agent Restart Is Hard

A stateless web server can restart cleanly, it has no state to restore. A stateful agent that has been running for 20 minutes has accumulated substantial state: context history, working memory, tool results, in-progress reasoning, partially completed sub-tasks. A naive restart loses all of this and forces the agent to redo expensive work, potentially causing harm by replaying side-effectful operations.

The challenge is compounded by the nature of agent failures. Stateless service failures are typically binary (up or down). Agent failures are more complex: the agent may have completed steps correctly before failing, or it may have failed partway through a step with uncertain completion status, or the failure may have left external resources in an inconsistent state. Each scenario requires a different recovery approach.

The Three Restart Scenarios

• Clean failure: The agent crashed before starting any step, or before any side effects occurred. Clean restart from the beginning is safe.
• Mid-step failure: The agent crashed during step N with uncertain completion status. The step may have partially executed. Recovery must determine completion status before proceeding.
• Post-step failure: The agent completed step N successfully but crashed before saving the checkpoint. The step is done but the agent doesn't know it. Recovery must detect this and not re-execute step N.

The Restart State Machine

A well-designed agent restart system operates as an explicit state machine. Every state transition is logged, every transition condition is explicitly defined, and the machine's current state is persisted so that recovery can resume from the correct point.

State Definitions

• RUNNING: Agent is actively executing. The agent process is alive and making progress. Periodic heartbeats confirm liveness.
• SUSPENDED: Agent has been intentionally suspended (checkpoint saved, process stopped). Awaiting restart signal. All state is persisted and the agent can resume cleanly.
• RECOVERING: Restart has been triggered. The agent is loading its last checkpoint, verifying state consistency, and preparing to resume execution.
• COMPLETED: The agent has successfully completed its task. Final output has been produced and delivered. State can be archived.
• FAILED: The agent has exhausted its restart budget or encountered an unrecoverable error. Requires human intervention or escalation to a fallback workflow.

Transition Conditions

• RUNNING → SUSPENDED: Graceful shutdown signal received, or checkpoint-based suspension triggered by budget threshold or phase boundary.
• RUNNING → RECOVERING: Process crash detected (heartbeat missed), or unhandled exception caught at the top-level execution loop.
• SUSPENDED → RECOVERING: Restart signal received (timer, user trigger, or system event).
• RECOVERING → RUNNING: State successfully loaded and verified, no inconsistencies detected.
• RUNNING → FAILED: max_restart_budget exhausted, or unrecoverable error detected during recovery.
• RUNNING → COMPLETED: All pipeline steps completed successfully, final output produced.

State Reconstruction Strategies

When an agent restarts, it must reconstruct enough state to resume without repeating completed work. The reconstruction strategy determines how faithfully state is restored and how quickly the agent can resume.

Full State Restoration

The most robust strategy: persist the complete agent state at each checkpoint (step outputs, working memory, context summary, tool states) and restore it fully on restart. The agent resumes exactly where it left off. Requires careful serialization of all state components. High storage cost but minimal recovery time.

Step-Output Restoration

A lighter strategy: persist only the outputs of completed steps, not the full working state. On restart, load the step outputs and reconstruct working memory by re-running any pure-computation steps. Only steps with side effects are truly skipped. Some re-computation occurs, but no side effects are replayed.

Task Specification Restart

The lightest strategy: preserve only the original task specification and the list of completed steps. On restart, the agent re-discovers the context from the task spec but skips completed steps. Only viable when step re-execution is cheap and side effects are well-protected by idempotency.

Preventing Side Effect Replay

The most critical invariant in agent restart is: side effects must not replay. If an email was sent in step 3, recovery must not send it again. If a payment was processed in step 5, recovery must not process it again. Preventing replay requires tracking the completion status of every side-effectful step and enforcing idempotency on all external operations.

The Completion Log Pattern

Maintain an immutable completion log: a persistent record of every side-effectful operation that has been executed, keyed by the step and the operation's idempotency key. Before executing any side-effectful operation during recovery, check the completion log. If the operation is already in the log, skip execution and load the previously stored result instead.

Two-Phase Commit for Critical Operations

For the highest-stakes operations (financial transactions), use a two-phase commit pattern: first write a 'pending' record to the completion log, then execute the operation, then update the log to 'completed'. On restart, if a 'pending' record is found without a 'completed' record, the agent must query the external service to determine whether the operation completed before failing. Only if the external service confirms the operation did NOT complete should it be re-attempted.

Classifying Restart Triggers

Different restart triggers require different recovery responses. A process crash has different implications than a graceful suspension, which has different implications than an explicit restart request.

Restart Trigger Matrix

• Process crash (SIGKILL, OOM, hardware failure): Most dangerous. The agent may have been in the middle of a write when it crashed, potentially corrupting the checkpoint. Recovery must verify checkpoint integrity before loading.
• Unhandled exception: The agent code itself threw an exception. The state is likely consistent but the step that caused the exception failed. Recovery should skip that step if it was idempotent, or apply remediation if not.
• Timeout exceeded: The agent exceeded its maximum runtime budget. All state is consistent. Recovery must assess whether to resume with a tighter budget or abort.
• Graceful shutdown: The agent was cleanly suspended (infrastructure update, scaling event). All state is consistently checkpointed. Clean recovery with full state restoration.
• Manual restart: Operator triggered the restart for debugging or remediation. Full state restoration, but the operator should review the state before resuming.

Memory and Context Recovery

Recovering the agent's context window is one of the most delicate parts of restart. The context contains the agent's understanding of the task, the decisions made, and the information gathered. Losing context means the agent effectively has amnesia.

Context Recovery Options

• Restore from checkpoint summary: Load the compressed context summary from the last checkpoint. The agent resumes with a compressed but accurate view of prior history. The summary must explicitly include all decisions, constraints, and tool results needed for remaining steps.
• Reconstruct from step outputs: Rebuild the context by reading all completed step outputs in order. More accurate than a summary but more expensive to reconstruct. Required when the context summary would be insufficient for the remaining steps.
• Inject restart notice: Add a special message to the reconstructed context: '[RECOVERY: Agent restarted from checkpoint at step N. All prior steps completed successfully. Resuming from step N+1.]' This orients the agent within the recovered context.

Tool State Recovery

Tools with stateful sessions (authenticated sessions, database connections, streaming API subscriptions) must have their state recovered explicitly. An agent that recovers its context but not its tool authentication state will fail on the first tool call.

Tool State Recovery Per Type

• Authentication tokens: Store tokens in the checkpoint with their expiry time. On recovery, check expiry before loading, expired tokens require re-authentication before the tool is available.
• Database connections: Database connections cannot be serialized. On recovery, establish a new connection pool. If the previous session had a transaction open at crash time, check for any orphaned transactions and roll them back.
• Paginated API position: If the agent was paginating through results when it crashed, store the cursor or page token in the checkpoint. Resume pagination from that cursor rather than starting over.
• Streaming subscriptions: Re-establish streaming subscriptions from scratch on recovery. Use a 'since' timestamp or sequence number to receive only events that occurred after the last processed event.

Restart Coordination in Multi-Agent Systems

When an agent restarts in a multi-agent system, its restart affects its peers. Worker agents that restart must re-register with the orchestrator. An orchestrator that restarts must reconcile the state of all worker agents. Coordination protocols prevent the system from entering an inconsistent state during individual agent recoveries.

Worker Agent Restart Protocol

• Step 1, Re-register: On restart, the worker registers with the orchestrator and reports its current state (last completed step, pending work item).
• Step 2, State reconciliation: The orchestrator compares the worker's reported state against its own records. Identifies any discrepancy (orchestrator thinks step N is done; worker reports it failed).
• Step 3, Work assignment: The orchestrator either resumes the worker's interrupted work item or assigns a new one, depending on the reconciliation result.
• Step 4, Confirmation: Worker confirms receipt of the work assignment and begins execution. Orchestrator marks the worker as active.

Orchestrator Restart Protocol

Orchestrator restart is more complex because it must reconcile the state of all active workers. On restart, the orchestrator loads its task state from the checkpoint, polls all worker agents for their current status, reconciles discrepancies, and resumes task assignment. Workers must be designed to tolerate a period of orchestrator unavailability without stalling, they should continue their current task and queue results until the orchestrator is back.

Restart Budgets and Escalation

Without a restart budget, an agent in a failure loop can restart indefinitely, consuming resources and potentially amplifying the underlying problem. A restart budget limits how many times an agent can restart before escalating.

Restart Budget Design

• Maximum restarts: Allow a fixed number of restarts (typically 3–5) before marking the agent FAILED. This prevents runaway restart loops.
• Restart backoff: Apply backoff between restarts (1 minute, 5 minutes, 15 minutes). This gives transient conditions time to resolve before the next restart consumes budget.
• Time-windowed budget: Rather than a lifetime budget, use a time-windowed budget (5 restarts in 1 hour). This allows legitimate restarts after long periods of stability without restricting short-term recovery.
• Budget escalation: When the restart budget is exhausted, escalate to a human operator or a higher-level recovery workflow (retry with a different model, route to a different agent, notify the user).

Production Restart Patterns

Production restart logic needs to handle the messy reality of distributed systems: network partitions during checkpoint saves, race conditions between multiple restart attempts, and partial state from crashed writes.

Checkpoint Integrity Verification

Before loading a checkpoint, verify its integrity: check that the checksum matches, that all required fields are present, and that the serialized step outputs can be deserialized without errors. A corrupt checkpoint must not be loaded, fall back to the previous valid checkpoint or perform a clean restart rather than loading corrupted state.

Restart Lock to Prevent Concurrent Recovery

In distributed environments, multiple recovery attempts for the same agent can run concurrently. Use a distributed lock (Redis SETNX or database advisory lock) to ensure only one recovery process runs at a time for each agent session. The first process to acquire the lock runs recovery; all others back off and check for completion.

Recovery Observability

• Log every restart event with: trigger type, restart count within budget, steps completed before failure, estimated recovery cost (compute + API calls).
• Track restart_count_per_session as a key metric. Sessions with high restart counts indicate systemic instability that needs investigation.
• Alert when a session enters FAILED state. This is a user-impact event requiring immediate attention.
• Store the complete restart history with each session. Post-incident analysis of why sessions needed multiple restarts reveals patterns that prevent future incidents.