The key is to treat an AI agent as a durable state machine, not as one long function call.

A good architecture looks like:

User request
    │
    ▼
Durable workflow
    │
    ├── plan ──────────────┐
    │                      │ retry independently
    ├── retrieve context ──┤
    │                      │
    ├── call LLM ──────────┤
    │                      │
    ├── execute tool ──────┤
    │                      │
    └── validate result ───┘
             │
             ▼
        completed state

Systems such as docs.temporal.io are designed around exactly this idea: workflows can resume after crashes, network failures, and infrastructure outages rather than starting over.

1. Put durable checkpoints around meaningful steps

Don't make the entire agent one retryable operation:

def run_agent(request):
    plan = llm(request)
    docs = search(plan)
    answer = llm(docs)
    send_email(answer)

Instead, make each externally observable step independently recoverable:

plan = workflow.execute_activity(
    make_plan,
    request,
    retry_policy=llm_retry,
)

docs = workflow.execute_activity(
    retrieve_documents,
    plan,
    retry_policy=io_retry,
)

answer = workflow.execute_activity(
    generate_answer,
    docs,
    retry_policy=llm_retry,
)

workflow.execute_activity(
    send_result,
    answer,
    retry_policy=side_effect_retry,
)

If generate_answer fails, you want to retry that step, not redo document retrieval and potentially incur another set of API calls.

2. Make retries explicit and bounded

Different failures deserve different policies.

A typical policy:

retry_policy = {
    "initial_interval": 2,
    "backoff_coefficient": 2,
    "maximum_interval": 60,
    "maximum_attempts": 5,
}

Avoid infinite retries. Otherwise a permanently broken dependency can turn into an expensive agent that runs forever.

3. Make every side effect idempotent

This is probably the most important part.

Suppose the agent does:

charge_card()

The request succeeds, but your worker crashes before recording the result.

A retry could charge the customer twice.

Instead, give the operation an idempotency key:

charge_card(
    amount=100,
    idempotency_key=f"workflow:{workflow_id}:charge"
)

Likewise for:

• sending emails
• creating tickets
• posting messages
• placing orders
• updating databases
• invoking external APIs

Your activities should be safe to execute more than once, because durable execution and retries necessarily create situations where the system has to determine whether an external side effect already happened.

4. Persist the agent state, not just the final answer

Store something resembling:

{
  "workflow_id": "abc123",
  "status": "running",
  "step": "tool_call",
  "messages": [],
  "plan": {},
  "tool_results": {},
  "attempts": {
    "tool_call": 2
  }
}

Then a worker crash becomes:

worker dies
   ↓
workflow state remains
   ↓
new worker picks it up
   ↓
resume from last durable point

This is the distinction between retrying a function and durably executing a workflow.

5. Keep nondeterministic work out of the workflow itself

If you're using a durable-workflow engine, the workflow definition should generally orchestrate activities rather than directly perform arbitrary network calls, random-number generation, wall-clock reads, etc.

Conceptually:

# Workflow
plan = await make_plan_activity(request)
result = await tool_activity(plan)
answer = await answer_activity(result)

rather than:

# Workflow
response = requests.post(...)

That separation makes replay/recovery predictable.

6. Add timeouts at multiple levels

Use at least:

activity timeout
    ↓
retry timeout/backoff
    ↓
workflow timeout

For example:

LLM call:       30 sec
Tool call:       2 min
Individual step: 5 min
Entire workflow: 30 min

Otherwise one hung provider can consume a worker indefinitely.

7. Don't blindly retry the LLM forever

AI-specific failures often aren't transient.

For example:

LLM → malformed tool call
       ↓
      retry
       ↓
LLM → same malformed tool call
       ↓
      retry
       ↓
LLM → same malformed tool call

Instead, distinguish transport failure from semantic failure:

try:
    result = call_llm(...)
except TimeoutError:
    retry()
except RateLimitError:
    retry_with_backoff()
except InvalidToolCall:
    repair_or_replan()

You can even put a small bounded repair loop around tool calls:

generate tool call
      ↓
validate
 ┌────┴─────┐
valid     invalid
  ↓          ↓
execute    repair
             ↓
          validate
             ↓
          max 2 attempts

8. Add observability around every step

Give every execution:

workflow_id
run_id
step_id
attempt
model
model_version
tool_name
latency
tokens
cost
error_type

Then you can answer questions like:

"Why did this agent take 11 minutes?" instead of merely seeing:

Agent failed. For production agents, I'd also track retry rate, failure rate by tool/model, time spent waiting on retries, and cost per successful workflow.

A practical production pattern

If I were building this today, I'd structure it as:

                ┌───────────────────┐
                │ Durable Workflow  │
                └─────────┬─────────┘
                          │
              ┌───────────┼───────────┐
              ▼           ▼           ▼
           Planner     Retriever     Tool
              │           │           │
              └───────────┼───────────┘
                          ▼
                       Validator
                          │
                    ┌─────┴─────┐
                    │           │
                  valid       invalid
                    │           │
                    ▼           ▼
                  Finish     Repair/Replan

Each box is a separately retryable activity, external side effects use idempotency keys, state is durably persisted, and retries have explicit limits/backoff.

If you're already using LangGraph, Temporal, Celery, AWS Step Functions, or plain Python/Node, the implementation differs quite a bit. Tell me which stack you're using and I can show the concrete implementation.

Agent failed. For production agents, I'd also track retry rate, failure rate by tool/model, time spent waiting on retries, and cost per successful workflow.

A practical production pattern

If I were building this today, I'd structure it as:

                ┌───────────────────┐
                │ Durable Workflow  │
                └─────────┬─────────┘
                          │
              ┌───────────┼───────────┐
              ▼           ▼           ▼
           Planner     Retriever     Tool
              │           │           │
              └───────────┼───────────┘
                          ▼
                       Validator
                          │
                    ┌─────┴─────┐
                    │           │
                  valid       invalid
                    │           │
                    ▼           ▼
                  Finish     Repair/Replan

Each box is a separately retryable activity, external side effects use idempotency keys, state is durably persisted, and retries have explicit limits/backoff.

If you're already using LangGraph, Temporal, Celery, AWS Step Functions, or plain Python/Node, the implementation differs quite a bit. Tell me which stack you're using and I can show the concrete implementation.

Adding durable execution and robust retry logic to a flaky AI agent workflow transforms it from a fragile script into a production-grade system. AI agents are inherently flaky due to external API rate limits, transient LLM provider outages, malformed JSON tool outputs, and network hiccups during multi-step reasoning loops.

To achieve durability, you must separate deterministic workflow orchestration (the agent's state machine, loops, and routing logic) from non-deterministic side effects (LLM inference calls, external database queries, and web tool execution).

Step 1: Choose Your Architecture Pattern

You have two main paths to implement this, depending on your tech stack:

Step 2: Structure Your Code to Separate Logic vs. Effects

To use durable execution cleanly, wrap your components so that the orchestrator can replay safely without triggering duplicate side effects (e.g., charging a credit card twice or re-querying an API).

• Activities (Non-Deterministic): Every LLM call, vector search, and tool execution should be an isolated "activity" equipped with automated retry policies (exponential backoff, maximum attempts, jitter).
• Workflows (Deterministic): The control flow—deciding whether to call a tool, loop back to the planner, or ask a human for approval—lives in the workflow code.

Conceptual Example (Python with Tenacity for retries + State Checkpointing)

python``` import logging from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type from openai import RateLimitError, APIConnectionError

1. Isolate flaky external/LLM calls with smart retries

@retry( retry=retry_if_exception_type((RateLimitError, APIConnectionError)), stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, min=2, max=60), reraise=True ) def call_flaky_llm_agent_step(prompt: str, context: dict) -> dict: # Safe wrapper around your LLM provider call response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt}] ) return {"output": response.choices[0].message.content}

2. Main workflow loop using a durable checkpointer concept

def run_durable_agent_loop(initial_state: dict, checkpointer): state = checkpointer.get_latest_state() or initial_state

while not state["is_finished"]:
    try:
        # Execute step with retry boundaries
        result = call_flaky_llm_agent_step(state["prompt"], state)
        state["history"].append(result)
        state["step"] += 1
        
        # Checkpoint progress immediately after success
        checkpointer.save(state)
        
    except Exception as e:
        logging.error(f"Agent step failed terminally after retries: {e}")
        state["error"] = str(e)
        checkpointer.save(state)
        raise e

Use code with caution.

Step 3: Best Practices for Production Resilience

- **Idempotency Keys for Tools:** If your agent uses tools that write data (e.g., sending an email or updating a ticket), ensure those tools accept an idempotency key derived from the workflow execution ID and step count. This prevents duplicate actions if a network timeout occurs right after the tool executes but before the success status is logged.
- **Handle LLM Output Parsing Failures:** Wrap structured output parsing (e.g., Pydantic parsing of LLM JSON) in localized retries that pass the validation error back to the LLM prompt ("Your output failed validation with error X, please correct it").
- **Set Clear Timeouts:** AI APIs can occasionally hang indefinitely instead of throwing a clean error. Define strict timeouts per activity step (e.g., 30 seconds for an LLM generation call, 10 seconds for a web search tool).

If you share **which agent framework** you are currently using (e.g., LangGraph, AutoGen, CrewAI, or raw code) and **where your flakes usually happen** (e.g., rate limits, bad JSON tool outputs, or server crashes), I can provide a **tailored implementation snippet** for your stack.

To make a flaky AI agent workflow production-ready, treat it less like a single function call and more like a durable state machine:

1. Persist progress between agent steps
2. Retry only the operations that are safe to retry
3. Make side effects idempotent
4. Add timeouts and fallback paths
5. Separate deterministic orchestration from unreliable external calls

Frameworks such as LangGraph and Temporal implement these ideas directly: LangGraph provides checkpoint-based durable execution, retries, timeouts, and error handlers; Temporal provides durable workflows, activity retries, and replay-based execution.

1. Break the agent into durable steps

Avoid:

def run_agent(user_input):
    answer = llm(user_input)
    docs = search(answer)
    result = call_api(docs)
    return result

If call_api() fails after 20 minutes, you may lose everything.

Prefer:

Receive request
      ↓
Plan
      ↓
Retrieve data
      ↓
Call tools
      ↓
Validate result
      ↓
Commit side effect

Each step should have:

• input state
• output state
• retry policy
• timeout
• observable status

A workflow engine can then resume from the last successful checkpoint instead of restarting the entire agent.

2. Retry transient failures, not everything

Good retry candidates:

• LLM provider 5xx errors
• rate limits
• network timeouts
• temporary database outages

Bad retry candidates:

• invalid user input
• malformed tool arguments
• permission failures
• deterministic code bugs

Example policy:

RetryPolicy(
    max_attempts=4,
    initial_interval=1,
    backoff_factor=2,
    jitter=True,
    retry_on=(TimeoutError, ConnectionError)
)

Use exponential backoff:

attempt 1: immediately
attempt 2: wait 1s
attempt 3: wait 2s
attempt 4: wait 4s

LangGraph's retry model follows this pattern with configurable retry conditions, backoff, and jitter.

3. Make tool calls idempotent

Retries can accidentally duplicate actions.

Dangerous:

charge_credit_card(amount)

If the request times out after charging, retrying may charge twice.

Better:

charge_credit_card(
    amount=100,
    idempotency_key="agent_run_123_payment"
)

Store:

operation_id → result

Then a retry becomes:

"Have I already completed this operation?"
        |
        +-- yes → return previous result
        |
        +-- no → execute

4. Put external calls behind activities/tasks

A useful pattern:

Workflow
   |
   +-- deterministic decisions
   |
   +-- Activity: call LLM
   |
   +-- Activity: call search API
   |
   +-- Activity: update CRM

The workflow decides what happens next.

Activities do unreliable work.

For example, Temporal's LangGraph integration uses Activities for things like LLM calls, HTTP requests, database queries, and other non-deterministic operations because those operations need retry and timeout behavior.

5. Add timeouts everywhere

Without timeouts, an agent can hang forever:

Waiting for tool response...
Waiting...
Waiting...

Use multiple timeout levels:

• LLM timeout: 30–120 seconds
• Tool timeout: depends on service
• Whole workflow timeout: hours/days if needed

LangGraph supports node-level timeouts alongside retries.

6. Handle failures as workflow states

Don't just do:

except Exception:
    return "failed"

Instead:

Tool failed
    |
    +--> retry
          |
          +--> success
          |
          +--> fallback tool
                  |
                  +--> human approval
                          |
                          +--> continue

Common recovery strategies:

LangGraph documents these as separate categories: transient failures, LLM-recoverable failures, user-fixable failures, and unrecoverable failures.

7. Store agent state explicitly

Example state:

{
  "request_id": "abc123",
  "messages": [...],
  "plan": [...],
  "completed_steps": [
      "retrieve_docs"
  ],
  "tool_results": {...},
  "status": "waiting_for_tool"
}

This gives you:

• replay
• debugging
• audit history
• resume after crashes

8. Add observability

Track:

• workflow ID
• step name
• retry count
• latency
• token usage
• tool failures
• final outcome

Useful logs:

workflow=customer_agent
step=search_documents
attempt=2
error=timeout
next_retry=4s

A production architecture often looks like:

                 API Request
                     |
                     v
              Durable Workflow
                     |
        +------------+-------------+
        |                          |
        v                          v
   Agent Planner              State Store
        |
        |
 +------+-------+
 |              |
 v              v
LLM Activity   Tool Activities
(retry)        (retry + timeout)
 |
 v
Validation
 |
 +--> Human approval if needed
 |
 v
Side effect Activity
(idempotent)

A good rule of thumb: the workflow should survive process crashes, duplicate execution, and partial failure without losing state or repeating unsafe actions. That is the core of durable execution.

{
  "request_id": "abc123",
  "messages": [...],
  "plan": [...],
  "completed_steps": [
      "retrieve_docs"
  ],
  "tool_results": {...},
  "status": "waiting_for_tool"
}

This gives you:

• replay
• debugging
• audit history
• resume after crashes

8. Add observability

Track:

• workflow ID
• step name
• retry count
• latency
• token usage
• tool failures
• final outcome

Useful logs:

workflow=customer_agent
step=search_documents
attempt=2
error=timeout
next_retry=4s

A production architecture often looks like:

                 API Request
                     |
                     v
              Durable Workflow
                     |
        +------------+-------------+
        |                          |
        v                          v
   Agent Planner              State Store
        |
        |
 +------+-------+
 |              |
 v              v
LLM Activity   Tool Activities
(retry)        (retry + timeout)
 |
 v
Validation
 |
 +--> Human approval if needed
 |
 v
Side effect Activity
(idempotent)

A good rule of thumb: the workflow should survive process crashes, duplicate execution, and partial failure without losing state or repeating unsafe actions. That is the core of durable execution.

Waiting for tool response...
Waiting...
Waiting...

Use multiple timeout levels:

Adding durable execution and intelligent retries to an AI agent workflow transforms it from a brittle, script-based prototype into a production-ready system. AI agents are uniquely flaky because they suffer from both standard infrastructure failures (network timeouts, rate limits) and probabilistic failures (malformed JSON tool outputs, hallucinated parameters, or context window drops).

Durable execution works by automatically checkpointing state after every step, allowing execution to pause, retry, or resume seamlessly from the exact point of failure without re-running completed LLM inference or external tool calls.

Core Strategies for Durable AI Agents

1. Separate Deterministic Orchestration from Non-Deterministic I/O
   • The Rule: The agent's control flow (decision making, state transitions) must be strictly deterministic, while side-effects (LLM inference, API calls, database writes, and tool executions) are treated as isolated external "activities" or "steps".
   • Why it matters: Frameworks like Temporal or Inngest replay the deterministic code history when recovering from a crash, skipping already-completed LLM calls and reusing their cached outputs.
2. Implement Step-Level Retries with Exponential Backoff
   • Do not wrap the entire agent loop in a massive try/catch block. Instead, apply granular retries to individual tool executions and LLM generation calls.
   • Configure distinct retry policies:
     • Rate limits (HTTP 429 / 503): Exponential backoff with jitter (e.g., wait 2s, 4s, 8s).
     • Malformed Tool Output / JSON parsing errors: Catch validation exceptions immediately, feed the error back into the LLM context as a corrective prompt ("Your tool output failed JSON validation with error X. Fix it."), and count it against a strict sub-retry limit.
3. Leverage Native Suspend/Resume for Human-in-the-Loop & Long-Running Tasks
   • Agents often need to wait hours or days for user approval, external webhook callbacks, or file processing.
   • Durable execution platforms natively support pausing execution indefinitely without keeping an active server process alive or holding database locks, then waking up securely via an external event or signal.

Implementation Approaches

Option A: Dedicated Durable Orchestration Engines (Most Robust)

You wrap your agent steps using a durable workflow orchestration engine. This provides infrastructure-level guarantees (surviving server crashes, deploys, and network partitions).

• Temporal: Define your agent logic as a Workflow and wrap LLM/tool calls as Activities. If an activity fails, Temporal retries it automatically; if the worker crashes, another worker picks up the workflow history and replays it to the exact failure point. (Libraries like PydanticAI and integrations with LangGraph make this increasingly straightforward).
• Inngest: Allows you to write serverless step functions where every step automatically checkpoints state and retries independently without needing a separate state database.

Option B: Framework-Native Persistence (Easiest to Start)

If you are already using agent frameworks like LangGraph or LlamaIndex , they feature built-in checkpointers (using PostgreSQL, Redis, or SQLite).

• Every time a graph node finishes executing (e.g., after an LLM node or a tool node), the state is committed to the checkpoint store.
• If a node fails, you can resume the graph execution from the last valid thread ID/checkpoint. Note that this handles state persistence well, but you still need to explicitly configure retry decorators (like tenacity ) around individual flaky API or tool calls inside your nodes.

To help tailor this to your setup, tell me:

• What framework or stack are you currently using to build your agent (e.g., LangGraph, raw Python, PydanticAI, custom FastAPI)?
• What is the primary source of flakiness you're facing (e.g., third-party tool timeouts, LLM rate limits, or server crashes mid-workflow)?

I can provide a code snippet or architectural pattern tailored to your environment.

Adding durable execution and retries to a flaky AI agent workflow requires separating the probabilistic reasoning (the LLM deciding what to do next) from the deterministic orchestration (managing state, crashing, and retrying network I/O).

Trying to solve infrastructure failures with prompt engineering or basic try/catch blocks inevitably fails. Instead, production architectures rely on a clear Brain vs. Muscle split: the agent framework handles reasoning, while a durable execution engine or explicit checkpointer manages state and network boundaries.

Step 1: Separate Non-Deterministic I/O into Isolated Units

The golden rule of durable execution is that side effects and external API calls cannot live directly inside volatile orchestration scripts . Wrap every individual LLM call, vector search, MCP (Model Context Protocol) call, and tool execution into an isolated function (an Activity or Task).

• Why: If an LLM call or a third-party API returns a rate-limit error (HTTP 429) or times out, only that specific unit of work retries. The parent workflow remembers that previous steps already completed successfully and does not re-run them (saving tokens and avoiding duplicate side effects like double-charging a credit card).

Step 2: Choose Your Durability Layer

Depending on your stack and complexity tolerance, implement one of these primary architectural patterns:

1. Dedicated Orchestration Engines (e.g., Temporal):
   • Best for complex, long-running agent workflows that need to survive server crashes or pause for hours/days waiting for human approval.
   • You write the agent's ReAct loop as a deterministic Workflow , while LLM generation and tool execution are registered as Activities equipped with custom exponential backoff retry policies. Check out the Temporal Documentation for PydanticAI or Temporal's Durable AI Guide to see this in action.
2. Lightweight Task Orchestrators (e.g., Prefect):
   • Best if you want code-as-you-go ergonomics without heavy infrastructure overhead.
   • You decorate your agent execution functions with @task(retries=3) or use the Prefect Durability Integration to automatically cache and retry failed steps.
3. Graph-Native Checkpointing (e.g., LangGraph Checkpointers):
   • Best if you are already heavily invested in graph-based agent topologies.
   • Configure a persistent checkpointer (PostgresSaver or RedisSaver ) set to sync mode so that the agent's state graph writes a checkpoint to your database after every single node transition.

Step 3: Enforce Strict Tool Idempotency

Retrying a tool call blindly can break things if the tool isn't safe to run twice (e.g., sending an email or executing a database write).

• Pass a deterministic idempotency key (derived from your workflow_id and unique step_id ) into every external mutation tool.
• Design your downstream tool APIs to recognize and deduplicate requests matching that key.

To help tailor this architecture to your specific stack, tell me:

• What agent framework are you currently using (e.g., LangGraph, CrewAI, custom loops)?
• Are your tools read-only (like web searches) or do they perform external mutations (like database updates or sending messages)?

Durable execution and retries solve two different problems:

• Retries: “This step failed; try the step again.”
• Durable execution: “The process died; resume the workflow from the last safe point without repeating completed work.”

For AI agents, you usually need both because workflows combine unreliable external calls (LLMs, APIs, tools) with expensive intermediate state.

1. Break the agent into durable steps

Avoid a single function like:

def run_agent(task):
    plan = llm.plan(task)
    docs = search(plan)
    answer = llm.write(docs)
    send(answer)

If it fails during send(), you may repeat the planning, search, and generation.

Instead, model it as steps:

Receive task
    ↓
Plan
    ↓
Gather information
    ↓
Call tools
    ↓
Generate response
    ↓
Deliver result

Each boundary becomes a checkpoint. A failure after “Gather information” can resume from there.

---

2. Persist workflow state

Store the agent state outside process memory:

AgentState = {
    "task_id": "...",
    "messages": [...],
    "plan": "...",
    "tool_results": [...],
    "status": "running"
}

Persist after meaningful transitions:

save_checkpoint(state)

result = call_llm()

state["draft"] = result
save_checkpoint(state)

Good checkpoint contents:

• conversation history
• tool outputs
• intermediate plans
• completed step IDs
• retry counts
• human approval status

Avoid storing only the final answer; you need enough information to resume.

---

3. Add retries at the step level

Do not retry the entire agent blindly.

Bad:

Agent failed → restart everything

Better:

Tool call failed → retry tool call
LLM timeout → retry LLM step
Payment failed → run compensation flow

Example retry policy:

retry_policy = {
    "max_attempts": 4,
    "initial_delay": 1,
    "backoff": 2,
    "max_delay": 60
}

This produces:

Attempt 1: immediately
Attempt 2: wait 1s
Attempt 3: wait 2s
Attempt 4: wait 4s

Use retries only for transient failures:

Retry:

• network timeout
• HTTP 429 rate limits
• HTTP 5xx errors
• temporary database outage

Do not retry:

• invalid input
• authentication failure
• bad tool arguments
• malformed application logic

Frameworks such as LangGraph support per-node retry policies, timeouts, and recovery handlers for this pattern.

---

4. Make every side effect idempotent

Retries can execute the same operation more than once.

Dangerous:

charge_credit_card()

If the request succeeds but the response is lost, retrying may charge twice.

Use idempotency keys:

charge_credit_card(
    payment_id="order_123_attempt_1"
)

Other examples:

---

5. Separate deterministic orchestration from nondeterministic work

Agent workflows contain things that change:

• LLM outputs
• current time
• random sampling
• external API responses

Those belong in retryable tasks/activities.

The workflow controller should mostly decide:

If step completed:
    continue

If failed:
    retry or compensate

not:

Generate random plan
Call API
Look at current time

Durable workflow engines such as Temporal use this model: workflows are replayed from history while external work runs as retryable activities.

---

6. Add timeouts

Retries without timeouts can leave stuck agents running forever.

Example:

call_llm(
    timeout=30,
    retry_policy={
        "max_attempts": 3
    }
)

Think in layers:

Workflow timeout
    ↓
Step timeout
    ↓
Network timeout
    ↓
Retry policy

---

7. Add recovery paths, not just retries

Some failures need a different route:

Generate answer
      |
      v
Validation failed
      |
      +--> Retry generation
      |
      +--> Ask human
      |
      +--> Use fallback model

Examples:

• LLM cannot parse tool output → ask model to repair
• Missing customer information → pause for user input
• Tool permanently unavailable → use alternate provider

LangGraph describes this as distinguishing transient failures, LLM-recoverable failures, user-fixable failures, and unexpected failures.

---

8. A production architecture often looks like this

                Durable Workflow Engine
                         |
        +----------------+----------------+
        |                |                |
     Planner          Tools           Reviewer
        |                |                |
     Retry             Retry            Retry
        |                |                |
     Checkpoint       Checkpoint       Checkpoint

                         |
                  Persistent State Store

Common implementation choices:

• LangGraph checkpointing + retries for agent-centric workflows.
• Temporal workflows + activities when you need strong durability across worker crashes, long-running jobs, and complex business processes.
• A combination: agent framework for reasoning, durable workflow engine for execution guarantees.

A good rule of thumb: make every expensive or failure-prone action a resumable step, make every step idempotent, and retry only the smallest unit that failed.

                Durable Workflow Engine
                         |
        +----------------+----------------+
        |                |                |
     Planner          Tools           Reviewer
        |                |                |
     Retry             Retry            Retry
        |                |                |
     Checkpoint       Checkpoint       Checkpoint

                         |
                  Persistent State Store

Common implementation choices:

charge_credit_card(
    payment_id="order_123_attempt_1"
)

Other examples:

---

5. Separate deterministic orchestration from nondeterministic work

Agent workflows contain things that change:

• LLM outputs
• current time
• random sampling
• external API responses

Those belong in retryable tasks/activities.

The workflow controller should mostly decide:

Tool call failed → retry tool call
LLM timeout → retry LLM step
Payment failed → run compensation flow

Example retry policy:

retry_policy = {
    "max_attempts": 4,
    "initial_delay": 1,
    "backoff": 2,
    "max_delay": 60
}

This produces:

Attempt 1: immediately
Attempt 2: wait 1s
Attempt 3: wait 2s
Attempt 4: wait 4s

Use retries only for transient failures:

Retry:

• network timeout
• HTTP 429 rate limits
• HTTP 5xx errors
• temporary database outage

Do not retry:

• invalid input
• authentication failure
• bad tool arguments
• malformed application logic

Frameworks such as LangGraph support per-node retry policies, timeouts, and recovery handlers for this pattern.

---

4. Make every side effect idempotent

Retries can execute the same operation more than once.

Dangerous:

The key is to separate orchestration from failure-prone work and make every meaningful step resumable.

A good production architecture looks like:

                 ┌───────────────┐
                 │ Durable       │
                 │ workflow      │
                 └───────┬───────┘
                         │
             ┌───────────▼───────────┐
             │ Agent loop / state     │
             │ persisted after steps │
             └───────┬───────────────┘
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       LLM call    Tool/API   DB/write
       activity    activity    activity
          │          │          │
       retries    retries    idempotency

1. Persist state between agent steps

Don't keep the entire agent run in process memory. Persist:

• conversation / messages
• current step
• tool calls and results
• intermediate outputs
• retry/attempt metadata
• workflow/run ID

Then a worker crash becomes "resume from the last checkpoint", rather than "start the agent over."

For example, LangGraph's checkpointer provides this model: successful work is persisted and a resumed execution doesn't redo completed work. Its sync, async, and exit durability modes let you trade persistence guarantees against overhead.

2. Retry at the smallest useful boundary

Don't blindly retry the entire agent.

Instead:

agent workflow
   ├── call LLM       ← retry
   ├── search API     ← retry
   ├── execute tool   ← retry
   ├── validate       ← usually don't retry
   └── write result   ← carefully retry/idempotency

For transient failures, use bounded exponential backoff + jitter:

RetryPolicy(
    max_attempts=4,
    initial_interval=1.0,
    backoff_factor=2.0,
    max_interval=30.0,
    jitter=True,
)

LangGraph exposes essentially this policy directly, including exception-specific retry rules.

A particularly important distinction is:

429 / rate limit       → retry
502 / 503 / network    → retry
timeout                → retry
invalid API argument   → don't retry
bad model output       → usually recover differently
permission denied      → don't retry

3. Put side effects behind idempotent operations

This is the part people often miss.

Suppose the agent does:

charge_credit_card()

The process crashes after the payment succeeds but before the workflow records success.

A retry could charge the customer twice.

Give external operations an idempotency key:

payment_id = f"{workflow_id}:payment"

charge_card(
    amount=100,
    idempotency_key=payment_id,
)

The same principle applies to:

• sending emails
• creating tickets
• submitting orders
• writing database records
• calling APIs that mutate state

Durable execution means work may be re-executed, so side effects need to tolerate that.

4. Add timeouts as well as retries

A retry without a timeout can leave an agent stuck indefinitely.

Use:

attempt
  │
  ├── succeeds → continue
  │
  └── timeout/error
        │
        ├── retryable → exponential backoff → attempt again
        │
        └── permanent → recovery/fallback

For example:

call_llm(
    timeout=30,
    retries=3,
)

LangGraph explicitly composes per-node timeouts with retry policies; a timed-out attempt can be retried with its own fresh timeout.

5. Separate "infrastructure retry" from "agent recovery"

These are different mechanisms.

Infrastructure retry:

"The API returned 503. Try the exact same operation again."

Agent recovery:

"The search tool returned an error. Tell the model what happened and let it choose another strategy."

For the second case, feed the error back into the agent:

LLM
 ↓
tool call
 ↓
tool fails
 ↓
record error
 ↓
LLM sees error
 ↓
choose alternate tool / fix arguments / abandon task

LangGraph specifically recommends this pattern for errors that the LLM itself can recover from.

6. For long-running agents, use a durable workflow engine

If your agent might run for minutes, hours, or days—or needs human approval—I'd strongly consider a workflow engine rather than building persistence/recovery yourself.

Temporal is a strong option. Its model is roughly:

Workflow = deterministic orchestration
Activity = failure-prone external work

LLM calls and tool calls become Activities, while the agent loop remains in the durable Workflow. Temporal automatically retries Activities according to retry policies and replays completed work rather than executing it again after a worker restart.

The OpenAI Agents SDK also has a Temporal integration specifically for durable, long-running agents.

A practical recipe

If you're starting with an existing flaky agent, I'd implement these in this order:

1. Give every run a durable ID.
2. Persist state after every meaningful agent step.
3. Move LLM/tool/API calls into retryable units.
4. Add 20–60s timeouts to external calls.
5. Retry only transient errors.
6. Use exponential backoff + jitter + maximum attempts.
7. Make every side effect idempotent.
8. Add explicit fallback/recovery paths for non-transient agent errors.
9. Record attempt count, error type, latency, and workflow ID for observability.
10. Test by deliberately killing workers halfway through runs.

If you're already using LangGraph, its built-in checkpointer + RetryPolicy + timeouts are probably the shortest path. If you're building a long-running production agent where crash recovery is a first-class requirement, Temporal is worth evaluating. LangGraph and Temporal can also be combined rather than treated as mutually exclusive choices.

Adding durable execution and retries to an AI agent workflow transforms it from a fragile script into a resilient, production-ready system. AI agents are notoriously flaky due to network timeouts, rate limits, API outages, and non-deterministic LLM responses.

1. Choose a Durable Execution Framework

Writing custom state-machines and retry loops in standard application code quickly becomes an unmaintainable mess. Instead, leverage established orchestration engines that handle state persistence, timeouts, and retries natively.

• Temporal : The industry standard for durable execution. It treats your workflow as code, automatically persisting state after every step (step₁ → step₂) so that if a server crashes mid-agent-thought, it resumes exactly where it left off.
• Inngest : A developer-friendly, serverless-native workflow engine that handles retries, step-by-step execution, and event-driven architectures without requiring a dedicated state cluster.
• LangGraph (with Persistence) : If you are building within the LangChain ecosystem, LangGraph allows you to save agent state to a database (like Postgres or Redis) after every node execution, enabling time-travel and resumption upon failure.
• Prefect or Dagster : Excellent if your AI agent workflow leans heavily into data pipelines, RAG ingestion, or batch processing steps.

2. Implement Granular, Step-Level Retries

Do not wrap your entire agent loop in a single massive try/catch block. Break your agent workflow into distinct, atomic steps and apply targeted retry policies to each:

• LLM Calls (High Flakiness) : Configure exponential backoff with jitter to handle rate limits (HTTP 429) and transient server errors (HTTP 500, 503).
• Tool Execution / External APIs : Wrap web searches, database lookups, or API calls in independent retry wrappers with strict timeouts (T ≤ 30 seconds) so a hanging API doesn't freeze the whole agent.
• Idempotency : Ensure that every step (especially tool calls like writing to a database or sending an email) is idempotent . If a step fails halfway and retries, running it twice should not result in duplicate side effects.

3. Design for Non-Deterministic Failures (Self-Correction)

Retrying a broken API call is easy, but retrying when an LLM outputs malformed JSON or hallucinates a tool argument requires agent-specific logic:

• Structured Output Validation : Use libraries like Pydantic or Instructor to validate LLM outputs against a rigid schema. If validation fails, catch the validation error, feed the exact schema mismatch error back to the LLM as a prompt correction, and retry the generation step.
• Circuit Breakers : If an external tool or LLM provider fails N consecutive times (e.g., N = 5), trip a circuit breaker to halt the workflow and alert a human or fallback to a cheaper/more stable model rather than burning through API rate limits.

4. Persist External Memory and State

Durable execution relies on externalizing state away from the volatile process memory of the agent runner:

• State Checkpointing : Save the full agent conversation history, intermediate tool outputs, and variable states to an ACID-compliant database after each turn.
• Human-in-the-Loop Interruption : When an agent hits a low-confidence threshold or encounters an unrecoverable tool error, use durable pause/resume primitives to wait for human intervention without losing context.

If you want, tell me:

• What stack/language you are using (Python, TypeScript, etc.)
• Where you are hosting your agent (AWS, serverless, Kubernetes)

I can recommend the exact framework or code pattern that fits your architecture.

The key is to treat the agent as a durable state machine, not as one big function with a retry around it.

A robust architecture looks like:

                 ┌──────────────┐
                 │ Agent state  │
                 │ + checkpoint │
                 └──────┬───────┘
                        │
                  ┌─────▼─────┐
                  │ Plan/LLM  │◄── retry
                  └─────┬─────┘
                        │
                  ┌─────▼─────┐
                  │ Tool call │◄── timeout + retry
                  └─────┬─────┘
                        │
                    checkpoint
                        │
                  ┌─────▼─────┐
                  │ Next step │
                  └───────────┘

1. Break the agent into durable steps

Don't do:

def run_agent():
    response = llm(...)
    result = tool_a(response)
    response = llm(result)
    result = tool_b(response)
    return response

Instead, make each meaningful operation a separately recoverable step:

state = {
    "messages": ...,
    "tool_results": ...,
    "status": "running",
}

llm_result = durable_step("reason", call_llm, state)
state = checkpoint(state, llm_result)

tool_result = durable_step("tool", run_tool, state)
state = checkpoint(state, tool_result)

...

If the process dies during tool_b, you want to resume from the last durable boundary—not pay for llm() and tool_a() again.

Frameworks such as LangChain's LangGraph implement this with checkpoints; successful work from a failed super-step can be preserved so it doesn't have to be recomputed.

2. Retry only failures that are actually transient

A good default policy is:

Use exponential backoff + jitter, e.g.:

attempt 1 → immediately
attempt 2 → ~1s
attempt 3 → ~2s
attempt 4 → ~4s

with randomness added so thousands of agents don't retry simultaneously.

LangGraph's current retry policy, for example, supports max attempts, exponential backoff, maximum interval, jitter, and custom exception filtering.

3. Put timeouts around every external operation

Retries without timeouts can make a bad dependency hang your entire workflow.

Use separate limits for:

LLM call       → 30–120s
normal API     → 5–30s
database       → 5–15s
long-running tool → explicit heartbeat/progress timeout

Then combine:

retry(
    timeout(call_llm, 60),
    max_attempts=4,
    backoff="exponential",
    jitter=True,
)

The important detail is that each retry gets its own timeout.

4. Make tool calls idempotent

This is the biggest gotcha.

Suppose the agent says:

charge_customer($100)

The request reaches Stripe, succeeds, and then your worker crashes before recording the result.

On retry, blindly calling the tool again could charge the customer twice.

Give externally visible operations an idempotency key:

run_id = "agent-run-8472"
step_id = "charge-customer"
idempotency_key = f"{run_id}:{step_id}"

charge_customer(
    amount=100,
    idempotency_key=idempotency_key,
)

For non-idempotent operations, use one of:

• idempotency keys
• transactional/outbox patterns
• a durable tool-result store
• explicit compensation/rollback
• human approval before the irreversible operation

Retries solve transient failure; idempotency solves duplicate execution. You need both.

5. Persist state outside the worker process

Don't rely on:

agent_memory = {}

or an in-memory queue.

Persist at least:

run_id
current_state
completed_steps
tool results
attempt counts
timestamps
errors
workflow status

Then a worker can disappear and another worker can pick up the same run.

If you're using LangGraph, a checkpointer provides this state persistence and associates execution with a thread_id.

For a framework-agnostic durable execution layer, Temporal is another strong option: its workflow history lets an agent resume after worker crashes, with completed LLM/tool activities replayed from durable history rather than executed again.

6. Separate workflow logic from side effects

A useful mental model is:

Workflow
  ├── decide what should happen
  ├── record state
  ├── schedule activity
  └── react to result

Activity
  ├── call LLM
  ├── call API
  ├── query DB
  └── send email

The workflow should be deterministic and durable; side effects belong in activities/tasks that have explicit retry and timeout policies.

Temporal's current AI-agent architecture uses essentially this split: the workflow orchestrates the agent loop while LLM and tool calls are durable activities.

7. Have an explicit terminal failure path

Don't retry forever.

                    transient?
                   /           \
                 yes            no
                  │              │
               retry          fail/handle
                  │
              attempts
              exhausted?
              /       \
            no         yes
            │           │
          retry     fallback
                        │
                  human / dead-letter

After, say, 3–5 attempts:

• switch to a fallback model/tool,
• ask the user for clarification,
• request human intervention,
• compensate a partial transaction,
• or put the run in a dead-letter/recovery queue.

LangGraph, for example, supports a node-level error handler that runs after retries are exhausted.

A practical production recipe

If you're building this today, I'd use:

                    Durable workflow
                           │
                 ┌─────────▼─────────┐
                 │ checkpointed state │
                 └─────────┬─────────┘
                           │
                    ┌──────▼──────┐
                    │ LLM activity│
                    │ retry 3–4x  │
                    │ timeout     │
                    └──────┬──────┘
                           │
                    checkpoint
                           │
                    ┌──────▼──────┐
                    │ Tool activity│
                    │ retry +      │
                    │ idempotency  │
                    └──────┬──────┘
                           │
                    checkpoint
                           │
                       next step

If you're already using LangGraph, its checkpointer + RetryPolicy + timeout + error-handler primitives give you most of this directly.

If you need very long-running workflows, strong guarantees around external side effects, or infrastructure-level durability, I'd lean toward Temporal and put your agent loop inside a Temporal workflow. Google's current Gemini/Temporal example uses exactly this pattern for durable agent execution.

The most important rule is: never put a single retry around the entire agent. Retry and checkpoint at the smallest meaningful unit of work, and make every side effect safe to repeat.

Adding durable execution and automatic retries to an AI agent workflow transforms it from a fragile prototype into a production-ready system. AI workflows are uniquely flaky due to rate limits, context-window timeouts, non-deterministic model outputs, and third-party API outages.

Here is how to implement durable execution and smart retries for your AI agent:

• Adopt a Orchestration Framework : Instead of writing custom retry loops and state storage in raw Python/Node.js, use a dedicated durable execution engine. These frameworks automatically persist the state of your agent's execution step-by-step.
  • Temporal : The industry standard for general-purpose durable execution. It treats your agent workflows as code, automatically retrying failed activities and persisting state to a database. You can explore the Temporal Documentation to get started.
  • Inngest : Great for serverless and event-driven architectures, offering simple step functions, reliable retries, and background jobs. Check out Inngest for easy integration.
  • LangGraph : Specifically designed for stateful, multi-actor AI applications, offering built-in persistence (checkpoints) to pause, resume, or replay agent steps. Visit the LangGraph Documentation to learn more.
  • Prefect / Dagster : Excellent if your AI agent workflow doubles as a data pipeline or sequential LLM chain.
• Isolate LLM Calls into Discrete "Activities" or "Steps" : Never wrap your entire agent loop in a single retry block. Break the workflow down into atomic operations:
  • Fetching data / RAG retrieval
  • The actual LLM completion call
  • Tool execution (e.g., executing a SQL query or calling an external API)
  • State evaluation / reflection
• Implement Semantic and Exponential Backoff Retries : Standard linear retries will hammer an already-overloaded LLM provider (like OpenAI or Anthropic hitting 429 Rate Limit errors).
  • Use exponential backoff with jitter (randomized delay) so concurrent agent threads don't synchronized-retry and re-trigger rate limits.
  • Catch specific errors (e.g., 429, 503, or JSON parsing failures) to trigger retries, while failing fast on unrecoverable errors (e.g., 400 Bad Request or invalid API keys).
• Enforce Output Validation with Self-Correction Retries : AI agents frequently output malformed JSON or fail tool-calling schemas. Wrap your LLM parsing steps in a validation layer (like Pydantic). If validation fails, feed the validation error back to the agent as a prompt correction within the same retry cycle rather than just throwing a hard exception.
• Persist State Passports (Checkpoints) : Ensure that when an agent calls an external tool and succeeds, that intermediate result is saved immediately. If the workflow crashes right before the next LLM call, the system should resume from the last successful checkpoint instead of re-running expensive or side-effect-heavy tool calls (like sending an email or charging a payment).

If you'd like to dive deeper, let me know:

• What framework or language are you currently building your agent in?
• Are you running into specific issues like rate limits (429), bad JSON outputs , or server timeouts?

I can provide a code snippet or architectural pattern tailored to your stack.