2026-07-05 · 18 min read
The Stateful Agent Problem: Why LLM Agents Are Just Distributed Systems in Disguise
Picture this. You ship an AI agent that handles customer refund requests end-to-end. It reads the order database, checks the refund policy, calls an internal API to issue the credit, then sends a confirmation email. Works perfectly in staging. Users love it. Then one Tuesday, the email service times out midway through. The agent crashes.
The credit was already issued. The email never sent.
You now have a customer with a refund they don’t know about, support tickets asking where the money went, and an agent that (if you retry it naively) will issue the credit a second time.
This is not an AI problem. This is the dual-write problem. Engineers solved this in distributed systems fifteen years ago. But nobody told the agent framework.
The Lie That Agent Frameworks Tell You
Every popular agent framework (LangChain, LlamaIndex, CrewAI, AutoGen) makes multi-step reasoning look trivial. You define tools, hand the framework a goal, and it figures out the steps. It feels like magic.
What they don’t show you is what happens when step 3 fails after step 2 already had side effects. They don’t show you what happens when the same task gets queued twice. They don’t show you what happens when two agent instances run in parallel and both try to update the same record. The demos are always single-threaded, always successful, and always short.
Running a multi-step LLM agent in production is a distributed systems problem. The LLM is just the decision-making layer. The hard problems are exactly the same ones that made microservices painful: state consistency, partial failures, idempotency, and distributed coordination.
The good news: distributed systems has decades of battle-tested patterns for exactly this. The bad news: most teams building agents right now are rediscovering them from scratch, painfully, in production.
What “State” Actually Means in an Agent
Before mapping patterns, it’s worth being precise about what state a multi-step agent actually carries. There are three layers, and they fail in completely different ways.
Working memory is the context window: the conversation history, the system prompt, the tool schemas. In the refund scenario, this is the agent knowing it’s handling order #4821 for $47. When this breaks, the agent loses its reasoning thread and gives a bad answer. Annoying. Recoverable.
Execution memory is the record of what steps ran and what they returned. This is the agent knowing it already called the orders DB and got a pending status back. When this breaks, you get duplicate tool calls. The agent re-issues the credit because it forgot it already did that. Also recoverable, if you built idempotency in.
World state is everything the agent mutated in external systems: the credit issued in the payment API, the status updated in the orders DB, the email queued in SendGrid. When this breaks mid-sequence (say, the credit is issued but the email service times out), you have inconsistency that lives outside your system entirely. No rollback. No undo. The payment API doesn’t know your agent crashed.
This is the one that matters. Everything that follows is about world state.
Pattern 1: The Saga — Replacing Transactions with Compensations
In a traditional monolith, you wrap a multi-step operation in a database transaction. Either all steps succeed and commit, or all steps fail and roll back. Clean.
In a distributed system (or an agent calling multiple external tools), there is no global transaction. Each tool call is its own atomic operation with its own success or failure. If step 3 fails, steps 1 and 2 have already committed to their respective systems.
The solution is the Saga pattern. Instead of rollback, you define explicit compensation actions for each step. If the process fails at step N, you run the compensation for steps N-1, N-2, down to step 1 in reverse order.
sequenceDiagram
participant A as Agent Orchestrator
participant ODB as Orders DB
participant PAY as Payment API
participant EMAIL as Email Service
A->>ODB: 1. Mark order as refund-pending
ODB-->>A: OK
A->>PAY: 2. Issue credit $47.00
PAY-->>A: OK (credit_id: cr_8x2k)
A->>EMAIL: 3. Send confirmation email
EMAIL-->>A: TIMEOUT
Note over A: Step 3 failed. Run compensations in reverse.
A->>PAY: COMPENSATE: Reverse credit cr_8x2k
PAY-->>A: OK
A->>ODB: COMPENSATE: Mark order as refund-failed
ODB-->>A: OK
Note over A: State is consistent. Alert support team.Here’s what this looks like in actual agent infrastructure. Compensation logic lives alongside the forward action. Not as an afterthought. As a first-class part of every tool definition.
# agents/saga.py
import logging
from dataclasses import dataclass, field
from typing import Callable, Any, Optional
log = logging.getLogger("agent.saga")
@dataclass
class SagaStep:
name: str
action: Callable[..., Any]
compensate: Callable[..., Any]
# If compensation itself fails, do we abort or keep rolling back?
compensation_is_critical: bool = True
@dataclass
class SagaResult:
success: bool
completed_steps: list[str]
failed_step: Optional[str]
compensation_ran: bool
output: Optional[Any] = None
error: Optional[Exception] = None
class SagaOrchestrator:
"""
Runs a sequence of steps with automatic compensation on failure.
Each step's result is passed as context to subsequent steps,
so the compensation actions have access to IDs and references
they need to undo the work (e.g., the credit_id from step 2).
"""
def __init__(self, steps: list[SagaStep]):
self.steps = steps
async def run(self, initial_context: dict) -> SagaResult:
context = dict(initial_context)
completed: list[tuple[SagaStep, Any]] = []
for step in self.steps:
try:
log.info(f"saga.step.start: {step.name}")
result = await step.action(context)
context[step.name] = result
completed.append((step, result))
log.info(f"saga.step.success: {step.name}")
except Exception as exc:
log.error(f"saga.step.failed: {step.name} -- {exc}")
# Run compensations in reverse order
compensation_ran = await self._compensate(completed, context)
return SagaResult(
success=False,
completed_steps=[s.name for s, _ in completed],
failed_step=step.name,
compensation_ran=compensation_ran,
error=exc,
)
return SagaResult(
success=True,
completed_steps=[s.name for s, _ in completed],
failed_step=None,
compensation_ran=False,
output=context,
)
async def _compensate(
self,
completed: list[tuple[SagaStep, Any]],
context: dict,
) -> bool:
all_compensated = True
for step, _ in reversed(completed):
try:
log.info(f"saga.compensate.start: {step.name}")
await step.compensate(context)
log.info(f"saga.compensate.success: {step.name}")
except Exception as exc:
log.error(f"saga.compensate.failed: {step.name} -- {exc}")
all_compensated = False
if step.compensation_is_critical:
# Compensation itself failed. Alert immediately.
# Do NOT continue silently -- this is a data consistency incident.
await alert_on_call(
f"CRITICAL: Saga compensation failed for step '{step.name}'. "
f"Manual intervention required. Context: {context}"
)
break
return all_compensated
# Usage: wiring up the refund agent as a saga
async def handle_refund_request(order_id: str, amount: float):
saga = SagaOrchestrator(steps=[
SagaStep(
name="mark_pending",
action=lambda ctx: orders_db.update_status(ctx["order_id"], "refund_pending"),
compensate=lambda ctx: orders_db.update_status(ctx["order_id"], "refund_failed"),
),
SagaStep(
name="issue_credit",
action=lambda ctx: payment_api.issue_credit(ctx["order_id"], ctx["amount"]),
compensate=lambda ctx: payment_api.reverse_credit(
ctx["issue_credit"]["credit_id"] # credit_id from the forward action
),
),
SagaStep(
name="send_confirmation",
action=lambda ctx: email_service.send_refund_email(
ctx["order_id"], ctx["issue_credit"]["credit_id"]
),
# Email sends are not reversible. Log and alert instead.
compensate=lambda ctx: log.warning(
f"Email may have sent before failure. order_id={ctx['order_id']}"
),
compensation_is_critical=False,
),
])
return await saga.run({"order_id": order_id, "amount": amount})Compensations are not rollbacks. A database rollback undoes the write atomically, as if it never happened. A compensation is a new forward action that logically reverses the first one. The credit reversal creates a new transaction in the payment system. The status update writes a new row. The audit trail shows everything. That’s not a bug in the design, it’s the point.
Pattern 2: Idempotency Keys — Because Agents Will Retry
Agents retry. Networks are flaky, tools time out, and LLMs occasionally hallucinate a failure that didn’t happen. The agent doesn’t know whether the action succeeded or failed. It just knows the response was dirty.
If the action isn’t idempotent, retrying creates duplicates. The customer gets charged twice. The Jira ticket gets created twice. The email sends twice. You’ve seen this class of bug before, just not wearing an agent costume.
The fix is the same one Stripe uses for payments: idempotency keys. Every tool call gets a deterministic key derived from the agent run ID and the step identity. Same key twice, you get the original result. No second execution.
# agents/idempotent_tools.py
import hashlib
import json
import redis
import time
from typing import Any, Callable, Optional
r = redis.Redis(host="localhost", port=6379, db=2, decode_responses=True)
IDEMPOTENCY_TTL = 86_400 # 24 hours -- long enough to cover any realistic retry window
def make_idempotency_key(run_id: str, step_name: str, inputs: dict) -> str:
"""
Derive a stable key from the run context + step identity + inputs.
Same inputs on the same step in the same run always produce the same key.
"""
payload = json.dumps(
{"run_id": run_id, "step": step_name, "inputs": inputs},
sort_keys=True,
)
return "idem:" + hashlib.sha256(payload.encode()).hexdigest()
async def idempotent_call(
run_id: str,
step_name: str,
inputs: dict,
action: Callable[..., Any],
) -> dict:
"""
Wraps any tool action with idempotency guarantees.
On first call: execute the action, store the result.
On retry: return the stored result, skip execution.
"""
key = make_idempotency_key(run_id, step_name, inputs)
# Check if this call already succeeded
cached = r.get(key)
if cached:
result = json.loads(cached)
result["idempotent_replay"] = True
return result
# Lock to prevent concurrent duplicate execution
# (two retries arriving simultaneously)
lock_key = f"{key}:lock"
lock_acquired = r.set(lock_key, "1", nx=True, ex=30)
if not lock_acquired:
# Another instance is currently executing this step.
# Wait briefly and check for the result.
time.sleep(1)
cached = r.get(key)
if cached:
return json.loads(cached)
raise RuntimeError(
f"Concurrent execution detected for step '{step_name}'. Retry later."
)
try:
result = await action(**inputs)
result_payload = {
"result": result,
"idempotent_replay": False,
"executed_at": time.time(),
}
r.setex(key, IDEMPOTENCY_TTL, json.dumps(result_payload))
return result_payload
except Exception:
# Do NOT cache failures. Allow the next retry to try again.
raise
finally:
r.delete(lock_key)One subtlety in the failure case: when the tool action raises an exception, the idempotency record is not written. The next retry executes again. This is correct. You only cache successful outcomes. Caching failures means the retry gets the cached failure back, which is exactly backwards from what you want.
Pattern 3: Checkpoint and Replay — Resuming Long-Running Agents
A short agent run (three or four tool calls) can afford to restart from scratch on failure. An agent that’s been running for ten minutes, made twenty tool calls, and is halfway through a complex research task absolutely cannot.
Checkpoints save agent state after each successful step. On failure, the agent picks up from the last checkpoint instead of step one.
flowchart TD
START([Agent Start]) --> S1[Step 1: Gather Requirements]
S1 --> CP1[(Checkpoint 1)]
CP1 --> S2[Step 2: Search Knowledge Base]
S2 --> CP2[(Checkpoint 2)]
CP2 --> S3[Step 3: Draft Response]
S3 -->|CRASH| FAIL([Process Killed])
FAIL --> RESUME[Agent Resumes]
RESUME --> LOAD[Load Latest Checkpoint]
LOAD --> CP2_RESTORE[(Restore from Checkpoint 2)]
CP2_RESTORE --> S3_RETRY[Retry Step 3 Only]
S3_RETRY --> S4[Step 4: Review and Send]
style CP1 fill:#3b82f6,color:#fff
style CP2 fill:#3b82f6,color:#fff
style FAIL fill:#ef4444,color:#fff
style RESUME fill:#10b981,color:#fff
style CP2_RESTORE fill:#10b981,color:#fff# agents/checkpoint.py
import json
import time
import redis
from dataclasses import dataclass, asdict
from typing import Optional
r = redis.Redis(host="localhost", port=6379, db=3, decode_responses=True)
CHECKPOINT_TTL = 3600 * 6 # 6 hours -- covers any realistic agent run duration
@dataclass
class AgentCheckpoint:
run_id: str
last_completed_step: int
step_name: str
context: dict # full context dict at time of checkpoint
tool_call_log: list # ordered log of every tool call made so far
created_at: float
class CheckpointManager:
def save(self, checkpoint: AgentCheckpoint):
key = f"agent:checkpoint:{checkpoint.run_id}"
r.setex(key, CHECKPOINT_TTL, json.dumps(asdict(checkpoint)))
def load(self, run_id: str) -> Optional[AgentCheckpoint]:
key = f"agent:checkpoint:{run_id}"
data = r.get(key)
if not data:
return None
return AgentCheckpoint(**json.loads(data))
def clear(self, run_id: str):
r.delete(f"agent:checkpoint:{run_id}")
checkpointer = CheckpointManager()
class CheckpointedAgent:
"""
Wraps a list of agent steps with checkpoint-and-resume logic.
On startup, checks for an existing checkpoint and skips already-completed steps.
"""
def __init__(self, run_id: str, steps: list):
self.run_id = run_id
self.steps = steps
async def run(self, initial_context: dict) -> dict:
# Try to resume from checkpoint
checkpoint = checkpointer.load(self.run_id)
if checkpoint:
context = checkpoint.context
tool_call_log = checkpoint.tool_call_log
resume_from = checkpoint.last_completed_step + 1
print(
f"Resuming run {self.run_id} from step {resume_from} "
f"({checkpoint.step_name})"
)
else:
context = dict(initial_context)
tool_call_log = []
resume_from = 0
for i, step in enumerate(self.steps):
if i < resume_from:
continue # already completed in a previous run
try:
result = await step.execute(context)
context[step.name] = result
tool_call_log.append({
"step": i,
"name": step.name,
"result": result,
"timestamp": time.time(),
})
# Save checkpoint after every successful step
checkpointer.save(AgentCheckpoint(
run_id=self.run_id,
last_completed_step=i,
step_name=step.name,
context=context,
tool_call_log=tool_call_log,
created_at=time.time(),
))
except Exception as exc:
# Checkpoint is already saved from the last successful step.
# The next run will resume from there.
raise RuntimeError(
f"Agent run {self.run_id} failed at step '{step.name}' (index {i}). "
f"Checkpoint saved. Retry will resume from step {i}."
) from exc
# Run completed. Clean up the checkpoint.
checkpointer.clear(self.run_id)
return contextThe context dict is what gets checkpointed. It grows step by step, each tool call’s output stored under the step name. Restoring from a checkpoint means restoring that exact dict and skipping the steps that already populated it. The agent has no idea it ever crashed.
Pattern 4: Distributed Locking — Preventing Parallel Agents from Stepping on Each Other
This one happens more than you’d expect. A user submits a request. The agent starts. The user refreshes the page. Your frontend retries. Now two agent instances are running in parallel for the same goal, both reading and writing the same records.
One instance issues the refund. The other started a millisecond earlier, already read the pre-refund state, and issues it again.
You need a lock on the resource, held for the full duration of the agent run, acquired before the first step fires.
# agents/distributed_lock.py
import time
import redis
import uuid
from contextlib import asynccontextmanager
from typing import Optional
r = redis.Redis(host="localhost", port=6379, db=4, decode_responses=True)
class AgentLock:
"""
Distributed lock using Redis SET NX with expiry.
Uses a unique token per lock acquisition to prevent accidental release
by a different holder (e.g., if the lock expires and is re-acquired
while the original holder is still running slowly).
"""
def __init__(self, resource_id: str, ttl_seconds: int = 300):
self.resource_id = resource_id
self.ttl = ttl_seconds
self.key = f"agent:lock:{resource_id}"
self.token: Optional[str] = None
def acquire(self) -> bool:
self.token = str(uuid.uuid4())
acquired = r.set(self.key, self.token, nx=True, ex=self.ttl)
return bool(acquired)
def release(self) -> bool:
# Only release if we still hold the lock (token matches).
# Lua script makes the check-and-delete atomic.
lua = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
return redis.call('DEL', KEYS[1])
else
return 0
end
"""
result = r.eval(lua, 1, self.key, self.token)
return bool(result)
def extend(self) -> bool:
"""Extend the lock TTL for long-running agents."""
lua = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
return redis.call('EXPIRE', KEYS[1], ARGV[2])
else
return 0
end
"""
result = r.eval(lua, 1, self.key, self.token, self.ttl)
return bool(result)
@asynccontextmanager
async def agent_exclusive_run(resource_id: str, timeout_seconds: int = 300):
"""
Context manager that ensures only one agent runs for a given resource at a time.
Raises if the lock cannot be acquired (another agent is already running).
"""
lock = AgentLock(resource_id, ttl_seconds=timeout_seconds)
if not lock.acquire():
raise RuntimeError(
f"Agent already running for resource '{resource_id}'. "
f"Reject this request or queue it for later."
)
try:
yield lock
finally:
lock.release()
# Usage
async def run_refund_agent(order_id: str, amount: float):
async with agent_exclusive_run(f"order:{order_id}"):
# Only one instance can enter this block for a given order_id at a time
saga = SagaOrchestrator(steps=[...])
return await saga.run({"order_id": order_id, "amount": amount})The Lua script on release is not optional. Without it, there’s a race: your token check passes, someone else’s lock expires and gets re-acquired by a third process, and you delete their lock instead of yours. Making it atomic inside Redis closes that window. One round trip, no gap.
Pattern 5: Event Sourcing — The Observability Foundation
The saga, checkpoints, and locks keep your agent from corrupting state. Event sourcing is what lets you understand what actually happened when something goes wrong anyway.
In most agent implementations, state is a dict that gets mutated as steps run. When something breaks, you have the final state and maybe a stack trace. Not the reasoning chain. Not which tool call returned what. Not whether the compensation ran or quietly failed.
Event sourcing flips this. You store a log of every event that changed state rather than the state itself. Pure event sourcing derives current state entirely by replaying the log. In practice, most agent implementations use a hybrid: the event log as a permanent audit trail, checkpoints for fast recovery. You get the observability benefits without the replay overhead. For agents this means appending a record to a durable store (Postgres, not Redis) for every meaningful action: run_started, tool_called, tool_succeeded, tool_failed, step_compensated, run_completed. Every record carries the run ID, step index, step name, full payload, and timestamp.
When a production incident happens, querying all events for a run_id gives you the complete ordered sequence. Every LLM call, every tool call, every failure, every compensation attempt. No guesswork. No reconstructing state from scattered application logs. The event log is the ground truth.
It also makes replay possible. A bug in your tool implementation corrupted ten agent runs? Fix the bug and replay the event logs through it. The inputs are all there. Only the execution changes.
One distinction that matters: the event log lives in Postgres, not Redis. Redis is ephemeral. Locks expire. Checkpoints get cleared on completion. The event log is permanent. You want durable storage you can query six months from now when a customer disputes a charge and you need to know exactly what the refund agent did on that run.
Designing for Failure
The compensation that can’t be undone. Emails, SMS messages, Slack notifications: not reversible. Once sent, they’re out. Design your saga so notification steps are always last, and treat them as best-effort with explicit logging when they fail after earlier steps succeeded. You can’t unsend the email, but you can alert your support team that the confirmation never arrived.
The checkpoint that grows too large. If your agent’s context dict swells to megabytes (raw document content, full API responses), Redis checkpointing becomes slow and expensive. Store large payloads in S3. Put only the reference (a key or URL) in the context. The checkpoint stores the pointer, not the data.
The idempotency key collision. If your run ID isn’t truly unique (weak UUIDs, reused sequence integers), two different agent runs can share keys and get each other’s cached results. Use ULIDs or UUID v4. Never sequential integers for run IDs.
The lock that never releases. If the agent process dies hard (OOM, power loss) while holding a lock and the Redis TTL was set too long, the resource stays locked for minutes or hours. Set TTLs aggressively short and extend them explicitly for long-running steps. The extend() method on the lock above exists for exactly this. Call it in a background heartbeat every 60% of the TTL duration.
The compensation that also fails. Worst case. The forward action succeeded, the compensation failed, and you now have inconsistent world state with no automatic recovery path. Do not swallow these errors. Do not log them quietly and move on. Page someone. This is a data consistency incident.
How This Maps to Classical Distributed Systems
| Distributed Systems Pattern | Agent Equivalent | What It Solves |
|---|---|---|
| Database transaction | Saga + compensation chain | Multi-step consistency without a global coordinator |
| Idempotency key | Per-step idempotent tool calls | Duplicate execution from retries |
| Write-ahead log (WAL) | Event store in Postgres | Full audit trail, crash recovery, replay |
| Checkpoint | Agent checkpoint in Redis | Resume long-running agents without restarting |
| Distributed lock | Redis lock on resource ID | Prevent parallel agents from stomping each other |
| Circuit breaker | Per-tool failure tracking | Stop retrying a broken external service |
| Two-phase commit | Not applicable | Too slow and fragile for async agent workflows |
Two-phase commit is notably absent. Theoretically the right answer for distributed atomic operations. In practice it requires all participants to support the protocol and blocks during the commit phase. Tool APIs don’t implement it. And even if they did, blocking on a commit phase across external APIs with real-world latency is a non-starter. Sagas are the right model here.
The Honest Tradeoff Table
The designing-for-failure section above covers the nuance. Here’s the summary version.
| Consideration | Naive Agent (LangChain default) | Production Agent (Patterns Applied) |
|---|---|---|
| Implementation complexity | Low — framework handles orchestration | High — saga, locks, checkpoints, event store |
| Failure behavior | Non-deterministic, often corrupts state | Predictable, compensates or checkpoints |
| Retry safety | Dangerous — duplicates side effects | Safe — idempotency prevents double execution |
| Debuggability | Dict state at time of crash | Full event log with every decision |
| Parallel execution safety | Races and conflicts | Serialized per resource via distributed lock |
| Infrastructure dependencies | Just the LLM API | LLM API + Redis + Postgres |
| Recovery from crashes | Restart from scratch | Resume from last checkpoint |
The infrastructure cost is real. You’re adding Redis and Postgres as hard dependencies of your agent runtime. For a weekend prototype, that’s genuinely absurd overhead. For a production system making real-world mutations (financial transactions, data writes, external API calls) these patterns aren’t optional extras. They’re the difference between a demo and something you can actually trust at 2 AM when it breaks.
What Comes Next
The patterns here cover single-agent workflows. Multi-agent systems add another layer on top: orchestration vs. choreography, shared state protocols, agent-level circuit breakers. Every one of those problems maps to a distributed systems pattern you’ve already seen. That’s a post on its own.
Get single-agent reliability right first. Every multi-agent bug is harder to debug than its single-agent equivalent. That’s not a guess.
The reason this all feels new is that agent frameworks are young and most people building them come from ML backgrounds, not systems engineering. The distributed systems community spent a decade learning these lessons through painful production incidents. The agent community is about to rediscover all of them.
There’s no shortcut. The physics of distributed systems don’t change just because the decision-making layer is an LLM.