Multi-Agent Orchestration: Choreography, Circuit Breakers, and the Coordination Problem Nobody Designed For

Picture this. You wire up three agents to handle a research task: a Researcher agent pulls sources, a Writer agent drafts a summary from those sources, and a Fact-Checker agent verifies the draft against the original material before it ships. In your demo, it’s magic. Three agents, one clean pipeline, a polished report out the other end.

Then in production, the Researcher times out on a slow API call. It doesn’t crash, it just returns an empty result. The Writer, having no concept that “empty” means “something went wrong upstream,” cheerfully drafts a summary out of nothing but the task description and its own priors. The Fact-Checker checks the draft against an empty source set, finds no contradictions (there’s nothing to contradict), and approves it. You ship a confident, fluent, entirely fabricated report to a customer.

Nobody’s prompt was bad. Every agent did exactly what it was told. The system still failed, because nobody designed for what happens when one node in a pipeline degrades instead of dying cleanly.

This Is Not a New Problem

I said the same thing in the last post about a single agent losing its place mid-task: strip away the LLM and you’re looking at a distributed system. With multiple agents, the resemblance stops being a metaphor and becomes literal. You have independent processes, each with its own state and failure modes, communicating over unreliable channels, trying to make forward progress on a shared goal. That’s a distributed system by definition, whether or not any of the nodes happen to call themselves “agents.”

Which means the two decades of coordination patterns from microservices apply directly. The main one you have to pick first: orchestration vs. choreography.

flowchart TB
    subgraph Orchestration
        direction TB
        O[Orchestrator Agent] -->|1. dispatch| R1[Researcher]
        R1 -->|result| O
        O -->|2. dispatch| W1[Writer]
        W1 -->|result| O
        O -->|3. dispatch| F1[Fact-Checker]
        F1 -->|result| O
        O -->|final output| OUT1([Report])
    end

    subgraph Choreography
        direction TB
        R2[Researcher] -->|event: sources.ready| BUS[/Event Bus/]
        BUS -->|triggers| W2[Writer]
        W2 -->|event: draft.ready| BUS
        BUS -->|triggers| F2[Fact-Checker]
        F2 -->|event: report.verified| BUS
        BUS -->|triggers| OUT2([Report])
    end

    style O fill:#4f46e5,color:#fff
    style BUS fill:#f59e0b,color:#000
    style OUT1 fill:#10b981,color:#fff
    style OUT2 fill:#10b981,color:#fff

Orchestration puts one agent (or plain code, which is usually the better call) in charge. It calls the Researcher, waits, validates the result, calls the Writer, waits, validates, calls the Fact-Checker. Every decision about what happens next lives in one place. When the Researcher comes back empty, the orchestrator is the one piece of code positioned to notice and stop the pipeline right there.

Choreography has no central brain. Each agent listens for events and reacts. The Researcher finishes and emits sources.ready. The Writer is subscribed to that event and wakes up. Nobody is watching the whole pipeline, because there is no “whole pipeline” from any single agent’s point of view, just a series of local reactions to messages.

Both are legitimate. But for agent systems specifically, choreography has a sharp edge that it doesn’t have for microservices: microservices fail loudly (5xx, timeout, exception). Agents fail quietly (a plausible-sounding wrong answer). A choreographed system has no natural place to catch “the output looks fine but is nonsense,” because no single component ever sees the whole picture. That’s exactly what happened in the empty-Researcher scenario above. An orchestrator that explicitly validated “did the Researcher actually return sources?” before calling the Writer would have caught it in one line.

Default to orchestration until you have a specific reason not to. Choreography earns its keep when agents are genuinely independent and loosely coupled, think a swarm of monitoring agents each watching a different system and firing alerts. For a pipeline where each step depends on the last one’s output being correct, not just present, put a human-legible chokepoint in the middle.

Building the Orchestrator

Here’s the orchestrator for the research pipeline, written as plain code, not another agent. The point of an orchestrator is to be the boring, deterministic, fully-testable part of the system.

# orchestrator/research_pipeline.py

from dataclasses import dataclass
from enum import Enum, auto


class StepStatus(Enum):
    OK = auto()
    EMPTY = auto()
    FAILED = auto()


@dataclass
class StepResult:
    status: StepStatus
    data: dict | None
    error: str | None = None


class PipelineAbortedError(Exception):
    """Raised when a step's output can't be trusted enough to continue."""


def run_research_pipeline(task: str) -> dict:
    sources = call_agent_with_guard(
        agent=researcher_agent,
        input={"task": task},
        validate=lambda r: bool(r.get("sources")),
        step_name="research",
    )

    draft = call_agent_with_guard(
        agent=writer_agent,
        input={"task": task, "sources": sources["sources"]},
        validate=lambda r: bool(r.get("draft")),
        step_name="draft",
    )

    verified = call_agent_with_guard(
        agent=fact_checker_agent,
        input={"draft": draft["draft"], "sources": sources["sources"]},
        validate=lambda r: "verdict" in r,
        step_name="verify",
    )

    if verified["verdict"] != "confirmed":
        raise PipelineAbortedError(
            f"Fact-checker rejected draft: {verified.get('reason')}"
        )

    return {"report": draft["draft"], "sources": sources["sources"]}


def call_agent_with_guard(agent, input: dict, validate, step_name: str) -> dict:
    result = agent.run(input)

    if not validate(result):
        # This is the line that would have caught the empty-Researcher bug.
        # The pipeline stops here instead of feeding garbage downstream.
        raise PipelineAbortedError(
            f"Step '{step_name}' returned an untrustworthy result: {result}"
        )

    return result

Notice what’s absent: no step calls the next step directly. The Writer agent has no idea the Fact-Checker exists. Every handoff passes through call_agent_with_guard, which is the one place in the system responsible for deciding whether a result is good enough to build on. That’s the whole value of orchestration in one function. It’s not about control, it’s about having a single seam where you can enforce “garbage in, no shipping out.”

Agent-Level Circuit Breakers

Validation catches bad results. It doesn’t help when an agent is bad on average right now, say, the model provider behind your Fact-Checker is degraded and returning malformed JSON on 80% of calls. Retrying each call individually just burns tokens and time while the underlying problem persists. This is precisely the failure mode circuit breakers were invented for in the Netflix Hystrix era, and it ports over unchanged.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failure rate > threshold
    Open --> HalfOpen: cooldown elapsed
    HalfOpen --> Closed: trial call succeeds
    HalfOpen --> Open: trial call fails

    note right of Closed
        Calls pass through normally.
        Failures are counted.
    end note

    note right of Open
        Calls fail fast, no agent invocation.
        Fallback path used instead.
    end note

    note right of HalfOpen
        One trial call allowed through
        to test if the agent recovered.
    end note

# orchestrator/circuit_breaker.py

import time
from dataclasses import dataclass, field


@dataclass
class CircuitBreaker:
    failure_threshold: int = 5
    cooldown_seconds: int = 30

    _failures: int = field(default=0, init=False)
    _state: str = field(default="closed", init=False)
    _opened_at: float = field(default=0.0, init=False)

    def call(self, fn, *args, **kwargs):
        if self._state == "open":
            if time.monotonic() - self._opened_at < self.cooldown_seconds:
                raise AgentUnavailableError("Circuit open, refusing to call agent")
            self._state = "half_open"

        try:
            result = fn(*args, **kwargs)
        except AgentCallError:
            self._on_failure()
            raise
        else:
            self._on_success()
            return result

    def _on_failure(self):
        self._failures += 1
        if self._state == "half_open" or self._failures >= self.failure_threshold:
            self._state = "open"
            self._opened_at = time.monotonic()

    def _on_success(self):
        self._failures = 0
        self._state = "closed"


class AgentUnavailableError(Exception):
    pass


class AgentCallError(Exception):
    pass

Wrap each agent’s call site in one of these, tracked separately per agent. When the Fact-Checker’s breaker trips, the orchestrator has an explicit decision to make: fall back to a cheaper deterministic check, queue the report for human review, or fail the whole pipeline outright. All three are better than hammering a degraded model 400 times an hour and hoping.

The Shared State Trap

The other classic distributed-systems failure that shows up almost immediately in multi-agent systems: agents stepping on each other’s state. It’s tempting to give every agent read/write access to one shared context object, “the task state,” so nobody has to explicitly pass data around. This is the agent-system equivalent of every microservice writing directly to the same database table, and it goes wrong the same way.

sequenceDiagram
    participant W as Writer Agent
    participant S as Shared Task State
    participant F as Fact-Checker Agent

    W->>S: read draft_v1
    F->>S: read draft_v1
    Note over W,F: Both agents working from the same version
    W->>S: write draft_v2 (revised wording)
    F->>S: write verified=true (against draft_v1!)
    Note over S: State now says "verified" for a<br/>draft that no longer exists

The Fact-Checker verified draft_v1, but by the time it wrote its verdict, the Writer had already overwritten the state with draft_v2. The system now confidently reports a verified draft that was never actually checked. Nobody’s code was buggy in isolation, this is a plain lost-update race condition, the same one that motivated optimistic locking and version columns in every database that’s ever shipped.

The fix is the same fix: agents don’t share mutable state, they exchange immutable messages.

# Each agent receives an immutable snapshot and returns a new one.
# Nobody mutates shared state; the orchestrator owns the sequence.

@dataclass(frozen=True)
class TaskState:
    version: int
    sources: tuple[str, ...]
    draft: str | None = None
    verdict: str | None = None
    verified_draft_version: int | None = None


def writer_step(state: TaskState) -> TaskState:
    new_draft = writer_agent.run(sources=state.sources)
    return dataclasses.replace(state, version=state.version + 1, draft=new_draft)


def fact_checker_step(state: TaskState) -> TaskState:
    verdict = fact_checker_agent.run(draft=state.draft, sources=state.sources)
    return dataclasses.replace(
        state,
        verdict=verdict,
        verified_draft_version=state.version,  # pin the exact version checked
    )

verified_draft_version is the whole fix. If any later step tries to ship a draft whose version doesn’t match what was actually verified, that’s a one-line check, not a debugging session six weeks from now trying to figure out why a “verified” report contained an unverified claim.

Tradeoffs

Consideration	Orchestration	Choreography
Failure visibility	Centralized, easy to log and alert on	Diffuse, needs distributed tracing to reconstruct
Coupling	Orchestrator knows about every agent	Agents only know event schemas
Adding a new step	Edit the orchestrator	Publish a new event, subscribe to it
Catching “quietly wrong” output	Natural, one chokepoint per handoff	Hard, no component sees the full picture
Scales to how many agents	Comfortable up to a dozen or so	Better for large, loosely related agent fleets
Operational model	One thing to reason about	Distributed system, full stop

Start with orchestration. It’s more code up front, but it’s the code that lets you sleep at night, because the one place a bad handoff can hide is the one place you’re already forcing every result to pass through a validator. Reach for choreography only once you have enough independent agents that a central orchestrator would itself become the bottleneck, and by then you’ll also need the observability tooling to make choreography debuggable in the first place.

What Comes Next

This covers getting a fixed pipeline of agents to hand off work correctly. It doesn’t cover how you’d know if the Fact-Checker started silently getting worse at its job, that’s an evaluation and observability problem, and it’s a different post. Nor does it cover what happens when a human needs to step into the loop mid-pipeline to approve or correct a decision an agent isn’t confident about. Both build directly on the chokepoints this orchestrator already gives you, which is exactly why it was worth building the chokepoints first.