LangGraph production architecture: what I learned building an eight-agent system

What does a production LangGraph architecture look like?

A production LangGraph architecture is a supervisor routing work to specialized agents over an explicit, typed graph state. Every step is checkpointed, so a run can pause, resume after a crash and be replayed for debugging. Tool calls are bounded by schemas, retry policies and attempt limits rather than left to the model's judgment. Evaluation runs as an acceptance gate on every change, not as a one-off demo. Human approval, where it is needed, is a node in the graph rather than a message in a chat window. That is the shape I arrived at after building and operating a LangGraph supervisor over eight agents in Jozveh-AI, and it is the shape I now use as the starting point in LangGraph development work for clients.

The rest of this guide walks through each of those decisions: why a supervisor, how to design the state, how checkpoints and retries actually behave, how to keep tool calling from becoming the main source of incidents, where humans fit, what to log, and how to evaluate. I close with what failed first in my own system and a checklist you can run against yours.

When is a supervisor over specialized agents better than one agent?

A single agent with a good prompt and a handful of tools is the right first build, and the LangChain multi-agent documentation says so directly: a single agent with the right tools and prompt can often achieve similar results, and the split into multiple agents pays off when one agent has too many tools and starts routing badly, when subtasks need specialized context, or when steps must happen in a fixed order (multi-agent patterns). I apply a blunter test. If I cannot write a one-sentence job description for an agent, it should not exist. If two agents share most of their tools and prompt, they are one agent.

In Jozveh-AI the work naturally splits into stages with different inputs, different failure modes and different acceptance rules: ingesting heterogeneous source material, retrieving and reranking evidence from a Neo4j graph, generating cited sections, checking claims, filtering ungrounded text and laying out a Persian, right-to-left document. Boundaries like those are what the eight agents are drawn along: each agent owns one job. The supervisor owns none of them; it only reads state and decides who runs next. That separation is what makes the system debuggable, because every agent can be tested in isolation with a fixed state fixture.

The supervisor is the only stateful party in my designs, and it does not hold a growing message history. It holds a small, typed state record, and each agent gets only the slice it needs, runs in its own clean context, and returns a typed update. Context isolation is the point: the main loop never bloats, and an agent's prompt never has to compete with another agent's transcript.

How should you design the graph state?

The state is the contract between agents, so I design it before I write a single prompt. In LangGraph the state is a TypedDictor Pydantic model, and each key has a reducer that decides how a node's update is applied; by default the update replaces the previous value, and an Annotated reducer such as operator.add or add_messages appends instead (Graph API). Three rules have held up for me.

Keep the state small and explicit

Put outcomes in state, not transcripts. A list of retrieved evidence ids, the current section index, the judge's verdict and the attempt counter belong in state. A 40-message chat log does not. Large state makes every checkpoint heavier and every agent prompt noisier.

Give every accumulating field an intentional reducer

The replace-by-default behavior is the number one source of "my agent forgot what it found" bugs. If a field should accumulate across agents, say so with a reducer. If it should be overwritten, leave the default and document why.

Make routing decisions data, not prose

The supervisor routes on fields such as status, attempts and verdict, not on free text. That is what lets me write a conditional edge as a plain function that I can unit test.

from operator import add
from typing import Annotated, Literal, TypedDict

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.graph import END, START, StateGraph
from langgraph.types import RetryPolicy


class PipelineState(TypedDict):
    source_ids: list[str]
    evidence: Annotated[list[dict], add]   # accumulates across agents
    draft: str                             # replaced by the latest writer run
    verdict: Literal["pending", "accepted", "rejected"]
    attempts: int


def retrieve(state: PipelineState) -> dict:
    ...  # returns {"evidence": [...]}


def write_section(state: PipelineState) -> dict:
    ...  # returns {"draft": "...", "attempts": state["attempts"] + 1}


def judge(state: PipelineState) -> dict:
    ...  # returns {"verdict": "accepted" | "rejected"}


def route_after_judge(state: PipelineState) -> str:
    if state["verdict"] == "accepted":
        return END
    if state["attempts"] >= 3:
        return "escalate"                  # bounded: never loop forever
    return "write_section"


builder = StateGraph(PipelineState)
builder.add_node("retrieve", retrieve, retry_policy=RetryPolicy(max_attempts=3))
builder.add_node("write_section", write_section)
builder.add_node("judge", judge)
builder.add_node("escalate", lambda s: {"verdict": "rejected"})

builder.add_edge(START, "retrieve")
builder.add_edge("retrieve", "write_section")
builder.add_edge("write_section", "judge")
builder.add_conditional_edges("judge", route_after_judge)
builder.add_edge("escalate", END)

with PostgresSaver.from_conn_string(DB_URL) as checkpointer:
    checkpointer.setup()                   # creates the checkpoint tables once
    graph = builder.compile(checkpointer=checkpointer)
    config = {"configurable": {"thread_id": "doc-42"}}
    graph.invoke({"source_ids": ["s1"], "evidence": [], "draft": "",
                  "verdict": "pending", "attempts": 0}, config)

How do checkpoints, retries and recovery work in practice?

A checkpointer saves a snapshot of the graph state after each step, keyed by a thread_id passed in the run config. The documented options are an in-memory saver for development and Postgres or SQLite savers for durable storage; the same checkpoint history is what powers pause and resume, fault tolerance and replay (persistence). In production I treat the in-memory saver as a test fixture only. If the process dies with an in-memory saver, every in-flight document dies with it.

Retries live at two levels and it matters which one you use. Node-level retries are configured with a RetryPolicy on add_node; its fields are max_attempts, initial_interval, backoff_factor and a retry_on predicate, and by default it retries on most exceptions but not on programming errors such as ValueError or TypeError, and on HTTP errors only for 5xx responses (retry policies). That is the right tool for transient provider and database failures. Graph-level retries, the "judge rejected, write again" loop, are a routing decision, and they must be bounded by a counter in state. The attempts field in the snippet above is not decorative; without it a strict judge and a weak writer will loop until the budget is gone.

Recovery is then a matter of invoking the graph again with the same thread_id. The checkpointer restores the last saved state and execution continues from the failed step. Two consequences follow that people discover the hard way. First, a node must be safe to run twice, because a crash between the side effect and the checkpoint will re-run it. I make writes idempotent by keying them on ids that live in state. Second, anything you want to inspect after a failure must be in state, because that is all the checkpoint contains.

How do you keep tool calling reliable?

Tool calling is where most production incidents in agent systems originate, in my experience, because it is the boundary between the model's output and code that has real effects. The Claude tool-use contract is simple: you send a tool definition with a name, a description and a JSON input_schema; the model answers with stop_reason: "tool_use" and one or more tool_use blocks; you execute and return a tool_result keyed by tool_use_id. The docs also note that a model may guess a missing required parameter rather than ask (tool use overview). OpenAI's function calling follows the same shape. Everything I do around that contract is about narrowing what can go wrong.

• Validate every argument before executing. The schema tells the model what to send; it does not stop the model from sending something else. I parse tool inputs with a strict model and return a structured error as the tool result so the agent can correct itself.
• Return errors as data, not exceptions. A tool that raises kills the node; a tool that returns {"ok": false, "reason": ...} gives the model a chance to recover and gives me a log line.
• Bound the number of calls per turn and per run. Both a per-node limit and a run-level budget, stored in state, so a confused agent cannot drain the account.
• Separate read tools from write tools. Read tools can retry freely. Write tools get idempotency keys and, where the effect is external, a human gate.
• Keep tool descriptions short and literal. Describe what the tool does and when not to call it. Most routing mistakes I have debugged were description mistakes.

Where does a human belong in the loop?

LangGraph's interrupt() pauses a run at any point inside a node, surfaces a JSON-serializable payload to the caller and waits; the run resumes when the caller invokes the graph again with Command(resume=...), and the resume value becomes the return value of the interrupt call. This requires a checkpointer and a thread_id, and the node restarts from its beginning on resume, so any code before the interrupt runs again (interrupts). That last sentence is the one to read twice. Put the interrupt at the top of the node, or make everything before it idempotent.

from langgraph.types import Command, interrupt


def approve_publish(state: PipelineState) -> dict:
    # Interrupt first: everything before this line re-runs on resume.
    decision = interrupt({
        "question": "Publish this document?",
        "draft_preview": state["draft"][:500],
        "verdict": state["verdict"],
    })
    if decision.get("approved"):
        return {"verdict": "accepted"}
    return {"verdict": "rejected"}


# First call pauses at the interrupt and persists the checkpoint.
result = graph.invoke(initial_state, config)

# Later, from a review UI or an operator tool, resume the same thread.
graph.invoke(Command(resume={"approved": True}), config)

I use human gates for exactly two things: irreversible external effects, and outputs the judge marked as low confidence. Everything else runs unattended, because a human gate that fires on every run is a queue, not a safeguard, and people start approving without reading.

What should you log to debug an agent in production?

The question I need to answer during an incident is always the same: which node, on which thread, with which state, called which tool with which arguments, and what came back. So that is the log record. LangGraph's streaming API gives most of it for free: updates mode emits each node's state delta, messages mode streams model tokens with metadata, custom mode carries anything a node emits through get_stream_writer, and debug mode combines checkpoint and task events (streaming). I consume updates and custom in the worker and write one structured line per node step.

Each line carries:

• thread_id, node name, step number and attempt number.
• Model, prompt version, input and output token counts, and cost computed from those.
• Every tool call with validated arguments and a truncated result, plus its latency.
• The judge's verdict and reason, and which grounding filter fired, if any.
• A hash of the state before and after, so I can diff without storing full snapshots twice.

Token and cost accounting per step is not optional. Cost surprises in agent systems come from loops more than from any single expensive call, and a loop is invisible in a per-request total but obvious in a per-step series. That is why both Jozveh-AI and OmidGPT carry per-token cost accounting as a built component rather than a reporting afterthought. This is also where model routing and caching decisions come from; I cover that side in the semantic caching and model routing guide.

How do you evaluate an agent beyond prompt testing?

Prompt testing checks that one node produces a plausible answer on a few examples. Evaluation checks that the whole graph produces acceptable output on a fixed dataset, every time the code changes. For a RAG-backed agent the layers I run are: RAGAS metrics over retrieval and answer quality; a custom claim-level LLM judge that checks each sentence against its cited evidence; and post-generation grounding filters that drop or flag text with no support. The judge is also a node in the graph, so its verdict drives routing at runtime and doubles as the offline metric. The full design is in RAG evaluation with RAGAS and an LLM judge.

The part that makes this an architecture decision rather than a QA detail is that tests are the acceptance criteria. In Jozveh-AI the suite is 673 tests across 77 files. The tests that give me the most confidence are not model tests at all: they cover routing functions, reducers, tool validators, idempotency and the judge's contract, and they run without a model call. That is what a typed state buys you: most of the graph can be tested as ordinary Python.

What fails first, in my experience

Across the systems I have built and the codebases I have reviewed in an AI technical audit, the failures arrive in a consistent order, and none of them are the model being too weak.

1. Unbounded loops. A judge and a writer disagree forever. The fix is an attempt counter in state and a conditional edge that escalates.
2. Silent state overwrites.A field that should accumulate uses the default replace reducer. The symptom is an agent that "forgets" evidence between steps.
3. Non-idempotent nodes under retry. A crash after a write but before the checkpoint re-runs the write. Duplicate documents, duplicate messages, duplicate charges.
4. Tool calls with fabricated arguments. The model guesses an id. Without input validation, the tool executes against the wrong record.
5. Retrieval quality, not generation quality. Faithfulness problems trace back to what was retrieved and how it was reranked far more often than to the writer prompt.
6. Cost drift. Nobody notices a loop until the invoice, because cost is aggregated per request instead of per step.

What I observed in Jozveh-AI

The second lesson was about the supervisor. A supervisor that reasons in free text about what to do next is pleasant to demo and impossible to test; a supervisor that routes on typed fields can be covered by ordinary unit tests, and that is the only kind I ship now. The third lesson was that the judge earns its place twice: as the runtime gate that sends weak sections back, and as the offline metric that tells me whether a change to retrieval helped. The full write-up is in the Jozveh-AI case study.

If you are deciding whether your own system needs this structure, the honest answer is that most prototypes do not, and most products that survive contact with users do. The transition is the work I scope in an AI agent engineering sprint: state design, checkpointing, retry handling, tool hardening and evaluation tests, delivered with the graph.

Production readiness checklist

• Every agent has a one-sentence job description; no two agents share most tools and prompt.
• State is a typed schema; every accumulating field has an explicit reducer.
• Supervisor routing reads typed fields and is covered by unit tests without model calls.
• A durable checkpointer (Postgres or equivalent) is configured; the in-memory saver is test-only.
• Transient failures use a node RetryPolicy; semantic retries use a bounded counter in state.
• Every node is safe to run twice; external writes carry idempotency keys.
• Tool inputs are validated before execution; tool errors return as data.
• Tool calls are capped per node and per run, with the budget stored in state.
• Human gates use interrupt() at the top of the node, only for irreversible effects or low-confidence output.
• One structured log line per node step, with thread, attempt, tokens, cost and tool calls.
• Per-step cost accounting exists and is charted, so loops are visible before the invoice.
• Evaluation (RAGAS, judge, grounding filters) runs on a fixed dataset on every change.
• The judge is a graph node, so runtime gating and offline metrics share one contract.
• A runbook explains how to resume, replay and cancel a thread.

Further reading

• LangGraph persistence: checkpoints, threads and savers
• LangGraph interrupts and human-in-the-loop
• LangGraph Graph API: state, reducers, edges and retry policies
• LangGraph streaming modes
• LangChain multi-agent patterns
• Claude tool use overview
• RAG evaluation with RAGAS and an LLM judge (this site)