LangGraph agents in production: five controls

9 min readSimon BudziakBy Simon Budziak

On this page10 sections
Title card reading Five controls for agents in production, with the LangGraph name and logo in the top-left corner. A navy background with a stylized state machine of glowing rounded panels connected by a winding gold thread that passes through a tall gold gate, where it splits toward a bright answer panel and a small human silhouette panel.

Getting a LangGraph agent to work in a demo takes an afternoon. Keeping one in production is a different job, and it is mostly not about the model. Across two systems we shipped, the same five controls did the work: an explicit state machine, a trace for every turn, an evaluation gate on every change, idempotent side effects, and confidence routing to a person. We learned them building an AI SDR for Open Loyalty and a patient education assistant for Shen.ai, both published as case studies. This post walks through each control, with the code shape we used and the production failure that made it non-negotiable.

Quality, not capability, is what kills agents in production

Agents are no longer the experiment. In LangChain’s State of Agent Engineering survey of 1,340 respondents (fielded November 18 to December 2, 2025), 57.3% already run agents in production and another 30.4% are actively building toward it. The same report names the blocker: 32% cite quality as their top barrier to production, ahead of latency at 20%. The model usually can produce the right answer. The system around it fails to notice when it does not.

That is the frame for everything below. LangGraph describes itself as “a low-level orchestration framework and runtime for building, managing, and deploying long-running, stateful agents” in the LangGraph overview, and the low-level part is the point. It does not make your agent reliable. It gives you the primitives, state, checkpoints, interrupts, conditional edges, to build reliability in yourself. The five controls below are what we actually build with those primitives when an agent has to survive real traffic.

Control 1: model the agent as an explicit state machine

A production AI agent has to be debugged as a sequence, not as a single answer. The first decision that pays off later is refusing the prompt-loop shape (one big prompt, tools, hope) and modelling the workflow as an explicit agent state machine: named stages, typed state, conditional edges you can read.

For the Open Loyalty AI SDR, conversation state moved through explicit stages for introduction, qualification, retrieval, sales briefing, and CRM write:

class SDRState(MessagesState):
    lead_id: str
    lead: dict
    score: int
    next_action: str
    kb_context: list[Document]
    thread_id: str

graph = StateGraph(SDRState)
graph.add_edge("intro", "qualify")
graph.add_conditional_edges("qualify", needs_product_qa)
graph.add_edge("retrieve_kb", "sales_brief")
graph.add_edge("sales_brief", "crm_write")
graph.add_edge("crm_write", END)

When a lead was routed wrong, the state showed which stage made the call, what retrieval returned, and which tool caused a side effect. A prompt loop can answer “what did the agent say”. An explicit graph answers “what did the agent do, in what order, from what inputs”, and that is the question production incidents actually ask.

Control 2: trace every production turn

You cannot fix a failure you cannot replay. LangSmith gave both systems an end-to-end record per conversation; its observability concepts define the unit plainly: “A trace is a collection of runs for a single operation.” Our traces carried model calls, tool calls, inputs, outputs, latency, prompt version, and run metadata, so a single run ID could answer which prompt produced a behavior and whether a tool call was new or a retry.

This is mainstream practice now, not gold-plating: the same State of Agent Engineering report found 89% of organizations have implemented some form of agent observability, and 62% have tracing detailed enough to inspect individual steps and tool calls. The gap is that far fewer act on it; LLM observability earns its cost on the day a hidden tool failure looks like an odd model response, which is how most of them look. How we wire this for clients is on our LangSmith services page.

The trap: without a trace, a retrieval or CRM failure is indistinguishable from a bad model day. Both Open Loyalty bugs that mattered were found in the trace, not in the transcript.

Control 3: put an evaluation gate on every change

A prompt that looks reliable in staging breaks on messy, multi-part questions. For Open Loyalty we built a small evaluation set from representative real conversations: pricing questions, objections, ambiguous intent, missing fields. Every meaningful change to the prompt, model, retrieval settings, or tool schema ran against the same set before it shipped, checking four concrete things: did the agent route correctly, extract the required fields, stay grounded in approved material, and avoid an unsafe or duplicate side effect. The retrieval side of that gate, chunking, hybrid search and a reranker, has its own build order, which we laid out in Enterprise RAG on LangChain: fix retrieval first.

That turns a debate about one transcript into a regression test, the same discipline LangSmith’s evaluation concepts describe for scoring an application against a dataset. Production failures fed back in: the team classified real failures (wrong routing, unsupported claims, missed fields, tool errors) and promoted the best examples into the set. Industry practice lags here, and it shows: per the same survey, observability adoption sits at 89% while evaluation trails it, with 52.4% running offline evals and 37.3% running online ones. An agent with traces but no eval gate can see itself regress and still ship the regression. We covered how to grade the fuzzy cases in LLM-as-judge evaluation in LangSmith.

Control 4: make every side effect idempotent

The nastiest Open Loyalty bug never appeared in the final assistant message. A small share of conversations produced two CRM updates instead of one. The trace showed the sequence: a CRM write timed out, the agent retried, the second call succeeded, and the first completed shortly afterwards. One conversation, two writes.

The fix was an idempotent tool call boundary, not a better prompt:

crm_write.invoke({
    "lead_id": state["lead_id"],
    "payload": payload,
    "idempotency_key": f"{thread_id}:{turn_id}",
})

Each write got an idempotency key derived from the conversation and turn, model retries and tool retries became separate decisions, and the eval set gained a simulated-timeout case asserting one lead update per turn. LangGraph itself makes this rule official: per the interrupts documentation, when a paused graph resumes, the node restarts from the beginning and any code before the interrupt runs again, which is why side effects in that position must be idempotent. We unpacked that replay boundary in LangGraph human-in-the-loop: interrupt vs middleware. The general rule: any node that touches money, email, or a CRM gets a key, and the retry logic assumes the call may already have happened.

A vertical flow diagram of a LangGraph agent in production. An explicit state machine node feeds a gold deterministic gate checking confidence, which splits into an answer path with idempotent writes and a human review path, with a note that every step lands in a LangSmith trace and behind an evaluation set.
The production shape both systems converged on: a deterministic gate decides the path, the mutation sits behind it with an idempotency key, and every step lands in the trace.

Control 5: route low confidence to a person, in code

For Shen.ai’s patient education assistant, a confident but unsupported explanation was not a UX defect, it was a safety problem. The critical decision, whether a question reaches the normal answer path or a clinician, lived in code rather than in a prompt:

def should_route_to_human(state: PatientEducationState) -> str:
    if state["confidence_score"] < 0.7 or state["requires_review"]:
        return "human_review"
    return "education_module"

graph.add_conditional_edges("retrieve_kb", should_route_to_human)

Confidence gating made uncertainty an explicit system state: low-confidence retrieval could not reach the answer node at all, and the conversation entered human-in-the-loop review instead. The tuned production numbers from that system: about 15% of conversations entered human review, clinicians judged about 80% of those escalations appropriate, and lowering the threshold from 0.8 to 0.7 cut unnecessary reviews by about 40%. Those figures describe that system’s tuning period, not a general benchmark, but the pattern transfers to any domain where a wrong answer costs more than a handoff. Choosing the signal that triggers the gate is its own design problem, covered in deterministic confidence gates.

A prompt asking the model to be cautious does not define when the system must stop. A conditional edge does. OpenAI’s October incident report, where two agents weakened their own monitoring under routine instructions, is the same lesson told from the failure side: OpenAI’s agents disabled their own monitoring.

The infrastructure underneath: durable checkpoints

All five controls assume state survives. A gate that pauses for a person is only useful if the pause persists, so both systems checkpointed to PostgreSQL. LangGraph’s persistence documentation recommends a database-backed checkpointer such as PostgresSaver in production, compiled into the graph and keyed by a stable thread_id:

from langgraph.checkpoint.postgres import PostgresSaver

with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
    graph = builder.compile(checkpointer=checkpointer)

For Shen.ai this is also what let patients leave and return without losing the state the safety checks depended on. Durable execution has one knob worth knowing: the checkpointer durability modes ("exit", "async", "sync") trade performance against crash recovery mid-run, and a long, expensive run that must not be lost is the case for "sync".

Real-world production use cases

The five controls above are not theory. Each comes from a system we shipped, and all three are published case studies:

Case study Industry The system The controls in it
Open Loyalty Loyalty software An AI SDR that qualifies inbound leads, answers from current product knowledge and writes outcomes to the CRM LangGraph state machine, LangSmith traces, an evaluation set from real conversations, idempotent CRM writes
Shen.ai Healthcare A patient education assistant that explains health scan results from approved medical knowledge LangGraph state machine, confidence routing to clinicians, LangSmith traces, PostgreSQL checkpoints
Freedman International Marketing localisation The reasoning core of an AI campaign localisation pipeline: brief analysis, cultural assessment and quality checks as a multi-agent system LangChain and LangGraph on Amazon Bedrock, every model call traced in Langfuse, a deterministic confidence gate, human sign-off at key decision points

Freedman is also the proof that the controls transfer: its tracing runs on Langfuse rather than LangSmith, and the control is the trace, not the vendor. The gate, the idempotent boundary and the human route look the same whichever observability backend records them.

The takeaway

LangGraph is trusted in production by companies like Klarna, Uber, and J.P. Morgan per its own overview page, and the reason is not that it makes agents smart. It is that its primitives let you make agents inspectable and governable. The five controls are one sentence each: model the workflow as an explicit state machine, trace every turn, gate every change on an evaluation set built from real conversations, give every side effect an idempotency key, and route uncertainty to a person through a conditional edge rather than a prompt. Both systems above shipped with all five, and every production incident since has been debugged through one of them. If you want the longer version of what we build on this stack, start with our LangGraph services page or the two case studies linked above.

Sources

Last updated:

How we build on LangChain in production