LangGraph fault tolerance: timeouts, retries, drain

One hung model call or one flaky API should not take down a long LangGraph run, and version 1.2 added three tools beside the retry policy it already had: per-node timeouts, error handlers and graceful drain. Time out every async network node, keep the default retries, route exhausted failures to compensation, and drain on SIGTERM with a grace period long enough for the drain to land. We cover what each mechanism does, the order they fire in, the retry arithmetic that decides your shutdown budget, and the defaults we change.

What LangGraph 1.2 added for failing nodes

The LangChain changelog dates langgraph v1.2.0 to May 12, 2026 and describes it as finer-grained control over node execution: timeouts, error recovery and graceful shutdown. The current release on PyPI is 1.2.20, uploaded on November 5, 2026, so the API has had six months of patch releases behind it. One limit from the changelog matters before you design around it: timeouts and error handlers are Python-only, while retry policies keep working in both Python and TypeScript.

The fault tolerance guide is the reference for all four, and it fixes their order. When a node attempt raises, including a timeout, the retry policy decides whether to try again. “Only after retries are exhausted does the error handler run.” Drain sits outside that loop entirely: it acts between supersteps, so it never interrupts any of the other three.

Retries rerun the whole node, so side effects need a key

RetryPolicy defaults are sensible and worth knowing by heart, because they drive everything downstream. Per the guide: max_attempts=3 counting the first try, initial_interval=0.5 seconds, backoff_factor=2.0, max_interval=128.0 seconds and jitter on. That is a textbook agent retry policy with exponential backoff: a failed node waits about half a second, then about a second, then gives up.

The default retry_on is narrower than “everything”. It skips a list of built-in errors, ValueError, TypeError, RuntimeError and OSError among them, and for exceptions from requests and httpx it retries only 5xx responses. So a 429 raised through those libraries is not retried unless you say so, while an SDK’s own exception class falls under the retry-anything rule. Check which one your client raises:

from langgraph.types import RetryPolicy, default_retry_on

def retry_on(exc: BaseException) -> bool:
    if is_rate_limit(exc):  # your 429 check
        return True
    return default_retry_on(exc)

builder.add_node("call_api", call_api,
    retry_policy=RetryPolicy(max_attempts=3, retry_on=retry_on))

The part that bites is the unit of retry. The guide says a retry “re-runs the entire node function from the start”, which gives at-least-once execution. Checkpoints are written at superstep boundaries, so graph writes from a failed attempt are discarded, but a payment, an email or a CRM update inside the node already happened. In the guide’s words: “Application code must make those effects idempotent.”

We learned this one in production before the API existed. On the Open Loyalty AI SDR, a CRM write timed out, the agent retried, the retry succeeded and then the first call completed too: one conversation, two lead updates. The fix was an idempotent tool call, a key derived from the conversation and turn, which we described in LangGraph agents in production: five controls. A retry policy without that key multiplies side effects; with it, the policy is safe to leave on everywhere.

A timeout is a retry trigger, not a verdict

You can pass timeout= as seconds, a timedelta, or a TimeoutPolicy with two separate clocks. run_timeout is a hard wall-clock cap on one attempt and is never refreshed. idle_timeout resets on progress: state writes, stream output, child tasks, and any LangChain callback from the node, so a model that is still streaming tokens does not trip it.

from langgraph.types import RetryPolicy, TimeoutPolicy

builder.add_node(
    "call_model",
    call_model,
    timeout=TimeoutPolicy(run_timeout=120, idle_timeout=30),
    retry_policy=RetryPolicy(max_attempts=3),
)

That pairing is the one we reach for on any node that calls a model: the idle clock catches a stalled connection in 30 seconds, the run clock caps a model that keeps talking. For a batch loop that emits nothing, refresh_on="heartbeat" plus runtime.heartbeat() per batch makes the idle clock mean what you intend.

Two caveats from the docs decide whether the timeout works at all. First, timeouts apply only to async nodes, and a sync node with a timeout is rejected at compile time. Second, the TimeoutPolicy reference says they rely on asyncio cancellation, so a node that blocks the event loop with time.sleep() or CPU-bound work will not see the timeout fire until the loop is released. Python’s asyncio documentation adds the general warning: asyncio.timeout() is built on cancellation and can misbehave if a coroutine swallows CancelledError. Wrap blocking calls in asyncio.to_thread and never catch cancellation bare.

When the clock fires, LangGraph raises NodeTimeoutError, clears the attempt’s writes and hands it to the retry policy. The NodeTimeoutError reference explains a deliberate choice: it does not inherit from the built-in TimeoutError, a subclass of OSError, precisely so the default policy treats it as retryable. That is why we read a timeout as a reason to try again. An agent timeout tells you the caller stopped waiting, never that the remote action did not happen, which is the Open Loyalty bug in one sentence.

The error handler is where a failure becomes a decision

Once retries are spent, the node’s error_handler gets the state and a NodeError carrying the node name and the exception, and it returns a Command that updates state and routes elsewhere. The guide’s own example is a payment node that compensates and jumps to finalize, a saga in one small function:

from langgraph.errors import NodeError
from langgraph.types import Command

def payment_error_handler(state, error: NodeError):
    return Command(
        update={"status": f"compensated: {error.error}"},
        goto="finalize",
    )

This turns agent recovery into a graph edge you can see in a trace. A handler can route to a human review node, a cheaper fallback model, or a refund step. Three rules from the guide shape how we write them. Failure provenance is checkpointed, so if the process dies before the handler finishes, the handler sees the same NodeError on resume. An interrupt() is never routed to the handler; interrupts bubble past both retries and handlers, which keeps human-in-the-loop pauses out of your error path, a boundary we covered in LangGraph human-in-the-loop: interrupt vs middleware. And if the handler itself raises, the exception propagates as if the node had none, so a handler should do one small, safe thing.

set_node_defaults() applies a default retry policy, handler and timeout at compile time, with per-node values overriding it; the handler default skips error-handler nodes, since a handler must never catch itself. Defaults on a parent graph are not inherited by subgraphs, so set them on each compiled subgraph too.

Graceful drain finishes the superstep and saves the checkpoint

Drain is for the moment your platform wants the process gone. You create a RunControl, pass it to invoke or stream, and call request_drain() from a signal handler. The guide is precise about the contract: “Drain is cooperative and operates between supersteps, never preempting work that is already running”. If more supersteps remain, the run raises GraphDrained with a saved, resumable checkpoint.

import signal
from langgraph.errors import GraphDrained
from langgraph.runtime import RunControl

control = RunControl()
signal.signal(signal.SIGTERM,
    lambda *_: control.request_drain("sigterm"))
try:
    graph.invoke(inputs, config, control=control)
except GraphDrained as e:
    log.info("drained: %s", e.reason)

On the next start, graph.invoke(None, config) with the same thread_id picks the run up. The checkpointer docs explain why nothing finished is redone: when one node in a superstep fails, LangGraph keeps the pending writes of the nodes that succeeded and does not rerun them. That is durable execution working as designed, as long as the agent checkpoint actually reaches the database; the docs note that durability="async" carries a small risk of a lost checkpoint on a crash, and "sync" persists before the next step starts.

What are the top 5 deployment strategies?

The five usually listed are recreate, rolling, blue-green, canary and shadow. For a long-running agent the difference between them matters less than what they share: whenever one of them replaces the running version, recreate and rolling at once, blue-green and canary at cutover, shadow when the mirrored version is promoted, old workers are retired, and on Kubernetes that means a SIGTERM. The Kubernetes pod lifecycle docs set the default grace period at 30 seconds, after which the remaining processes get KILL. A canary that rolls back, a rolling update that cycles pods, a blue-green cutover that drains the old color: each one is a shutdown, many times a day, aimed at runs that may be mid-node.

That makes drain the code path a long-running agent hits on every release, not just in outages, and a run that dies there without it reruns its current node, side effects included.

Size the drain budget from the retry math

Here is the trap the docs state in one table row and leave you to multiply. When drain is requested while a node is retrying, the retry loop runs to exhaustion or success, and only then does drain take effect. So the worst case for one node is all its attempts plus the backoff between them.

With run_timeout=60 and the default policy, that is three attempts of 60 seconds plus waits of about 0.5 and 1 second: roughly 181.5 seconds before drain can land, six times the Kubernetes default grace. The process gets killed long before the clean stop arrives. The fault tolerance guide warns of the same gap from the other side: request_drain() does not cancel running asyncio tasks, so pair it with a timeout for a hard upper bound.

Our rule is to compute that number per graph and make the platform wait at least that long. On LangSmith Deployment, the hosted runtime from the makers of LangSmith, the knob already exists: the self-hosted environment variables list BG_JOB_SHUTDOWN_GRACE_PERIOD_SECS, which defaults to 180 seconds with a maximum of 3,600, after which the server forces termination. Note that 180 is 1.5 seconds short of the example above. Per the scalability and resilience page, a run that does not finish in time is put back in the queue, and the same variables page says a run canceled by a shutdown resumes from its last checkpointed step. On your own Kubernetes, raise terminationGracePeriodSeconds to the computed budget, or cut the budget: a lower run_timeout, or two attempts on nodes where the third rarely succeeds.

How we set ceilings on our own agent runs

We run our own multi-step agent jobs, a controller that hands work to sub-agents, and they follow the same shape LangGraph now ships. Every step has a ceiling. A step past its ceiling is killed and replaced once with the same brief, and a second miss ends the run as failed, loudly, with its work left where a person can resume it. That is a retry policy of two attempts, a timeout, and an error handler that routes to a person, written by hand.

The ceiling exists because of a failure we had before it: one job reported “still waiting” for seven hours while the thing it was waiting for had already failed. An unbounded wait cannot tell slow from dead, so every external wait gets a deadline that changes what the run does next. That is the whole case for idle_timeout, and for an agent circuit breaker above it when a dependency keeps failing across runs.

The trap: an error handler that quietly writes a fallback answer turns a loud failure silent. Route to a person or a visible failed state, and count how often the handler fires.

The takeaway

With 1.2, a production graph has all four controls, and they compose in a fixed order: timeout, retry, handler, with drain waiting at the superstep boundary. Put a TimeoutPolicy on every async node that touches the network, keep the default RetryPolicy but add your provider’s 429, key every side effect so a rerun is harmless, route exhausted failures through a handler to compensation or a person, and wire request_drain() to SIGTERM with a grace period sized from your own retry math. If you are still choosing what state and traces those nodes should carry, start with LangGraph agents in production and our LangChain consulting notes.

Sources

• LangChain docs: Fault tolerance in LangGraph
• LangChain docs: Changelog, langgraph v1.2.0 (May 12, 2026)
• PyPI: langgraph release history
• LangGraph reference: TimeoutPolicy
• LangGraph reference: NodeTimeoutError
• LangChain docs: Checkpointers
• Python docs: Coroutines and tasks (asyncio)
• Kubernetes docs: Pod lifecycle
• LangSmith docs: Environment variables for self-hosted deployments
• LangSmith docs: Scalability and resilience