Token attribution in LangChain: tokens are not costs

10 min readSimon BudziakBy Simon Budziak

On this page7 sections
Title card reading Token cost attribution in LangChain. A stylized navy invoice panel with a thin gold border, its single bottom summary line splitting into three narrower gold branch lines that run out to three small terminal windows, each carrying a gold bar of a different weight and brightness.

The month-end invoice on a mixed-provider pipeline we ran came in well over forecast, and nothing on it said why. Days went into correlating traces back to session IDs before the culprit turned up, one customer’s batch import job misconfigured to run hourly. It took days because a provider invoice is a model-level aggregate: it reports what a model consumed and nothing about which user, workflow or agent node consumed it. Two callbacks in langchain-core close that gap, UsageMetadataCallbackHandler for session and workflow rollups and get_usage_metadata_callback() for per-request scope, and both have been there since langchain-core 0.3.49. This post covers which scope belongs where, the three current defaults that quietly produce wrong numbers, and why a correct token count is still not a cost.

What changed since this was first written. The section this post used to be organised around, that OpenAI requires stream_usage=True while Anthropic includes usage by default, no longer describes reality. Both providers now default it on. The failure mode moved from a flag you forgot to set into a default your production wiring can silently switch off, and the whole comparison below is rebuilt against the shipped source rather than carried over.

Why the invoice cannot answer the question

Provider billing is organised by model, because that is the only thing the provider can see. One user request in a routing pipeline might hit a small model for classification, a fast model for extraction and a larger model for synthesis. The cost of that user’s request is then fragmented across three aggregates on two invoices, and no field in either one says which request it belonged to.

Homegrown accounting does not survive contact with production either. Counting tokens before .invoke() gives you an estimate rather than a measurement, and it misses tool call outputs, retry tokens and the usage chunk that arrives after a stream has finished. Attribution has to happen at the call graph, not at the provider account.

The two callbacks operate exactly there. They read AIMessage.usage_metadata off each completed generation and key it by the model name reported in the response, so a mixed-provider pipeline lands in a single dictionary.

Two callbacks, two scopes

The distinction is lifetime. UsageMetadataCallbackHandler is an object you own, and it accumulates for as long as you keep it alive. get_usage_metadata_callback() is a context manager, and its scope is the with block.

The handler, for session and workflow rollups

Instantiate it once and pass it in config={"callbacks": [...]} on every call you want counted.

from langchain.chat_models import init_chat_model
from langchain_core.callbacks import UsageMetadataCallbackHandler

callback = UsageMetadataCallbackHandler()
classifier = init_chat_model(CLASSIFIER_MODEL)
extractor = init_chat_model(EXTRACTOR_MODEL)

classifier.invoke("Classify this document",
                  config={"callbacks": [callback]})
extractor.invoke("Extract obligations",
                 config={"callbacks": [callback]})

print(callback.usage_metadata)
# keyed by model name, one entry per model that answered

Two properties of the handler are worth knowing before you build on it. The dictionary keys are model name strings taken from each response, not provider names, so a pipeline that routes across model versions produces one entry per version. And there is no reset() method on the class: the only way to start a fresh count is to construct a new handler. That is what makes it natural for billing rollups, one handler per session, usage_metadata persisted when the session ends.

The context manager, for per-request guards

from langchain_core.callbacks import get_usage_metadata_callback

BUDGET_TOKENS = 5_000

with get_usage_metadata_callback() as cb:
    result = pipeline.invoke({"input": user_prompt})
    total = sum(m["total_tokens"]
                for m in cb.usage_metadata.values())
    if total > BUDGET_TOKENS:
        logger.warning("over budget", extra={"tokens": total})

The context manager registers itself through a context variable, so calls inside the block are counted without any config plumbing. That convenience is also the thing that broke: until recently the variable was reset on the normal path but not when the block raised, so an exception inside the with left the callback attached and every later model call in that context kept accumulating into it. It was reported as langchain#38989 in July 2026 and fixed by moving the reset into a finally, which first shipped in langchain-core 1.5.5 on 14 August 2026. If you rely on the block’s scope for a per-request guard, that is your floor version.

The streaming flag became a default, and your wiring can turn it off

Streaming used to be the section where this post told you to set stream_usage=True on ChatOpenAI. That advice was already obsolete when it was written. LangChain enabled the flag by default in PR #33205, merged on 6 October 2025, roughly six months before this post first ran.

Both providers now default it on, and they do it differently, which is the part that matters in production.

ChatAnthropic sets stream_usage: bool = True as a plain field default. It is unconditional. You get usage on streamed responses unless you deliberately turn it off.

ChatOpenAI computes the default at initialisation, and the condition is your wiring. The shipped source enables it only when the model is constructed with the default endpoint and the default client. Set base_url, set the OPENAI_BASE_URL environment variable, pass a proxy, or hand it a custom http_client, and the default is left off. The class docstring is explicit about why: the flag “is enabled unless openai_api_base is set or the model is initialized with a custom client, as many chat completions APIs do not support streaming token usage” (langchain-openai, chat_models/base.py). A base URL set by the LangSmith gateway is carved out and still gets the default, because that proxy does support it.

That is a better rule than the old one, and a more dangerous one. The teams most likely to lose streaming usage data are the teams furthest along: anyone routing OpenAI traffic through an internal gateway, a compatibility proxy or a custom httpx client for retries and timeouts has switched the default off without touching it.

A navy decision card titled Will streaming usage be counted. Four stacked rows. Row one, ChatAnthropic, always counted, the field default is True. Row two, ChatOpenAI on the default endpoint and default client, counted. Row three, ChatOpenAI with base_url or OPENAI_BASE_URL set, not counted unless you set the flag. Row four, ChatOpenAI with a custom client, proxy or http_client, not counted unless you set the flag. A note underneath reads that the LangSmith gateway base URL is the one carved-out exception.
The OpenAI default is computed from how the model was constructed, so a gateway or a custom client silently opts you out.

The remedy is a single line and there is no downside to it. Set stream_usage=True explicitly on any model you build with a non-default endpoint or client, at initialisation rather than per call, and stop depending on a default whose condition lives in your infrastructure config.

from langchain_openai import ChatOpenAI

# Behind a gateway, so the default would be off.
llm = ChatOpenAI(
    model=SYNTHESIS_MODEL,
    base_url=INTERNAL_GATEWAY_URL,
    stream_usage=True,
)

A correct token count is still not a bill

This is the part the original version of this post got backwards, and it is worth being precise about because the error runs the wrong way round.

In langchain-core, output_tokens is documented as the “Count of output (or completion) tokens. Sum of all output token types”, while output_token_details is documented as a breakdown that need not sum to the full output token count (langchain-core, messages/ai.py). Reasoning tokens are inside output_tokens, not missing from it. Summing output_tokens does not undercount.

The real problem is that summing tokens at all prices nothing, because the token types inside those details are billed at different rates.

Anthropic publishes the multipliers plainly: “5-minute cache write tokens are 1.25 times the base input tokens price”, “1-hour cache write tokens are 2 times the base input tokens price”, and “Cache read tokens are 0.1 times the base input tokens price” (Anthropic, “Prompt caching”). That is a twentyfold spread between the cheapest and the most expensive input token, on identical counts. OpenAI does not publish one number at all: cached input is billed at a cached-input rate “when the model offers one”, and the docs say plainly that “Rates and discounts vary by model” (OpenAI, “Prompt caching”), so the pricing table for the model you actually deploy is the only authority. We covered where those savings come from in prompt caching across Claude, GPT and Gemini.

LangSmith models this correctly and the mechanism is a good specification to copy. Its documentation states that “The cost for a run is computed greedily from most-to-least specific token type”, pricing cache_read at its own rate and applying the default input price only to the remainder (LangSmith, “Cost tracking”). One caveat before you lean on it as the ledger: LangSmith does not apply model pricing changes retroactively, and backfilling them is not supported.

The takeaway to build on. total_tokens is a usage metric, not a cost. Persist the full usage_metadata including input_token_details and output_token_details, then price each token type separately. A billing layer that stores one integer per request has thrown away the fields that determine the bill, and it cannot be reconstructed later.

Attributing to a user, a workflow and a node

The callbacks give you counts. Attribution is joining those counts to application identity, and that join is yours to write.

from langchain_core.callbacks import UsageMetadataCallbackHandler

def process_request(user_id: str, session_id: str, prompt: str):
    # Fresh per request. This is the critical line.
    callback = UsageMetadataCallbackHandler()

    config = {
        "callbacks": [callback],
        "metadata": {
            "user_id": user_id,
            "session_id": session_id,
        },
    }
    result = pipeline.invoke({"input": prompt}, config=config)

    persist_usage_record(
        user_id=user_id,
        session_id=session_id,
        model_breakdown=callback.usage_metadata,
    )
    return result

One handler instance per request, never one per application. The handler merges every generation it sees into one dictionary keyed by model, and it is thread safe, which means concurrent requests sharing a handler will accumulate cleanly into the same totals and destroy the attribution you built it for. Correct code and a meaningless number.

The metadata dictionary is doing separate work. It surfaces user_id and session_id as run metadata in LangSmith, which is what makes ad hoc cost queries grouped by user possible without a deploy. LangSmith’s own documentation adds a trap here that is easy to hit: when aggregating across threads, child runs that do not carry the thread metadata are excluded from thread-level totals. Tag the whole trace, not just the entry point.

Cost per agent node

Session totals tell you what a request cost. They do not tell you which node in the graph spent it, and in a LangGraph application that is the number that changes what you do next. The context manager fits because its scope is a block, so it can be a node.

import operator
from typing import Annotated
from typing_extensions import TypedDict
from langchain_core.callbacks import get_usage_metadata_callback
from langchain_core.runnables import RunnableConfig

class AgentState(TypedDict):
    messages: list
    node_costs: Annotated[dict, operator.or_]

def research_node(state: AgentState, config: RunnableConfig):
    with get_usage_metadata_callback() as cb:
        result = research_llm.invoke(state["messages"],
                                     config=config)
    return {
        "messages": [result],
        "node_costs": {"research": cb.usage_metadata},
    }

Annotated[dict, operator.or_] is a state reducer, LangGraph’s mechanism for merging updates that several nodes write to the same key. Each node returns only its own entry and the graph merges them, so per-node usage accumulates without a side channel and without nodes knowing about each other. Give every node the same three lines and state["node_costs"] is a complete cost map by the time the graph finishes.

That map is what makes optimisation targeted rather than speculative. A node carrying a large static system prompt and retrieved context on every turn is where prompt caching pays, and a node re-reading a full conversation each iteration is where trimming pays, but the two look identical from a session total. You cannot reduce a cost you cannot locate.

What LangChain owns and what you own

The split is clean, and mistaking it is how teams end up building an accounting layer they did not need.

langchain-core owns measurement: the counts, the per-model keying, the token type breakdown. LangSmith owns investigation: traces, run metadata, cost derived from a pricing table you control. Your database owns the ledger: the join to user and tenant, the pricing you invoice at, the retention.

Three things carry forward.

  • Match scope to purpose. get_usage_metadata_callback() for per-request guards and per-node measurement, on langchain-core 1.5.5 or later if you need the block scope to hold when the block raises. UsageMetadataCallbackHandler for session and workflow rollups.
  • Stop trusting the streaming default. It is on for Anthropic unconditionally and for OpenAI only on the default endpoint with the default client. Set stream_usage=True explicitly the moment a gateway, proxy or custom client enters the picture.
  • Persist the details, not the total. Cache reads, cache writes and reasoning tokens are priced differently enough that a single total_tokens integer cannot be turned back into money.

Dealing with unexplained token costs in a multi-model pipeline? Book a call.

Sources

Last updated:

How we build on LangChain in production