LLM tracing

ProductionObservabilityPublished By Simon Budziak

LLM tracing records every step an AI application takes to produce one output, such as a retrieval query, a tool call, and a model call, as structured spans grouped into a single trace. It matters because most LLM failures are not crashes: the request succeeds while the answer is quietly wrong.

A trace is the full record of one request. Each operation inside it becomes a span carrying its own inputs, outputs, timing, and token usage, nested under the step that triggered it. When an answer is wrong, the span tree shows which step introduced the error, whether retrieval pulled weak sources or a tool returned stale data.

How is LLM tracing different from logging?

Logs record flat events and confirm that something failed. Tracing preserves the causal path behind an output, which is what makes LLM observability possible: dashboards, evaluations, and alerts all read from the same structured spans. Dedicated platforms such as Langfuse, LangSmith, and Arize Phoenix exist to store and query these traces at scale.

Why do multi-step agents depend on it?

An AI agent can make many model and tool calls per request, and reasoning backward from one bad final answer gives no visibility into which call broke. With tracing, engineers walk the span tree in execution order, compare runs before and after a fix, and feed failed traces into AI agent evals as regression test cases. Reading that sequence is how we found the failures in Open Loyalty’s SDR, where odd answers turned out to be retrieval and CRM problems.

Frequently asked questions

Is LLM tracing the same thing as LLM observability?

No. Tracing is the capture layer that records spans. Observability is the broader practice built on top of those traces, including monitoring, evaluation, and alerting on production behavior.

What should a span capture in an LLM application?

At minimum the inputs and outputs of one step, its duration, errors, and token usage, plus identifiers like the user or session so traces can be filtered later.

Summarize this page with

Train your team to build this