Inference is the process of running a trained language model on new input to produce an output, the step that happens every time a prompt is sent and a response comes back, distinct from training, which is the earlier, far more expensive process of building the model itself.
Why does inference cost money on every single call?
Training happens once, expensively, and produces a fixed set of weights. Inference happens every time anyone uses the model, and each call recomputes predictions across billions of parameters for every token it reads and writes. That per-call cost is why a chatty AI agent looping through many steps burns real spend on cycles that move nothing forward, unlike training’s one-time bill.
What actually determines how fast or expensive inference is?
Mostly the size of the input and output measured in tokens: filling the context window with retrieved documents, tool definitions, and history all adds to the bill. The sampling settings matter too, though differently; temperature changes how the model samples its next token but not how much compute that token costs. Production systems that watch this closely, trimming context and caching repeated prefixes, are the ones that keep inference costs from quietly outrunning the value the system delivers, the same discipline behind halving Claude Code’s token usage.
Frequently asked questions
Is inference the same as training?
No. Training builds the model once, an expensive process that produces its weights. Inference is running that already-trained model on new input, which happens on every single call and is billed per token, not per model.
Does a higher temperature make inference slower?
Not meaningfully. Temperature changes how the next token is sampled, not the amount of computation each token costs. Inference speed and cost are driven mainly by how many tokens go in and come out.