What Are Evaluation Metrics for AI Models? Choosing the Right One

ProductionEvaluationPublished Updated By Simon Budziak

Evaluation metrics are the numbers used to judge how well an AI model or system performs. Examples are accuracy, precision, recall, F1, error rates or a judge's pass rate. Which metric you pick decides what counts as good, so it must match the cost of each kind of mistake.

Which evaluation metrics are used for which task?

Different tasks need different metrics, and a metric that fits one task misleads on another.

Task Common metrics Watch out for
Classification Accuracy, precision, recall, F1, AUC Accuracy hides rare-class failures
Regression Mean absolute error, mean squared error Large errors dominate squared error
Ranking and search Precision at k, recall at k, nDCG Depends on good relevance labels
Translation and summaries BLEU, ROUGE Reward word overlap, not meaning
Generated text and agents Task completion rate, LLM-judge pass rate The judge itself can be wrong

Pick the metric from the cost of each mistake, not from habit. A fraud model that misses fraud and one that blocks good customers fail differently, so recall and precision carry different prices.

Why can a high accuracy score mislead?

Suppose 1,000 cases hold 20 real positives. A model that always answers negative is 98% accurate and finds none of them. Its recall is 0. A model that finds 15 of the 20 but also flags 30 negatives has 15 true positives, 5 misses and 30 false alarms: precision is 15 / 45, about 0.33, recall is 0.75 and accuracy is 96.5%. The second model is more useful and scores lower on accuracy. Always compare against a trivial baseline, and score on a held-out set that you did not tune on.

How do you measure AI agents and language models?

Classic metrics need one right answer. For open-ended output, teams use task-level measures, such as the share of tasks completed correctly, and rubric scores from a language model acting as a grader. See AI agent evals and LLM as a judge. Count cost and latency next to quality, since a metric that improves while cost per task doubles is a trade, not a win.

How do we check that an evaluation metric deserves trust?

A metric from a language-model judge is only as good as its agreement with people. In our write-up on LLM-as-judge in LangSmith, the practical rule we reached is to calibrate before trusting any judge score. LangSmith’s documentation recommends starting with at least 20 labeled examples, balanced in both 0 and 1 labels, and defines the alignment score as the percentage of examples where the evaluator’s judgment matches the human expert’s (LangSmith docs, read 19 November 2026). The balance matters because a set of mostly passing examples cannot expose a judge that never fails anything. We treat an uncalibrated judge as a confident wrong signal, and we use its pass rate to spot a change from last week, not as a verdict.

What makes a metric hard to game?

A metric becomes a target the moment a team is paid on it. Keep a hidden test set, report several metrics together and sample real outputs for a person to read. Tie the number back to the business result, such as errors caught per thousand invoices, so improvements on paper show up in operations.

Written with AI assistance and reviewed by Simon Budziak. The production notes come from systems Soba Labs builds and runs.

Frequently asked questions

Is 80% accuracy good in machine learning?

It depends on the base rate. If 95% of cases are negative, a model that always says negative scores 95%. Compare against that baseline and look at precision and recall for the rare class.

Which evaluation metric should I use?

Start from the cost of mistakes. If missing a true case is costly, track recall. If false alarms are costly, track precision. Use F1 to balance both, and a held-out set you did not tune on.

Summarize this page with

Train your team to build this