Which evaluation metrics are used for which task?
Different tasks need different metrics, and a metric that fits one task misleads on another.
| Task | Common metrics | Watch out for |
|---|---|---|
| Classification | Accuracy, precision, recall, F1, AUC | Accuracy hides rare-class failures |
| Regression | Mean absolute error, mean squared error | Large errors dominate squared error |
| Ranking and search | Precision at k, recall at k, nDCG | Depends on good relevance labels |
| Translation and summaries | BLEU, ROUGE | Reward word overlap, not meaning |
| Generated text and agents | Task completion rate, LLM-judge pass rate | The judge itself can be wrong |
Pick the metric from the cost of each mistake, not from habit. A fraud model that misses fraud and one that blocks good customers fail differently, so recall and precision carry different prices.
Why can a high accuracy score mislead?
Suppose 1,000 cases hold 20 real positives. A model that always answers negative is 98% accurate and finds none of them. Its recall is 0. A model that finds 15 of the 20 but also flags 30 negatives has 15 true positives, 5 misses and 30 false alarms: precision is 15 / 45, about 0.33, recall is 0.75 and accuracy is 96.5%. The second model is more useful and scores lower on accuracy. Always compare against a trivial baseline, and score on a held-out set that you did not tune on.
How do you measure AI agents and language models?
Classic metrics need one right answer. For open-ended output, teams use task-level measures, such as the share of tasks completed correctly, and rubric scores from a language model acting as a grader. See AI agent evals and LLM as a judge. Count cost and latency next to quality, since a metric that improves while cost per task doubles is a trade, not a win.
How do we check that an evaluation metric deserves trust?
A metric from a language-model judge is only as good as its agreement with people. In our write-up on LLM-as-judge in LangSmith, the practical rule we reached is to calibrate before trusting any judge score. LangSmith’s documentation recommends starting with at least 20 labeled examples, balanced in both 0 and 1 labels, and defines the alignment score as the percentage of examples where the evaluator’s judgment matches the human expert’s (LangSmith docs, read 19 November 2026). The balance matters because a set of mostly passing examples cannot expose a judge that never fails anything. We treat an uncalibrated judge as a confident wrong signal, and we use its pass rate to spot a change from last week, not as a verdict.
What makes a metric hard to game?
A metric becomes a target the moment a team is paid on it. Keep a hidden test set, report several metrics together and sample real outputs for a person to read. Tie the number back to the business result, such as errors caught per thousand invoices, so improvements on paper show up in operations.