What does accuracy measure, exactly?
One ratio: correct predictions over total predictions, from 0 to 1. On a classifier it counts true positives and true negatives together, which means it weighs every kind of correct the same and every kind of wrong the same. That symmetry is the metric’s convenience and its flaw, because real decisions rarely price a missed fraud the same as a false alarm. The number is only as honest as the data it is computed on, which is what model validation exists to guarantee: measure on examples the model never trained on, or the score measures memory.
When does accuracy lie?
Whenever the classes are imbalanced or the error costs differ. The classic failure: 1 percent of transactions are fraudulent, a model that flags nothing is 99 percent accurate and worthless. A model can be accurate on average and useless on the case the business exists to catch. That is why evaluations read precision (of the flagged, how many were real) and recall (of the real, how many were flagged) next to accuracy, and why a probability-producing model also needs calibration, so its 80 percent confident is right about 80 percent of the time.
What should a buyer ask when a vendor quotes accuracy?
Three questions. Accurate on what data: a benchmark, the vendor’s demo set, or a sample of your own cases? Accurate against what baseline: 90 percent impresses less when always guessing the majority scores 88. And accurate at what cost per error type: the quote is incomplete until you know what the misses cost, since a point of recall on fraud and a point of accuracy on routing are different money.
Does accuracy apply to LLMs and agents?
Only after you define correct. A classifier’s answer is right or wrong; a generated paragraph needs a rubric, a judge, or a human grade before anything can be counted, which is the job of AI agent evals. For agents doing multi-step work, the more honest unit is task completion rate: did the whole job finish correctly, not did one prediction match. The discipline transfers even where the formula does not: held-out cases, a defined notion of correct, and suspicion of any score that looks too good.