What does overfitting actually look like?
A gap. The model scores near perfectly on the data it trained on and visibly worse on data it has never seen. In statistics terms it has low bias and high variance: it bends to fit every point, including the points that were noise. The training score is a claim about memory; the test score is a claim about learning, and only the second predicts production behavior. The opposite failure, underfitting, is a model too simple to capture the pattern at all, bad on both sets.
What causes a model to overfit?
Three usual suspects. A model with more capacity than the data justifies, which memorizes because it can. Training data that is too small or too narrow, so noise looks like signal. And training that runs too long, polishing the fit past the point of generalization. In machine learning practice, all three reduce to the same mistake: optimizing the score you can see instead of the behavior you need.
How do you prevent overfitting?
Hold data back and penalize complexity. Model validation splits data so the test set never touches training, and cross-validation rotates the split so the check does not depend on luck. Regularization adds a cost for complexity, so the model prefers simpler explanations; early stopping halts training when held-out performance stops improving; and more diverse data makes the noise average out. No fix removes the need for the honest test set, it is the instrument everything else is tuned against.
Does overfitting still matter in the LLM era?
Yes, it just changed clothes. Large language models can overfit their benchmarks: when test questions leak into training corpora, scores inflate without capability, the problem named benchmark contamination. Teams fine-tuning on small internal datasets meet classic overfitting directly, and even prompt tweaks can overfit a demo set and regress in production, which is why the eval discipline exists. Any number measured on data the system has seen is a number to distrust, the same rule since the first regression, and watch for drift after deployment, because a fit that was honest can still go stale.