It matters because most production workloads use a fraction of what a large general LLM actually knows: a product that triages tickets or drafts reports in one domain can be served by a much smaller model that matches the teacher on that narrow slice.
How does model distillation actually work?
Collect inputs for the target task, run the large teacher on them to produce high-quality answers, then apply fine-tuning to train the student on those input-output pairs. The student never learns more than the teacher chose to demonstrate, which is why distillation transfers a specific skill well but rarely reproduces the teacher’s full breadth. More advanced variants train on the teacher’s probability distributions over tokens instead of finished text, passing along information about where the teacher was uncertain.
Distillation, quantization, or fine-tuning: which do you need?
They solve different problems and they stack. Fine-tuning adapts a model’s behavior with your data. Quantization shrinks an existing model’s memory footprint with no retraining. Distillation builds a new, smaller model whose payoff lands entirely in cheaper, faster inference. Start with quantization when a large model almost fits your budget; reach for distillation when even a compressed teacher misses your latency target, or when one narrow skill deserves its own cheap deployment.