How does quantization actually work?
Model weights are numbers, stored at training time in high-precision floating point. Quantization maps them into coarser formats such as eight-bit integers or four-bit floats, keeping a small scale factor per group of weights so values stay in range. Post-training quantization applies this to a finished model with no retraining, which makes it the first compression lever most teams pull before considering anything more expensive. Published evaluations consistently find four-bit weights hold close to full quality on common benchmarks, while quality degrades noticeably at two bits.
When is a quantized model the wrong answer?
Compression failures surface on edge cases first, so evaluate candidates on your own hard inputs rather than generic benchmarks. Quantization also cannot add a capability the base model lacks; it only preserves what exists, more cheaply. Treat it as one lever among several: prompt caching attacks a different part of the serving bill, and a distilled model can be quantized in turn when model distillation has already produced a smaller student. Measure whichever combination you ship through inference latency and quality checks in production, not lab scores alone.