Quantization
Reducing the numeric precision of a model's weights (for example from 16-bit to 4-bit) so it needs less memory and runs faster, with a small loss in accuracy.
Definition
A model's weights are stored as numbers, typically 16-bit floating point. Quantization maps them to a smaller representation, such as 8-bit or 4-bit integers, with scaling factors that preserve most of the information. Memory use falls roughly in proportion, so a model that needed two high-end GPUs can fit on one, and a mid-sized model can run on a laptop. Because less data moves through memory, inference often gets faster as well.
The cost is precision. Rounding billions of weights introduces small errors that accumulate into slightly worse output, more noticeable at lower bit widths and on smaller models. Post-training quantization is applied to a finished model; quantization-aware training builds tolerance in during fine-tuning. Activations and the key–value cache can be quantized too, with similar trade-offs.
Example: a company wants to run an open-weight model on its own servers for data-residency reasons. The 16-bit version needs more GPU memory than they have; the 4-bit version fits, and on their eval set accuracy drops by about one point, which they accept.
Why it matters for your business
- It is what makes self-hosting and on-device AI affordable, which matters for privacy and data residency.
- Hardware cost for the same model can fall substantially, and latency with it.
- Quality loss is real and task-dependent, so evals must be rerun on the quantized model.
- It is only relevant for models you host; hosted APIs handle it on your behalf.
How Teksmobile uses it
We use quantized open-weight models when a client needs on-premises or private-cloud inference, and in AI for mobile apps where models run on the device. The choice between full precision and quantized serving is made with measured evals, as described in our guide to small language models.
Related terms
Part of the Teksmobile AI Glossary. For longer reads, see our guides.
Frequently asked questions
How much accuracy is lost?
8-bit quantization is usually close to lossless on common benchmarks. 4-bit introduces a measurable but often acceptable drop, larger on small models and on demanding reasoning tasks. Always measure on your own evals.
Does quantization apply to hosted models like Claude or GPT?
Not in a way you control. Providers manage their own serving. Quantization is a decision you make when self-hosting open-weight models.
What formats will we encounter?
Common ones include GGUF for CPU and mixed inference, and GPTQ or AWQ for GPU serving. Tooling and runtimes change quickly, so check current support for your hardware.