What Is Quantization? How AI Models Get Small Enough to Run on Your Own Hardware
Model downloads say 8-bit, 4-bit, int8 or Q4. Here's what that means in plain English: what a bit-width is, how much smaller it makes a model, what it costs in precision, and how to decide.
SkillsUSA California State Champion, Information Technology Services (ITS)
4 min read
On this page
If you’ve looked at running an AI model on your own computer or server, you’ve probably seen names like “8B”, “int8”, “4-bit” or “Q4” next to the download links. They look like jargon, but they answer one very practical question: will this model fit on the hardware you have, and how much does it give up to get there?
That’s what quantization is about. Here’s a plain-English version, without the math you don’t need.
A model is a very large list of numbers
An AI language model is, underneath, a huge set of numbers called weights (or parameters). When a model is described as “8B”, that means it has about 8 billion of them. Running the model means loading all of those numbers into memory and doing arithmetic with them, over and over, for every word it writes.
So the size of a model in memory comes down to two things: how many weights it has, and how much space each weight takes.
What “bits per weight” means
Each weight is stored as a number with a fixed number of bits. The Hugging Face documentation puts it this way: weights are typically stored in full precision, 32-bit floating point, but half precision (16-bit) is “increasingly popular” given how large models have become. Quantization goes further and stores the weights “in a lower precision”, such as 8-bit or 4-bit integers, “while trying to preserve as much accuracy as possible.”
The size math is simple arithmetic: parameters × bits ÷ 8 = bytes. For a model with 8 billion parameters:
- at 16 bits per weight, the weights take about 16 GB;
- at 8 bits, about 8 GB;
- at 4 bits, about 4 GB.
That’s the weights alone. A running model needs extra memory on top of that, for the conversation it’s working on and for the software running it, so treat these as the floor, not the total.
- 16-bit16 GB
- 8-bit8 GB
- 4-bit4 GB
16-bit: up to 65,536 values per weight. At this size the curve looks unchanged.8-bit: up to 256 values per weight. At this size the curve looks unchanged.4-bit: up to 16 values per weight. The stored curve (solid) turns into steps.
Sizes are the weights only (parameters × bits ÷ 8) for the 8B model. A running model needs some memory on top of that.
What you give up: precision
Fewer bits means fewer possible values. A number with b bits can hold at most 2 to the power of b different values:
- 16 bits: up to 65,536 different values;
- 8 bits: 256 (as the Hugging Face guide notes, “only 256 values can be represented in int8”);
- 4 bits: just 16.
Quantization maps each original weight onto the nearest value it can still store, using a scale chosen for each group of weights. The rounding is real: at lower bit-widths the model’s answers can change, and how much depends on the model and on the method used. That’s why there are many quantization methods, and why some need a calibration step “for greater accuracy and extreme compression,” as the Transformers documentation puts it.
The animation above shows the idea on a simple curve. With 16 or 8 bits, the stored version sits right on top of the original. With 4 bits, it turns into steps.
Have a task your team repeats every week? Let’s map it and see whether automating it pays.
Plan my automationIt’s mostly about memory, not magic
Quantization’s main, reliable benefit is that the model takes less memory, so it fits on hardware it otherwise wouldn’t, or leaves room for other work. Speed and energy are less predictable. The Hugging Face guide notes that quantization saves energy “in theory,” and that in testing on some graphics cards, 4-bit formats made small models use more energy, not less, because unpacking the numbers is extra work. The honest summary: expect a smaller model, and measure the rest on your own hardware.
What it means for a business
Running a model on your own hardware can be worth it when the data shouldn’t leave the building, or when you want a predictable cost instead of paying per request. Quantization is often what makes that practical on a single workstation or a modest server.
If you’re considering it, a few rules of thumb hold up:
- Start from the task, not the model. Decide what the model has to do (summarize tickets, sort emails, answer questions from your documents) before picking a size.
- Use the highest precision that fits. If an 8-bit version fits comfortably, start there. Drop to 4-bit when you need the room.
- Test with your own examples. Run the same set of real requests through the versions you’re considering and compare the answers yourself. Published scores are measured on someone else’s tasks.
- Quantization doesn’t fix accuracy problems. A model that makes things up at 16 bits will still make things up at 4 bits. Checks and a human review stay in the workflow either way.
If a local model is on your list, I can help you figure out what it needs to run well and whether it’s worth doing compared with a hosted service. That’s part of the business automation work I do. For how a model connects safely to your business tools once it’s running, see Connecting an AI Assistant to Your Business Tools Safely.
Sources
- Quantization overview, Hugging Face Transformers documentation: storing weights in lower precision, and methods that need calibration
- Quantization concept guide, Hugging Face Optimum documentation: int8 and its 256 values, scale and zero-point, calibration, and energy in practice
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, Jacob et al., 2017: the integer quantization scheme the guide above is based on