Understanding quantisation

What you’ll learn

What quantisation does, what the format names like Q4_K_M mean, what quality you trade away, and how to pick a quantisation that fits your hardware.

The concept

A model’s weights are numbers. In full precision each is stored in 16 bits, which is accurate but memory-hungry: a 70B model in 16-bit precision needs roughly 141GB just to hold the weights, well beyond any single consumer card.

Quantisation stores those numbers in fewer bits, 8, 5, 4, or even lower, which shrinks the model dramatically. A 4-bit quantisation of that same 70B model needs roughly 43GB rather than 141GB. The catch is that lower precision loses some information, and at some point that shows up as worse output. The art is picking the point where the model still fits your hardware and the quality loss is small enough not to matter for your use.

These figures are approximate. Actual memory use depends on context length, batch size, and the KV cache (the running memory of the conversation), all of which grow as your conversations get longer. Treat a quantisation’s stated size as a floor, not a precise budget, and leave headroom.

Reading the format names

The common naming, from the GGUF format that llama.cpp introduced, looks like Q4_K_M. Roughly:

• The number after Q is the bits per weight. Q4 is 4-bit, Q8 is 8-bit. Fewer bits means a smaller file and more quality loss.
• The K refers to the “k-quant” method, which allocates precision more cleverly than older approaches.
• The trailing letter (S, M, L) is the size within that family. Q4_K_M is the medium 4-bit k-quant, and it is the most common default because it balances size and quality well.

Choosing one

A reasonable approach:

1. Work out how much memory you have to spend, and leave a couple of gigabytes of headroom for context and the operating system.
2. Pick the highest-quality quantisation that fits with that headroom. Within a model, a larger file means less aggressive compression and better quality.
3. If nothing good fits, choose a smaller model at a higher quantisation rather than a large model crushed down to a very low one. A well-quantised smaller model often beats a badly quantised larger one.

The hardware matrix does step two for you: give it your memory and it shows the best quantisation of each model that fits.

What can go wrong

• Going too low to fit a bigger model. Very low quantisations (below 4-bit) can degrade a model noticeably. Fitting a larger model this way is often a false economy.
• Forgetting the context budget. A model that fits at short context can run out of memory once the conversation grows, because the KV cache grows with it.
• Comparing sizes across models. A 4-bit 70B and a 4-bit 7B are not the same kind of thing. Quantisation level only compares meaningfully within the same model.

Next steps

With quantisation understood, the model catalogue lists the practical quantisations and approximate memory for each model, so you can match one to your hardware.