← Back to Blog
Nov 7, 2026 · 14 min read

Quantize Your Own GGUF: Picking Q4 vs Q5 vs Q6 With Numbers, Not Vibes

llama.cppLocal LLMQuantizationGGUFTutorial

You have a 7B fine-tune with no public GGUF file. You have one GPU with 12 GB. Q4_K_M fits comfortably, Q6_K fits only at short context, and every guide on the internet says the same thing: use Q4_K_M.

That advice is fine for a chat model. It is wrong for a RAG pipeline at 32k context, where the KV cache can cost more memory than the weights themselves. It is also a guess about your domain, because whoever made that quant calibrated it on Wikitext, not on your data.

This walkthrough goes from a Hugging Face checkpoint to several quantized files, then measures which ones actually cost you quality. Every command is from llama.cpp's current docs, and the reference numbers come from the project's own perplexity scoreboard.

What you need

  • cmake 3.14 or newer, a C++ compiler, git
  • Python 3.10+ in a venv, for the conversion script
  • The model in safetensors, either a local directory or an HF repo id
  • Disk space around 2.5x the fp16 size, since you keep the source GGUF plus every quant you produce
  • Between 1 MB and 10 MB of plain text for calibration

Step 1: build llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# pick one backend
cmake -B build -DGGML_CUDA=ON       # NVIDIA
# cmake -B build -DGGML_METAL=ON    # Apple Silicon, on by default on macOS
# cmake -B build -DGGML_VULKAN=ON   # AMD or Intel on Linux

cmake --build build --config Release -j"$(nproc)"

The binaries land in build/bin. The ones used here are llama-quantize, llama-imatrix, llama-perplexity, llama-bench, and llama-server. Recent builds also ship a single llama router binary where llama serve and llama cli call the same programs, so both spellings work.

Step 2: convert to GGUF at bf16

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python convert_hf_to_gguf.py --outtype bf16 \
  --outfile ./models/mistral-7b-bf16.gguf \
  ./hf/mistral-7b

If the weights live on the Hub, drop the local path and pass --remote org/model-name instead. Use bf16 or f16 as the source, never an already quantized file. Requantizing a Q4 gives you a worse Q4, because the damage from the first pass is already baked in.

The conversion script's pinned transformers is version 4, and newer architectures want version 5. If conversion dies on an unknown architecture, pip install -U transformers and try again.

Step 3: collect calibration text from your own corpus

This is the step people skip, and it is the only one that makes your quant different from a public download.

mkdir -p calib
pdftotext -layout docs/*.pdf calib/raw.txt
# or, for a log-heavy agent:
# cat /var/log/myapp/*.log > calib/raw.txt

wc -c calib/raw.txt

One to ten megabytes is plenty. The importance matrix records how strongly each weight contributes to the output activations for the text you feed it, so calibrating on documentation, tickets, or SQL beats calibrating on encyclopedia articles when that is what you serve.

Save a chunk of the same text as a held-out file for step 6. Do not measure perplexity on the exact text you used for calibration.

One honest note before you spend a day curating this: llama.cpp's own scoreboard tested matrices built from 1k, 10k, 100k, 1m and 10m Wikitext tokens and found no consistent improvement as the corpus grew. In-domain matters. Size past a few megabytes mostly does not.

Step 4: build the importance matrix

./build/bin/llama-imatrix \
  -m ./models/mistral-7b-bf16.gguf \
  -f calib/raw.txt \
  -o imatrix.gguf \
  -ngl 99

-ngl 99 pushes every layer onto the GPU, which is the difference between minutes and an hour. Output defaults to the GGUF format. Add --no-ppl if you do not care about the perplexity log it prints along the way.

Step 5: quantize several levels

Do not produce one file and hope. Produce three, then measure.

for q in Q4_K_M Q5_K_M Q6_K; do
  ./build/bin/llama-quantize --imatrix imatrix.gguf \
    ./models/mistral-7b-bf16.gguf \
    ./models/mistral-7b-$q.gguf \
    $q
done

If you want a mixed recipe, the per-tensor flags are where the extra quality lives. Output embeddings and token embeddings are cheap to keep at higher precision because they are a small share of total weights:

./build/bin/llama-quantize --imatrix imatrix.gguf \
  --output-tensor-type q6_k \
  --token-embedding-type q4_k \
  ./models/mistral-7b-bf16.gguf \
  ./models/mistral-7b-mix.gguf \
  Q4_K_M

Two flags worth knowing: --pure disables the k-quant mixing so every tensor gets the same type, and --allow-requantize lets you quantize an already quantized file, which the docs explicitly warn against for quality reasons.

Step 6: measure the damage

Perplexity first, on the held-out file:

./build/bin/llama-perplexity -m ./models/mistral-7b-bf16.gguf -f calib/test.txt
./build/bin/llama-perplexity -m ./models/mistral-7b-Q4_K_M.gguf -f calib/test.txt

Perplexity alone is blunt. KL divergence tells you how far the quantized model's token distribution drifted from the fp16 one, and that is the number worth comparing. Record the fp16 logits once, then reuse the file:

./build/bin/llama-perplexity -m ./models/mistral-7b-bf16.gguf \
  -f calib/test.txt --kl-divergence-base base.kld

./build/bin/llama-perplexity -m ./models/mistral-7b-Q4_K_M.gguf \
  -f calib/test.txt --kl-divergence-base base.kld --kl-divergence

The logit file is large. The docs put it at 11 GiB for a LLaMA 2 model and 37 GiB for LLaMA 3 on the Wikitext-2 test set, so check free disk before you run it.

For scale, here is llama.cpp's published scoreboard for Llama 3 8B on a single RTX 4090 with the CUDA backend. Size in GiB, perplexity, and mean KL divergence against fp16:

  • f16, 14.97 GiB, PPL 6.233 (reference)
  • q8_0, 7.96 GiB, PPL 6.234, KLD 0.001355
  • q6_K, 6.14 GiB, PPL 6.253, KLD 0.005452
  • q5_K_M, 5.33 GiB, PPL 6.289, KLD 0.010762
  • q4_K_M with imatrix, 4.58 GiB, PPL 6.383, KLD 0.028152
  • q4_K_M without imatrix, 4.58 GiB, PPL 6.407, KLD 0.031273
  • q3_K_M with imatrix, 3.74 GiB, PPL 6.734, KLD 0.084358
  • q3_K_M without imatrix, 3.74 GiB, PPL 6.889, KLD 0.101913
  • q2_K with imatrix, 2.96 GiB, PPL 8.648, KLD 0.332223
  • q2_K without imatrix, 2.96 GiB, PPL 9.752, KLD 0.445132
  • iq1_S with imatrix, 1.88 GiB, PPL 58.1, KLD 2.211

Three things fall out of that table.

From q6_K down to q4_K_M, KL divergence grows about five times while perplexity moves from 6.253 to 6.383. That is a real but small shift, and it is why Q4_K_M became the default.

Below 4 bits per weight the curve turns. q3_K_M doubles the divergence again, and q2_K stops being a language model you would ship, at a perplexity of 8.6 against 6.2.

The imatrix earns its keep exactly where bits are scarce: KLD drops from 0.0313 to 0.0282 at q4_K_M, from 0.1019 to 0.0844 at q3_K_M, and from 0.445 to 0.332 at q2_K. At q5 and q6 the gain is in the noise.

Caveat on reading any of this: perplexity is not comparable across models with different tokenizers, and a fine-tune usually scores higher perplexity than its base model while producing better answers. Use these numbers to compare quants of the same model, nothing else.

Step 7: size it against your actual RAM

Weights are only part of the bill. The KV cache scales with context length and, for a model using grouped query attention, it is cheaper than you might expect:

bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes per element

For Llama 3 8B at fp16 KV: 2 x 32 x 8 x 128 x 2 = 131,072 bytes, which is 128 KiB per token. At 8k context that is 1 GiB. At 32k it is 4 GiB, nearly as much as the q4 weights.

So budget it like this, with a 1 GiB allowance for the compute buffer and fragmentation:

  • 16 GB machine, 8k context: Q5_K_M at 5.33 GiB plus 1 GiB KV plus 1 GiB overhead lands near 7.3 GiB. Everything fits, and dropping to Q3 buys you nothing you need.
  • 8 GB machine, 4k context: Q4_K_M at 4.58 GiB plus 0.5 GiB KV plus 0.75 GiB overhead is about 5.8 GiB. Tight but workable.
  • 12 GB GPU at 32k context: Q4_K_M plus 4 GiB of KV cache is already 8.6 GiB before the buffer. Q3_K_M with an imatrix is the honest choice here, and now you know what it costs.

Step 8: serve it and check throughput

./build/bin/llama-bench -m ./models/mistral-7b-Q4_K_M.gguf -p 512 -n 128

./build/bin/llama serve -m ./models/mistral-7b-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 8192 -ngl 99

llama serve exposes an OpenAI compatible endpoint at /v1/chat/completions plus a browser UI, so anything that speaks the OpenAI API can point at it without a code change.

Throughput is the other half of the tradeoff. On the same Llama 3 8B scoreboard, text generation at 128 tokens ran at 76.7 t/s for Q4_K_S, 50.9 t/s for Q8_0, and 29.2 t/s for f16 on that 4090. Doubling precision cut generation speed by a third against Q4. Whether 0.0014 KL divergence is worth that depends on what you are doing, and now it is a decision rather than a default.

When this is the wrong tool

If you just want to run a model someone already quantized, download it. Unsloth and Bartowski publish quants for most popular models, and a Q4_K_M from either will beat your first attempt.

If you have no GPU for the imatrix step, the GGUF-my-repo Space on Hugging Face builds quants for free and syncs from llama.cpp main every six hours.

If you want model lifecycle management, Ollama gives you pull, run, and serve in one command, at the cost of not controlling the tensor mix.

Build your own quant when the model is a fine-tune with no public GGUF, when you need a specific bits-per-weight to fit a specific card, or when the calibration corpus is the whole point.

What to do next

Pick your smallest acceptable KLD, not your smallest file. Run the pipeline once, write the three numbers down (GiB on disk, mean KLD, tokens per second), and stop guessing at quant names.

Then keep the imatrix file. It is small, it is reusable across every future quant of the same base model, and rebuilding it is the slow part.

References

Need Help Implementing This?

I help teams design and build scalable cloud infrastructure, DevOps pipelines, and production-grade systems.

Book a Free Consultation