Best Quantization Methods for llama.cpp

Key Takeaways

• On a 7B model, switching from F16 to Q4_K_M increases perplexity by 0.0535, a small difference that can still influence how natural or coherent the output feels. The Q8_0 format, by contrast, adds just 0.0004 perplexity. This tiny gap explains why most developers now prefer Q4_K_M, it offers a good balance of quality and speed.
• Q8_0 tokens are generated 29% more slowly than Q4_K_M tokens. This slowdown stems from how generation relies on reading data from memory. Larger files, like Q8_0, require more bytes to be fetched per token, which bottlenecks the process.
• IQ4_XS, a type of importance matrix quantization, saves roughly 400 MB compared to Q4_K_M at nearly the same perplexity. However, decoding on older Pascal GPUs like the GTX 1080 Ti is 25% slower because these cards lack the SIMD instructions I-quants depend on for fast unpacking.
• For 70B models, size becomes critical. Q4_K_M at 42.5 GB comfortably fits across two 24 GB consumer cards, with room for caching. Q5_K_M at 49.9 GB exceeds this limit, making it less practical for such setups. Larger formats like Q8_0 demand even more memory, often requiring four cards or high-end workstations.
• To optimize long-context inference, it’s best to quantize the key-value cache separately using --cache-type-k q8_0. This approach reduces VRAM use without sacrificing the weight precision of the main model.
• On May 15, 2026, six vulnerabilities in llama.cpp’s GGUF parser were disclosed. These flaws occur before inference begins and can be exploited if model files are not verified. The most serious, CVE-2026-27940, can cause arbitrary file reads on 32-bit systems.

The Perplexity Numbers

Perplexity gauges how surprising a model’s responses are. Lower perplexity means the model is more confident and accurate. The differences listed here come from the same llama.cpp tracking data, measured against F16 on a 7B model, supported by benchmarks on Llama 3.1 8B and Qwen 3 14B from bric.pe.kr.

Data sources include llama.cpp discussion #2094 for perplexity differences and the llama.cpp quantize README for file sizes, along with bartowski’s 70B GGUF repo.

The entire Q4 to Q8 range shows only a 0.0531 perplexity point difference. In real-world conversation, this tiny gap is often indistinguishable. It’s roughly a 1.4-point accuracy difference when moving four bits down in precision.

Below Q4, output quality drops noticeably. Users report more repeated phrases and factual errors at these levels. Q3 is mainly a fallback setting when models cannot fit otherwise, not a tuning choice.

Legacy, K-Quants, and I-Quants

The GGUF format offers three quantization families, each with distinct characteristics. The older formats (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0) apply one scale per 32 weights, making decoding fast but less accurate for skewed weight distributions. Q4_0 has been replaced by K-quants and is no longer recommended.

K-quants (Q2_K through Q6_K) introduce a two-tier structure: 32-weight blocks are grouped into 256-weight super-blocks, with each group having its own quantized scale. Suffixes like _S, _M, and _L control how precision is distributed. For instance, Q4_K_M stores most tensors at 4 bits, but raises sensitive ones to 5 or 6 bits, averaging about 4.89 bits per weight. Q4_K_S keeps most tensors closer to 4 bits, at about 4.67 bits per weight.

I-quants (IQ4_XS and below) leverage importance matrices and codebooks, similar to image compression. IQ4_XS uses about 4.46 bits per weight, roughly 4.17 GiB for an 8B model, compared to Q4_K_M’s 4.58 GiB. That 400 MB difference often justifies choosing IQ4_XS when space is tight.

The VRAM Ceiling Decides 70B

On a 12 GB RTX 3090, both Q4_K_M and Q5_K_M fit comfortably. The bric.pe.kr benchmarks show Q4_K_M at 96 tokens/sec using 6.2 GB VRAM, while Q5_K_M runs at 92 tokens/sec with 7.0 GB. Both are practical options.

At 70B, size limits the choice. Q4_K_M at 42.5 GB fits across two 24 GB cards, leaving room for caches. Q5_K_M at 49.9 GB exceeds this capacity, requiring more resources. Q8_0, at 75 GB, demands four cards or high-memory systems like Mac Studio with 192 GB RAM. For most users, Q4_K_M is the practical upper limit for 70B models on consumer hardware.

Speed is similarly constrained by memory bandwidth. Since weights are read directly from VRAM rather than computed, larger files slow down token generation. On Llama 3.1 8B, benchmarks show Q4_K_M at 71.9 tokens/sec, Q5_K_M at 67.2, Q6_K at 58.7, and Q8_0 at 50.9. The biggest slowdown occurs moving from Q6_K to Q8_0, where bits per weight jump from 6.56 to 8.50.

For long-context use, consider quantizing the key-value cache separately with --cache-type-k q8_0. This reduces VRAM needs during inference without lowering the main model’s weight precision.

Why Pascal Punishes I-Quants

Although I-quants seem attractive (smaller files and comparable perplexity) hardware capabilities matter. On an RTX 3090, IQ4_XS runs at 88 tokens/sec versus Q4_K_M’s 96, an 8% slowdown. On a GTX 1080 Ti, the difference is more stark: 19 tokens/sec versus 25, a 25% reduction. Pascal GPUs lack the SIMD instructions needed for fast unpacking, so I-quants end up slower despite smaller size.

The takeaway: on newer GPUs like Ampere or later, IQ4_XS offers a good trade-off, saving space with minimal speed impact. On older Pascal cards, Q4_K_M remains faster and more practical. That’s why I-quants are most beneficial at 70B and larger models, where saving several gigabytes outweighs minor speed differences.

IMatrix and Quantize Workflow

The importance matrix is a key factor many overlook. It’s created by running the model over a calibration dataset, recording which weights produce the largest activations. These statistics help guide quantization, ensuring sensitive tensors retain more precision. The llama.cpp quantization internals detail how the imatrix tool hooks into the computation graph to gather activation data, and I-quants are designed to work with this process.

# Build llama.cpp, then quantize model with calibration matrix.
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release -j

# 1. Convert Hugging Face checkpoint to F16 GGUF
python convert_hf_to_gguf.py ../Meta-Llama-3.1-8B-Instruct \
 --outfile llama-3.1-8b-f16.gguf --outtype f16

# 2. Build importance matrix from calibration text file
./build/bin/llama-imatrix \
 -m llama-3.1-8b-f16.gguf \
 -f calibration.txt \
 -o imatrix.gguf

# 3. Quantize to Q4_K_M (K-quants do not require imatrix)
./build/bin/llama-quantize \
 llama-3.1-8b-f16.gguf llama-3.1-8b-Q4_K_M.gguf Q4_K_M

# 4. Quantize to IQ4_XS (I-quants should use --imatrix)
./build/bin/llama-quantize \
 --imatrix imatrix.gguf \
 llama-3.1-8b-f16.gguf llama-3.1-8b-IQ4_XS.gguf IQ4_XS

# Note: production use should verify calibration, match domain data, and re-measure perplexity.
# A generic imatrix helps, domain-specific is better.

The --tensor-type flag can override the quantization for specific tensors via regex. The internals documentation explains that attention projection tensors are more sensitive to quantization errors than feed-forward tensors, which is why variants like _M exist. If a model degrades at Q4, selectively increasing precision for attention tensors can be more efficient than moving to Q5 across the board.

Parser Flaws That Hit Quantized Models

Quantization isn’t the only concern. On May 15, 2026, six vulnerabilities in llama.cpp’s GGUF parser were disclosed, as detailed in the original advisory. These flaws occur before inference, during model loading, and can be exploited if files aren’t trusted.

The most critical, CVE-2026-27940, lacks an upper limit on the general.alignment field. Setting it high can cause integer overflows, leading to out-of-bounds reads. Other vulnerabilities involve huge memory allocations and unchecked conversions, affecting the Python implementation as well. All these issues highlight the importance of sourcing models from trusted providers and verifying integrity before use.

Measuring Damage on Your Own Hardware

The published perplexity tables serve as a starting point. Ultimately, the best way to decide is to test both quantization formats yourself. Download the same model in Q4_K_M and Q8_0, then measure perplexity and throughput on your hardware.

Note: The following code is for illustration only. Always consult official docs for production use.

# Download two quantized models
huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
 Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
 Meta-Llama-3.1-8B-Instruct-Q8_0.gguf --local-dir models

# Measure perplexity
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf -ngl 99

# Benchmark throughput
./build/bin/llama-bench -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99

# Note: For real workloads, run perplexity on your domain-specific data, not just generic texts. Coding and math tasks are more sensitive to quantization errors.

While aggregate numbers are useful, the true impact shows in complex reasoning and non-English text. Coding tasks, in particular, reveal differences that aren’t obvious with simple benchmarks. At Q4, GGUF matches GPU-specific formats for code generation; below Q4, it does not.

For more on how GGUF compares with other formats like AWQ, GPTQ, and FP8, see our guide to quantization techniques for AI inference. For insights into engine-level effects of format choices, check our comparison of local inference engines. Our GGUF Q-level guide offers practical advice for selecting the right format for your needs.

Related Reading

More in-depth coverage from this blog on closely related topics:

• Best Chunking Methods for Document Retrieval
• How to Edit OpenStreetMap for First Time
• WebAssembly Component Model Security Overview
• Implementing Model Context Protocol in Python
• PostgreSQL 18 New Features for Performance