How to Speed Up Large Language Models

The KV-Cache Bottleneck

The KV cache is the reason context windows are expensive. Every token a model processes stores a key and value vector, and the cache grows linearly with sequence length. At 128K tokens, the cache can consume more GPU memory than the model weights themselves. This is why a 70B model that fits on two H100s in FP16 can still run out of memory under a long prompt with a large batch.

As we covered in the LLM Architecture Gallery analysis, modern models already fight this with architectural choices like Grouped Query Attention and Multi-Head Latent Attention, which shrink the cache by sharing or compressing key/value heads. But those are baked into the model at training time. The techniques in this article apply to any checkpoint you already have, without retraining.

Google’s TurboQuant result matters because it attacks the cache directly rather than asking model authors to change architecture. The approach, described in Google’s research blog post and an arXiv preprint, quantizes the cache to 3 bits with what Google reports as zero accuracy loss. It will be presented at ICLR 2026. The mechanism is two-stage: PolarQuant converts vectors to polar coordinates, then QJL spends a single residual bit per vector to correct the error left over from the first stage.

Weight Quantization Formats

Weight quantization compresses the model weights themselves, separate from the cache. Three formats dominate production deployments, and each targets a different runtime. The distinctions matter because choosing the wrong one for your stack can make a quantized model slower than the full-precision baseline.

GGUF is llama.cpp’s container format, built for CPU and Apple Silicon inference with optional GPU offload. Its Q4_K_M variant retains roughly 92% of FP16 quality while shrinking the model about four times. AWQ targets GPU serving through vLLM and works by protecting the small fraction of “salient” weights that contribute disproportionately to activations. GPTQ is the older GPU-focused method that uses second-order Hessian information to compensate for quantization error, with a large library of pre-quantized checkpoints. bitsandbytes is the only one of the four that supports training through quantized weights, which is what makes QLoRA fine-tuning possible.

# Convert a Hugging Face model to GGUF, then quantize to Q4_K_M
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

# FP16 GGUF from the HF checkpoint
python convert_hf_to_gguf.py /models/Llama-3.1-8B-Instruct \
 --outfile llama-8b-f16.gguf --outtype f16

# 4-bit k-quant, the usual production default
./build/bin/llama-quantize llama-8b-f16.gguf llama-8b-q4km.gguf Q4_K_M

# Note: production use should validate perplexity against the FP16
# baseline on your own eval set before shipping a quantized checkpoint.

The benchmark numbers from JarvisLabs testing on Qwen2.5-32B with an H200 GPU, as compiled in the PremAI quantization guide, show why kernel support matters more than the format label. FP16 baseline ran at 461 tokens per second with 56.1% Pass@1 on HumanEval. AWQ without an optimized kernel dropped to 67 tokens per second, slower than FP16. AWQ with the Marlin kernel hit 741 tokens per second at 51.8% Pass@1. GPTQ with Marlin ran 712 tokens per second at 46.3%.

The trade-off is visible in the table. AWQ with Marlin is the fastest and keeps most of the quality. bitsandbytes preserves quality best but runs roughly four times slower because it quantizes on the fly rather than from a pre-computed checkpoint. GPTQ shows the largest accuracy drop on code generation in these tests. The numbers come from a single third-party benchmark on one model, so validate on your own workload before committing.

KV-Cache Compression

TurboQuant attacks the cache rather than the weights, which is a newer and less settled frontier. Google reports that 4-bit TurboQuant achieves up to 8x faster attention-logit computation than 32-bit unquantized keys on H100 GPUs. The method was evaluated on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval with Gemma and Mistral models, and Google says it quantizes the cache to 3 bits without training or fine-tuning and without accuracy loss on those benchmarks.

Independent coverage is more measured. Hackaday’s analysis noted that Google’s blog does not clearly state what the “at least 6x” figure is measured against, provides no direct comparison with Nvidia’s NVFP4 format, and that some numbers are inconsistent or unspecified. NVFP4, Nvidia’s own approach, reduces cache precision from 16-bit to 4-bit and, compared to FP8, cuts latency up to 3x and halves memory with less than 1% accuracy impact. Until independent benchmarking arrives, treat TurboQuant’s headline as a vendor claim rather than a settled result.

The broader point holds regardless of which vendor wins. The KV cache is the fastest-growing memory consumer in long-context serving, and every major hardware and infrastructure player is now treating it as a first-class optimization target. MLCommons added KV-cache and vector-database workloads to its MLPerf Storage benchmark in September 2026, which gives buyers a vendor-neutral way to compare how storage systems handle the cache traffic that long-context serving generates.

Speculative Decoding

Speculative decoding changes the generation loop rather than shrinking memory. A small draft model proposes several tokens, and the target model verifies them in a single parallel forward pass. Verifying multiple tokens costs roughly the same as generating one, so accepted guesses translate directly into speedup. The idea has roots in 2018 blockwise parallel decoding and was formalized for transformers in the 2022 paper “Fast Inference from Transformers via Speculative Decoding.”

The catch is the draft model. It must be maintained, loaded into GPU memory alongside the target, and re-tuned whenever the base model changes. That dual-model requirement is a structural barrier for single-GPU and edge deployments.

DeepSeek’s DSpark, released under the MIT license in June 2026, refines both sides of the trade-off. In production tests reported by VentureBeat, DSpark improved aggregate throughput by 51% for DeepSeek-V4-Flash at a service target of 80 tokens per second per user, and per-user generation speedups of 60% to 85% for V4-Flash and 57% to 78% for V4-Pro over its prior multi-token-prediction baseline. DeepSeek also reports 661% and 406% aggregate throughput increases, but those measure behavior under strict speed targets where the older baseline collapses, not typical user-facing speed.

Nvidia’s Nemotron-Labs-Diffusion, published as an arXiv preprint in July 2026, removes the draft model entirely. The same weights serve as both drafter and verifier through two attention paths, which means the operational overhead of “self-speculation” is roughly the overhead of running one model instead of two. Nvidia reports the 8B instruct variant averages 6.82 accepted tokens per draft step versus 2.75 for Eagle3, and 8.69 versus 2.81 on structured coding and math content. Those figures come from an Nvidia-authored preprint that had not undergone peer review or independent replication at publication.

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

# vLLM with a separate draft model for speculative decoding
# Note: flags and draft-model pairing change between vLLM releases;
# confirm against the current docs for your version before deploying.
from vllm import LLM, SamplingParams

llm = LLM(
 model="meta-llama/Llama-3.1-8B-Instruct",
 speculative_model="meta-llama/Llama-3.2-1B-Instruct",
 num_speculative_tokens=5,
)

outputs = llm.generate(
 ["Summarize the failure modes of KV-cache eviction under batching."],
 SamplingParams(temperature=0.0, max_tokens=256),
)
print(outputs[0].outputs[0].text)

The practical limit on speculative decoding is acceptance rate, not draft speed. A draft that proposes long blocks of tokens the target rejects wastes compute on verification. This is why DSpark’s confidence scheduler trims low-confidence trailing guesses under heavy load instead of verifying a fixed block every time. It is also why the two techniques interact: a quantized draft model produces noisier proposals, which lowers acceptance, which erodes the speculative speedup.

Combined Effects and Limits

These three techniques stack, but they interact rather than compound. Weight quantization and KV-cache compression both reduce memory pressure, which is what lets you raise batch size or context length on the same hardware. Speculative decoding reduces the number of sequential forward passes, which is what matters at low concurrency where the GPU sits idle between token steps. Applying all three to the same deployment can backfire if a quantized draft model’s noisier proposals drag down acceptance rate.

The infrastructure layer is adapting in parallel. The MLPerf Storage benchmark expansion signals that cache traffic is now a first-class procurement criterion, not an afterthought. A KV cache that spills to disk turns a memory problem into a storage-latency problem, and the vendors now have a benchmark to argue over.

The honest summary is that the headline numbers are real but conditional. TurboQuant’s 6x memory reduction is a vendor claim pending independent benchmarking, and even the Hackaday review flags the comparison baseline as unclear. AWQ’s 741 tokens per second depends on the Marlin kernel and a specific model on a specific GPU. DSpark’s 85% figure is per-user speed at matched capacity, not throughput under all loads. Nvidia’s 6.82 accepted tokens per step is an unreviewed preprint.

For teams already running open-weight models on their own hardware, the practical sequence is straightforward. Start with weight quantization in the format your serving engine natively supports, because that is the highest-use, lowest-risk change. Then profile whether the cache or the decode loop is your binding constraint before adding cache compression or speculative decoding. As noted in our local inference tools comparison, the runtime matters less than the quantization format you lock into, and the format choice is what determines whether you can later adopt these optimizations at all.

Key Takeaways:

• Google’s TurboQuant compresses the KV cache to 3 bits per channel, cutting memory at least sixfold with no accuracy loss on long-context benchmarks, per Google Research.
• Weight quantization quality depends on the kernel as much as the format: AWQ ran 741 tok/s with Marlin versus 67 tok/s without it on Qwen2.5-32B.
• DeepSeek’s DSpark delivered 60% to 85% per-user generation speedups for V4-Flash over its prior MTP-1 baseline in production tests.
• Nvidia’s Nemotron-Labs-Diffusion removes the separate draft model, reporting 6.82 accepted tokens per step versus Eagle3’s 2.75.
• All three techniques reduce memory or sequential passes, but they interact; validate on your own workload before trusting vendor benchmarks.

Related Reading

More in-depth coverage from this blog on closely related topics:

• EU AI Act Article 50 Disclosure Guide
• Moonshine Game Streaming Setup Guide
• Using Apple Silicon for AI Inference
• Apple Silicon MacBook Accelerometer Guide
• Gemma 4 12B Model: Unified Multimodal AI

Sources and References

Sources cited while researching and writing this article:

• TurboQuant: Redefining AI efficiency with extreme compression
• arXiv preprint
• Quantization Techniques for AI Inference in 2026: GGUF, AWQ, GPTQ, and FP8 – Sesame Disk
• TurboQuant: Reducing LLM Memory Usage With Vector Quantization
• DeepSeek open sources DSpark, a new framework to speed up LLM inference by up to 85%
• arXiv preprint