Hardware Needed to Run Large AI Models

Hardware Requirements to Run 70B Models Locally in 2026: VRAM, Quantization, and Real Throughput

A 70B model at full FP16 precision requires about 141GB of memory, and no consumer GPU currently offers more than 32GB. This difference explains why most attempts to run 70B models locally fail before generating the first token. Quantization reduces this requirement, and it performs better at 70B scale than with smaller models: a 70B at Q4_K_M uses around 42.5GB and retains nearly all of its FP16 quality, while a 7B model loses more quality at the same precision. The practical minimum for running a dense 70B in 2026 is 48GB of fast memory, and whether you use dual GPUs, unified memory, or a workstation card affects both your throughput and power consumption.

Key Takeaways:

• A dense 70B at Q4_K_M requires about 42.5GB of weights; expect to use 48-50GB of VRAM or unified memory once KV cache is included.
• Q4_K_M maintains FP16-level quality at 70B scale, increasing perplexity by about 1.5%. Q2_K increases perplexity by roughly 11% and is not recommended.
• No current consumer GPU can hold a 70B model at Q4 precision alone. Dual RTX 3090/4090/5090 setups, Apple unified memory, or workstation cards are necessary.
• Throughput depends on memory bandwidth rather than core count: dual RTX 3090s deliver 14-16 tokens per second, dual RTX 5090s 25-30 tokens per second, and a 128GB Strix Halo about 5-6 tokens per second on the same dense model.
• For most users asking how to run 70B models, a mixture-of-experts (MoE) model like gpt-oss 120B fits in about 14GB and runs on a single 24GB card.

The VRAM Math for 70B Models

VRAM in gigabytes equals the number of parameters in billions multiplied by bytes per parameter. At FP16 precision, that calculation is 70 × 2 = 140GB of weights alone. Quantization reduces the bytes per weight, and the actual file sizes matter because GGUF builds include metadata, tokenizer tables, and mixed-precision layers beyond the theoretical minimum.

Measured GGUF file sizes for Llama 3.3 70B, the standard dense 70B model in 2026, are approximately 141GB at FP16, 75GB at Q8_0, 58GB at Q6_K, 50GB at Q5_K_M, 42.5GB at Q4_K_M, 34GB at Q3_K_M, and 26GB at Q2_K, according to InsiderLLM’s running-70B guide. Adding a few gigabytes for KV cache headroom brings the Q4_K_M target to about 48-50GB, which is the practical minimum for a usable dense 70B. The Analytics Insight AI PC breakdown estimates the same Q4 figure at 40-45GB for weights and recommends 64GB of total memory for comfortable operation.

One correction to earlier local AI coverage: the RTX 5090 comes with 32GB of GDDR7 memory, not 24GB. This extra 8GB allows a single 5090 to load a 70B at Q2_K precision outright or run Q4_K_M with significant layer offload to system RAM. However, it still cannot hold Q4_K_M entirely in VRAM.

Quantization Formats and Their Quality Cost

Quantization compresses model weights with some loss of precision, similar to how saving a photo as a JPEG reduces file size with some quality loss. The GGUF naming system indicates the average bits per weight (Q4, Q5, Q6, Q8) plus the variant (the K-quant family, with S/M/L suffixes specifying how precision is distributed across tensors). Larger models handle aggressive quantization better than smaller ones.

Quality data from Presenc AI’s 2026 quantization benchmarks illustrate the trade-offs. On a Llama 4 70B, perplexity increases by 0.0% at Q8_0, 0.3% at Q6_K, 0.9% at Q5_K_M, and 1.5% at Q4_K_M compared to FP16. Perplexity jumps to 3.8% at Q3_K_M and 11% at Q2_K. Reasoning benchmarks degrade faster than perplexity: on GSM8K math accuracy, Q4_K_M loses about 1.5%, Q3_K_M about 5%, and Q2_K about 14%. Users running agents or tool-use loops should use Q4 or higher.

Sources: file sizes from InsiderLLM; perplexity differences from Presenc AI.

Two format families are relevant for 70B models. GGUF with K-quants is the portable option: it runs on CPU, NVIDIA, AMD, and Apple Silicon through llama.cpp and Ollama, and most community checkpoints use it. AWQ and GPTQ are GPU-only formats designed for vLLM and similar servers; AWQ outperforms GPTQ by about 0.5-1.0% perplexity at the same bit-width on modern Llama and Qwen models, but neither works on Macs. For details on how these formats interact with runtime choices, see our guide to quantization methods for llama.cpp.

Hardware Choices That Actually Fit 70B

Single consumer GPUs generally cannot hold a dense 70B model. The RTX 3090 and 4090 both have 24GB of memory, which is less than the 26GB Q2_K file size, so they require offloading layers to system RAM. The RTX 5090’s 32GB can hold Q2_K outright and Q3_K_M only with a very short context. The 5090 is the only consumer card capable of loading a 70B model, but running it at Q2 precision is more of a proof of concept than a practical daily setup.

Dual-GPU configurations make running 70B models feasible on consumer hardware. Two RTX 3090s combine 48GB of memory and can run Q4_K_M at 4-8K context lengths. The RTX 3090 is also the last consumer NVIDIA card with NVLink, which benefits tensor-parallel workloads, although llama.cpp’s default layer-split mode pools VRAM without speeding up generation. Two RTX 4090s or two RTX 5090s provide 48GB and 64GB pools respectively, with more compute power per card. A mixed pair of RTX 5090 and RTX 4090 offers 56GB, enough for Q6_K or even Q8_0 precision, according to Compute Market’s multi-GPU guide.

Unified-memory systems avoid VRAM limits. Apple’s M5 Ultra Mac Studio starts with 96GB of unified memory and 1.2TB/s bandwidth, capable of holding a 70B model at Q8_0 precision with room for KV cache, according to Vucense’s 2026 hardware report. AMD’s Ryzen AI Max+ 395 (Strix Halo) offers up to 128GB of on-package LPDDR5x memory at 256 GB/s, and the TechTimes review of the GMKtec EVO-X2 notes that its integrated GPU can use up to 96GB of that pool as effective VRAM. On the professional side, the RTX PRO 6000 Blackwell Workstation Edition has 96GB of GDDR7 memory at 1,792 GB/s, enough to hold a 70B model at high precision on a single card.

Real Throughput by Configuration

Token generation speed depends on memory bandwidth rather than compute power. Each token requires reading the model weights from memory once, so the maximum tokens per second equals memory bandwidth divided by the model size in memory. This explains why a 1,792 GB/s RTX 5090 outperforms a 307 GB/s Mac on smaller models, while the same Mac can run a 70B model that the 5090 cannot load.

Sources: dual-GPU and A100 throughput from Compute Market and InsiderLLM; Mac and Strix Halo figures from Vucense and TechTimes. The M5 Ultra figure is a bandwidth ceiling, not a measurement, because the machine shipped on September 22, 2026 and independent 70B runs were not yet available.

The Strix Halo’s 5-6 tokens per second on a dense 70B model is slow, but the same chip runs Qwen3-235B at about 11 tokens per second because that model is a mixture of experts activating only about 22 billion parameters per token. It also runs 7B-13B models at 30-45 tokens per second, fast enough for interactive use. MoE models make the capacity-versus-throughput trade-off manageable, which is why TechTimes describes the EVO-X2 as a cloud-subscription alternative rather than a throughput-focused machine.

KV Cache: The Memory Nobody Budgets For

Model weights are only part of the total memory requirement. The KV cache stores attention states for every token in the context window and grows linearly with sequence length. For a 70B model at 4K context, the FP16 KV cache uses about 2.4GB; at 32K context, it requires roughly 14GB; and at 128K context, it approaches 39GB. This explains why a dual RTX 3090 setup running Q4_K_M (42.5GB of weights within 48GB total) has almost no room left once the context length exceeds 8K tokens.

A practical guideline from the PromptQuorum hardware guide is to add 25% to your weight estimate for an 8K context and up to 100% for a 32K context. Quantizing the KV cache separately, which llama.cpp supports via the --cache-type-k q8_0 flag, halves that memory cost without affecting weight precision. Ollama defaults to a 2,048-token context, so if you do not increase it, memory use remains low. Running retrieval-augmented generation or agent chains at 32K or 128K context lengths consumes 14-39GB of memory that a plan based only on weights will not cover.

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

# Run a 70B Q4_K_M GGUF across two GPUs with a quantized KV cache.
# --tensor-split takes ratio weights, not gigabytes: 50,50 splits evenly.
# --main-gpu holds the KV cache, which grows with context and can OOM
# the smaller card even after a clean model load.

llama-cli \
 --model models/Llama-3.3-70B-Instruct-Q4_K_M.gguf \
 --n-gpu-layers 999 \
 --tensor-split 50,50 \
 --main-gpu 0 \
 --split-mode layer \
 --ctx-size 32768 \
 --cache-type-k q8_0 \
 --cache-type-v q8_0

# Note: production use should verify actual per-device VRAM assignment in
# the load log and re-run at your real context length. A split that loads
# cleanly at 8K can OOM at 32K because KV cache lives on --main-gpu.

Buyer Guidance and Build Costs

Prices in 2026 are affected by the memory shortage. The PromptQuorum guide reports RTX 5090 street prices ranging from $4,300 to $5,000, compared to a $1,999 MSRP, and discontinued RTX 4090 cards now sell used for more than their original price. Memeburn notes that DRAM prices are 300-600% higher than 2024 lows. Treat the following figures as mid-2026 street estimates and check current listings before purchasing.

• Dual RTX 3090 (48GB, approximately $1,400-$2,000 for the cards). The budget option for dense 70B models. Provides 14-16 tokens per second at Q4_K_M, with a combined power draw of 700W. It is the only consumer pair with NVLink. The trade-off is heat and noise: two 3090s require a power supply of 1000W or more.
• Dual RTX 4090 (48GB, approximately $3,200-$4,000). Same memory capacity, delivers 20-24 tokens per second, and consumes about 900W. The 4090s are triple-slot cards, so case clearance is a consideration. Being discontinued, used prices exceed the original MSRP.
• Dual RTX 5090 (64GB, approximately $8,600-$10,000). The fastest consumer option, delivering 25-30 tokens per second with support for 16-32K context lengths. Requires a 1600W power supply. Suitable only for heavy production workloads.
• Mac Studio M5 Ultra (96GB base, $5,499). Runs a 70B model at Q8_0 precision without offloading, operates silently with low power consumption, and cannot be upgraded after purchase. The nearest single-GPU equivalent is an RTX PRO 6000, which costs roughly $12,000-$14,000 per card.
• Strix Halo mini PC (128GB, approximately $1,500-$2,400). The most affordable way to fit a 70B model in one machine, delivering 5-6 tokens per second on dense models. Runs on a standard 120V outlet and requires ROCm configured with the GPU device set to gfx1151; otherwise, Ollama and llama.cpp fall back to CPU-only mode, roughly halving throughput.

Dense 70B models are no longer the default choice they were in 2024. A mixture-of-experts model like gpt-oss 120B loads in about 14GB with its native MXFP4 weights and runs on a single 24GB card, delivering faster performance than a dense 70B. Dense 70B models still perform best on the most difficult single-shot reasoning tasks, where MoE routing sometimes directs tokens to the wrong expert. However, for coding, chat, and retrieval-augmented generation, the MoE approach is more cost-effective and fits hardware available today. If your workload is single-user and interactive, start by choosing an engine from our comparison of local inference tools before investing in a dual-GPU setup.

FAQ

How much VRAM do I need to run a 70B model locally in 2026? You should allocate 48-50GB for the Q4_K_M precision level: about 42.5GB for weights plus several gigabytes for KV cache. Q6_K requires 64GB, Q8_0 needs 84GB, and FP16 demands 152GB or more.

Can a single RTX 4090 or 3090 run a 70B model? Not at a usable quality level. Both have 24GB of memory, less than the 26GB Q2_K file size, so they must offload layers to system RAM, reducing generation speed to roughly 1-5 tokens per second. Use two cards or unified memory instead.

Is the RTX 5090 enough for 70B? It cannot hold a 70B model at full Q4_K_M precision. Its 32GB memory fits Q2_K outright and Q3_K_M only with a very short context. Two RTX 5090 cards provide 64GB and can run Q4_K_M comfortably.

Does memory bandwidth matter more than core count? Yes, for models of this size. Token generation speed depends on memory bandwidth, so an M3 Ultra with 819 GB/s and a Strix Halo with 256 GB/s deliver very different throughput on the same 43GB model, even though both can hold it.

What quantization should I use for a 70B model? Q4_K_M is suitable for most users. It maintains quality close to FP16 while using about a quarter of the memory. Use Q5_K_M or Q6_K if you have more memory and your workload involves long contexts or multi-step reasoning. Avoid Q2_K entirely.

Is dense 70B still worth buying hardware for? Only if you require the most advanced single-shot reasoning and can accept speeds between 5 and 30 tokens per second. For most other tasks, a MoE model with 120B total parameters and roughly 5B active parameters fits on a single 24GB card and runs faster.

Related Reading

More in-depth coverage from this blog on closely related topics:

• Cloudflare Outage February 2026 Lessons
• How to Speed Up Large Language Models
• Best Infrastructure as Code Tools Comparison
• EU AI Act Article 50 Disclosure Guide
• Moonshine Game Streaming Setup Guide

Sources and References

Sources cited while researching and writing this article:

• Running 70B Models Locally , Exact VRAM by Quantization
• AI PCs in 2026: What Hardware is Needed to Run AI Models Locally?
• Local LLM Quantization Quality Benchmarks 2026 (GGUF, MLX, AWQ, GPTQ) | Presenc AI
• Multi-GPU LLM Setup 2026 , Run 70B-405B Locally | Compute Market
• Local LLM Hardware 2026: What Actually Runs 70B Models
• Local AI Inference Mini PC Now Runs 235B Models: AMD Ryzen AI Max+ 395 vs. Cloud Costs
• Local LLM Hardware Requirements 2026: 8GB to 70B by VRAM
• Best Hardware for Running Open-Source AI Models Locally in 2026