Using Apple Silicon for AI Inference

Key Takeaways:

A 256GB M3 Ultra reports roughly 192GB usable for GPU work, not 256GB. macOS caps Metal’s working set at about 75% of physical RAM by default.

At a 128K-token context, an M3 Ultra 512GB decodes QwQ 32B at 36.87 tok/s while an RTX 5090 cannot fit the model at that context length at all. Capacity is a real advantage.

Prefill is the failure mode. Unified memory streams tokens well but digests long prompts slowly, and the gap widens toward 32K and 128K contexts.

vLLM on Mac exists only through community ports. Neither replicates vLLM’s high-concurrency batching that is its main reason to exist on Linux.

Apple Silicon is the right answer for single-user, low-power, mobile work. It does not work well for concurrent serving, production agent loops, and fine-tuning.

What 256GB Actually Buys

The first correction is that 256GB of unified memory is not 256GB of usable GPU memory. macOS caps the Metal working set at roughly 75% of physical RAM by default. On a 256GB M3 Ultra Mac Studio, that means about 192GB is available to the GPU backend, as the reviewer’s own spec breakdown and Greg Stencel’s analysis of the Metal memory limit both document. Ollama reporting “96GB available” on a 128GB machine is not a bug; it reflects the recommendedMaxWorkingSetSize property in Apple’s Metal API, which functions as a hard cap in practice.

You can raise it with an unsupported kernel parameter, sudo sysctl iogpu.wired_limit_mb=122880 for roughly 120GB on a 128GB machine, but the default is what most users will encounter. The practical implication is that a model needing 250GB of weights plus KV cache will not fit on a 256GB machine regardless of the marketing claims.

The capacity provides headroom for context. Apple handles this differently than llama.cpp. The most common client framework, llama.cpp, loads the entire context window’s KV cache up front, so a QwQ 32B model that is 19GB by itself consumes about 51GB of system memory at full context. Apple’s MLX framework only allocates KV cache as it is used, so the same model sits at 19GB when loaded and grows toward 51GB at peak. That is why the M3 Ultra can run higher-precision models than a fixed-VRAM card. Native BF16 of QwQ 32B can use over 180GB at maximum context window, as Max Weinbach’s benchmark review for Creative Strategies details.

The Prefill Problem: Where the Mac Loses

LLM inference has two phases. Prefill handles the entire prompt and produces the first token. Decode generates subsequent tokens one at a time. Decode is memory-bandwidth bound, and Apple’s unified memory performs well there. Prefill is compute bound, and that is where the architecture falls behind CUDA hardware.

The same review that praises the M3 Ultra’s generation speed identifies this as its limitation: unified memory streams generation efficiently but processes long prompts slowly before the first token, and the delay increases with context length. That delay adds up in long-document retrieval and agent loops with large system prompts, because every turn repeats it.

The measured decode numbers show where Apple performs well and where it does not. At a 128K-token context window, on the same prompt, seed, and model, using MLX on the Macs and llama.cpp on the PC, the benchmark review reports the following single-batch results:

Model (4-bit)	M3 Ultra 256GB	M3 Ultra 512GB	M3 Max	RTX 5090
QwQ 32B	33.32 tok/s	36.87 tok/s	18.33 tok/s	15.99 tok/s at 32K context; 128K did not fit
Llama 8B	128.16 tok/s	135.22 tok/s	72.50 tok/s	47.15 tok/s
Gemma2 9B	82.23 tok/s	88.50 tok/s	53.04 tok/s	35.57 tok/s
Microsoft Phi-4 14B	71.52 tok/s	75.91 tok/s	41.15 tok/s	34.59 tok/s
DeepSeek R1 4-bit	did not fit	19.69 tok/s	did not fit	did not fit

Source: Creative Strategies’ M3 Ultra review, all runs at 128K context or the model’s maximum. These are single-batch figures from one reviewer’s setup, so read them as directional.

The table requires careful reading. The Mac performs better on the models that fit and on long context: the 5090 cannot hold QwQ 32B at 128K at all, and it cannot load DeepSeek R1 4-bit at any context. The Mac falls behind on prefill latency, which does not appear in a decode-focused table. A 32K-token prompt on unified memory takes noticeably longer before emitting the first token than the same prompt on a discrete GPU with dedicated Tensor Cores. For interactive chat, that means the difference between responsive and sluggish.

No First-Class vLLM or TGI

Hugging Face’s Text Generation Inference entered maintenance mode in December 2025 and the repository went read-only on March 21, 2026, with Hugging Face’s own Inference Endpoints documentation now directing users to vLLM or SGLang. Neither replacement runs natively on a Mac. vLLM’s officially listed GPU backends are NVIDIA CUDA, AMD ROCm, and Intel XPU. Apple Silicon support exists only through community ports.

There are two of them, and they take different paths. vllm-metal is a Metal GPU backend that unifies MLX and PyTorch under one compute pathway and shipped inside Docker Model Runner on macOS. vllm-mlx is built directly on Apple’s MLX framework. The vllm-metal repository is community maintained under the vLLM GitHub organization and had 1,734 stars and 252 forks when checked on September 14, 2026, with active development. Neither is part of vLLM core.

The practical constraints are specific. vllm-metal only loads MLX-format models from the mlx-community on Hugging Face, not arbitrary AWQ or GGUF checkpoints. It requires native arm64 Python 3.12, and because Metal needs direct hardware access there is no container GPU passthrough, so it runs on the host rather than inside a container. That is an architectural difference from a CUDA deployment, not a configuration change.

On raw throughput, vllm-mlx leads. A paper published on arXiv in January 2026 by Wayner Barrios of Wiqonn Technologies reports that vllm-mlx achieved 21% to 87% higher throughput than llama.cpp across models from Qwen3-0.6B to Nemotron-30B on an M4 Max, with continuous batching scaling to 4.3x aggregate throughput at 16 concurrent requests. That is one of the more encouraging data points for Mac serving, and it comes from a single-author paper rather than an independent lab.

# vllm-metal: MLX-format models only, host execution, no container GPU passthrough
curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate

# Model id must point at an mlx-community conversion, not an AWQ or GGUF repo
vllm serve mlx-community/Qwen3-8B-4bit

# Benchmark your own hardware rather than trusting a published number
vllm bench serve \
 --backend openai \
 --base-url http://localhost:8000 \
 --model mlx-community/Qwen3-8B-4bit \
 --num-prompts 20 \
 --random-input-len 512 \
 --random-output-len 128

# Note: production use should pin the vllm-metal version, record the macOS
# version and chip, and re-run after warm-up. Published figures for this
# backend have shown wide run-to-run variance.

That variance is worth noting. Docker’s own published comparison, run in February 2026, found llama.cpp roughly 1.2 to 1.3 times faster than vllm-metal on a 1B model, with vllm-metal showing a 134 to 343 tok/s spread across individual runs. A backend that swings by more than a factor of two between runs is hard to size a capacity plan around. The project’s own release notes claim an 83x time-to-first-token improvement and 3.6x throughput gain in v0.2.0 over v0.1.0, but that is a self-comparison against its own previous version, not against llama.cpp.

MLX vs PyTorch Maturity

Apple’s MLX framework is the strongest part of the Mac inference story, and it is also a smaller world than PyTorch. The ml-explore/mlx repository had 28,408 stars and 2,246 forks when checked on September 14, 2026, and is actively maintained. It provides true zero-copy operations that use unified memory, lazy evaluation that fuses operations, and native quantization support with efficient dequantization kernels.

The gap is ecosystem depth, not framework quality. MLX has its own model conversions, its own quantization path, and its own community model hub. A team whose stack is built on CUDA-specific libraries cannot simply move it; they port it. The clearest signal of the divide is that Apple’s own MLX framework is getting a CUDA backend, led by a community developer, which shows the framework authors expect cross-platform workflows rather than a Mac-only environment.

Fine-tuning is the sharpest break. MLX supports training, and Exo Labs is building an Apple Silicon-only training node for LLMs, but the tooling, memory management, and multi-node story lag the CUDA ecosystem by a wide margin. If your work involves fine-tuning at any scale, Apple Silicon is a development convenience, not a training platform.

Multi-User Batching

vLLM’s real advantage on Linux and NVIDIA is high-concurrency serving. PagedAttention stores KV cache in fixed-size non-contiguous blocks, like an operating system managing virtual memory pages, so the same memory holds many more concurrent sequences. Continuous batching creates a new batch every iteration instead of waiting for in-flight requests to finish, so a short prompt does not queue behind a long one.

That concurrency advantage is the part that ports least cleanly to Apple Silicon. Both Mac vLLM ports are stronger as single-node, low-concurrency tools than as the multi-tenant serving engine vLLM is on Linux. The arXiv paper’s 4.3x scaling at 16 concurrent requests is real progress, but it is measured on an M4 Max in a research setting, not a production traffic mix. For a workload that needs vLLM’s feature set under load, the realistic 2026 pattern is to develop locally on Apple Silicon and deploy to Linux with GPUs for production serving.

The other concurrency cost is prefill again. In a multi-user queue, every request’s prompt must be prefilled before its first token, and Apple’s slower prefill means each request occupies the machine longer before yielding output. Aggregate throughput on a Mac declines faster than on a CUDA card as concurrency rises, for the same reason.

When Apple Silicon Is the Right Answer

The key factor is how many people or processes hit the model at once, and how long their prompts are.

Apple Silicon is the right answer when a single user runs one stream at a time, especially on models that exceed 32GB. A solo researcher running a 100B-plus-parameter model on one machine has no CUDA equivalent without a multi-GPU setup. It is also the right answer for low-power and mobile work: the M-series platform draws roughly 25 to 35W under local inference load, versus 550 to 575W for an RTX 5090, which matters for a laptop or a machine running continuously. And it is the right answer when the model needs to stay on the machine for privacy reasons and the workload is interactive rather than batched.

Apple Silicon is the wrong answer when you serve concurrent users, because the batching advantage does not port and prefill latency compounds. It does not work well for production agent loops, where every turn re-prefills a large system prompt and the delay is paid repeatedly. And it does not work well for fine-tuning, where the CUDA ecosystem’s tooling depth is not close.

The reviewer’s own conclusion captures the split better than any benchmark table: an M3 Ultra Mac Studio paired with a rented NVIDIA 8xH100 rack, where Hopper and Blackwell handle servers and the Mac handles the desk. The Mac is not competing with a data center GPU. It is competing with the other thing on your desk, and for single-user work on large models, nothing else does what it does.

The mistake is buying it for the wrong workload. A team that reads “256GB lets you run anything” and deploys a Mac Studio as a shared inference endpoint will hit the prefill wall and the batching ceiling within a week. A team that buys it as a local development and single-user workstation, and keeps production serving on Linux with GPUs, gets exactly what the marketing promised. For more on the engine choices that make that split work, see our comparison of local inference engines and the RTX 5090 vs M3 Ultra hardware breakdown.

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: