A 284B Model on a 64GB CPU, No GPU
A project called llama-modchip has published a working example that runs DeepSeek-V4-Flash, a 284-billion-parameter mixture-of-experts model, on a 64GB CPU workstation with no GPU. The headline is easy to misread. The machine is not a small laptop, the model is not running at its original precision, and the software is a patched inference stack rather than stock llama.cpp.
The release, version 2.4.0, reports a 56.56 GiB model file, 57.61 to 58.9 GiB of resident memory at a 256K context, 43.16 to 44.36 tokens per second for prompt prefill, and 9.06 to 10.74 tokens per second for generation. With the project’s speculative decoder, generation rises to 11.5 to 14.7 tokens per second. The test machine is an AMD Ryzen 9 9955HX with 16 Zen 5 cores, 64GB of DDR5, and no discrete GPU.
| Metric | Project figure |
|---|---|
| Model | DeepSeek-V4-Flash, 284B total / 13B active |
| File size | 56.56 GiB |
| Peak memory at 256K context | 57.61 to 58.9 GiB |
| Prompt prefill | 43.16 to 44.36 tok/s |
| Generation, no speculation | 9.06 to 10.74 tok/s |
| Generation with speculation | 11.5 to 14.7 tok/s |
| GPU | 0 |
What is actually running
DeepSeek-V4-Flash is an official DeepSeek model with 284B total parameters and 13B activated parameters per token. The official release uses FP4 for the MoE experts and FP8 for most other weights, supports a context window of one million tokens, and ships under the MIT license. The modchip build keeps the model architecture but re-quantizes it far below the official precision.
The 284B label describes the whole model, not the work done for each token. Only a fraction of the experts fire on any given token, which is why a sparse model can be faster than a dense model of the same size. The 13B active parameter count is the relevant number for compute and memory bandwidth during generation.
How 56.56 GiB fits
The official FP4 and FP8 weights are far larger than 64GB. The project shrinks them with two ideas. Dense attention, MLA, and indexer projections are transcoded to a custom block format the project calls Fractal-64 at 3.75 bits per weight. The 256 routed experts, which make up most of the parameters, go much lower, to IQ1_S at 1.5625 bits per weight, with some sensitivity-anchored layers kept at IQ1_M or IQ2_XXS.
The average works out to roughly 1.6 bits per parameter. That is the core trade: the model fits because it is compressed to a small fraction of its original size, not because the CPU has become a GPU.
The project also uses an analytical compression step it calls SwiGLU-H2. It estimates the importance of each expert slice from the norms of the gate and up projection weights, without running a calibration corpus through the 284B model. The practical benefit is that the transcode can be done from the weights alone. The cost is that the ranking approximates what every prompt will need.
The final file is 60,739,342,752 bytes, or 56.56 GiB, split across three GGUF shards. The custom tensor type is not recognized by unmodified llama.cpp, Ollama, or LM Studio. Loading the file in those tools fails with an unknown tensor type error. Users need the project’s patched runtime, a prebuilt Linux x86_64 binary from the release, or a build from its patches.
Why it runs at 9 to 14 tokens per second
CPU inference of a sparse model is mostly a memory bandwidth problem. During single-token generation, the CPU streams the active expert weights from DDR5 for each token. A 284B model with 13B active parameters is still a large working set. Stock llama.cpp on the project’s 76.87 GiB IQ2_XXS baseline measured 5.91 tokens per second. The 9 to 10 token result comes from a set of low-level changes: AVX-512 kernels for the custom quant formats, huge pages to cut TLB misses, cache-aware tiling across registers and L1, L2, and L3, and thread pinning to physical cores.
Prompt prefill is a different workload. It processes many tokens in a batch, so the arithmetic intensity is higher and the CPU can use matrix-matrix kernels instead of matrix-vector kernels. That is why prefill reaches 43 to 44 tokens per second while generation stays near 9 to 10. The two numbers should not be compared directly.
The optional speculative decoder borrows an idea from chess engines. A small 1.64MB Stockfish-PVS drafter, paired with the project’s RouteCrack predictor, proposes tokens from the live context. The main model verifies those tokens in a batch. When the acceptance rate is high, the machine produces more than one token per forward pass. The project reports 94.5% draft acceptance and 11.5 to 14.7 tokens per second with this path.
Long prompts remain expensive. At 44 tokens per second, a 32K-token prompt takes about 12 minutes to process before the first output token, ignoring any cache reuse. A full 256K prompt takes roughly 1.6 hours at the same rate. The 256K context is supported, but it is not interactive for cold prompts.
What the benchmark does not prove
The result is a project-reported example on one machine. It has not been independently reproduced by a neutral lab. The repository includes benchmark scripts and raw results, and it links community runs on a Threadripper and an AWS Graviton3, but those are still contributor reports, not a controlled third-party audit.
The memory margin is thin. A 64GB machine has about 59.6 GiB available before the operating system and other processes, because memory is sold in decimal gigabytes while the kernel reports GiB. The project’s 58.9 GiB peak leaves about 0.7 GiB of headroom. A desktop environment, a browser, Docker, or a different kernel configuration can consume that. The project says the model runs entirely in RAM with no swap, but a 64GB label alone does not guarantee a clean fit.
Quality is the other open question. The project reports 100/100 on its own quality and tool-calling gate, and near-identical top-1 and top-3 outputs for its default profile. Those numbers come from the project’s tests, not from an independent evaluation. A file at roughly 1.6 bits per parameter is not the same artifact as the official FP4 and FP8 release. Coding, long-context retrieval, and rare-language tasks are the places where aggressive quantization usually shows damage first. Anyone using this for production should run their own evaluation on their own prompts.
The runtime is also a supply-chain decision. The model and the prebuilt binary are third-party downloads. The custom format has one implementation, in this project, and the patches track a specific llama.cpp revision. That is a different maintenance burden from running a mainstream GGUF in Ollama or LM Studio.
Who this is for
The practical case is a CPU-only workstation that needs to run a large model locally, with no GPU budget and no data leaving the machine. A 9 to 14 token per second chat is usable for coding assistance, document analysis, and batch generation. It is not a replacement for a GPU inference server, and it is not a good fit for many concurrent users.
For most people, a 30B-class MoE is the better trade. The same project reports 25 to 60 tokens per second for Nemotron-3.5-30B-A3B on a 16-core Zen 5 machine, with a 14.35 GiB file. That fits in 32GB, runs faster, and leaves room for the operating system. The 284B result shows how far compression and CPU kernels can be pushed. Treat it as a specific experiment for a 64GB desktop, not a general recommendation.
The broader point is that the gap between CPU and GPU inference is now a software gap as much as a hardware gap. Custom quantization, cache-aware kernels, and speculative decoding can move a frontier model from “needs a datacenter” to “needs a high-end desktop.” The tradeoffs are real: bits are thrown away, the runtime is specialized, and the memory ceiling is tight. But the direction is clear. For a comparison of how GPU inference costs are changing, see Nvidia GPU innovations and AI inference costs.