Mixture of Experts in 2026: Routing, Load Balancing, and the Real Cost of Sparse Models

In April 2026, DeepSeek published the technical report for its V4 series, showing a 1.6-trillion-parameter Pro model activating only 49 billion parameters per token, trained on more than 32 trillion tokens. By July, Moonshot AI had shipped Kimi K3 with 2.8 trillion parameters, activating just 16 of 896 experts per forward pass. The numbers are getting larger, but the active slice is getting smaller. That is the story of the Mixture of Experts (MoE) architecture in 2026: the gap between total capacity and per-token compute is widening faster than hardware can keep up, and the routing mechanism that bridges that gap has become the hardest engineering problem in large language model training.

Key Takeaways

• MoE activates only a fraction of total parameters per token. A 1.6T model with 49B active does the FLOPs of a 49B dense model while carrying 1.6T of capacity in memory.
• Routing is the single hardest problem: routing collapse during pre-training, router shift during RL post-training, and load imbalance at every stage. Each failure mode has a distinct fix.
• Auxiliary-loss-free balancing, pioneered by DeepSeek-V3, adds a dynamic bias term to expert affinity scores rather than injecting a competing gradient into the loss function. It has become the default for 2026-scale MoE models.
• RL post-training on MoE architectures introduces a new failure mode called router shift, where small routing changes across policy updates cause importance ratios to spike and training to collapse. Routing Replay and RSPO are two documented fixes.
• MoE saves on compute but not on memory. A 671B model needs the same VRAM as a 671B dense model even though it only uses 37B parameters per forward pass. Deployment cost is dominated by memory, not FLOPs.

The Routing Problem: Why Expert Selection Defines Everything

A dense transformer runs every token through every parameter. A 70-billion-parameter model spends all 70 billion on every token, whether the input is a calculus proof or a grocery list. The sparse architecture replaces each feed-forward network with multiple parallel sub-networks called experts, plus a small routing network that decides which experts process each token. Only the selected experts fire.

Auxiliary-Loss-Free Load Balancing: The DeepSeek Breakthrough

The concept traces to Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton’s 1991 paper Adaptive Mixtures of Local Experts, which proposed specialized sub-networks steered by a gating network. The mechanism described in 1991 is the same one running in 2026. Only the scale has changed. What is notable is how completely sparse designs have taken over the open-weight tier. As of mid-2026, every frontier-class open-weight model uses some form of expert routing: DeepSeek V4-Pro, Kimi K2.6, Kimi K3, GLM-5.1, Qwen3.8-Max, and Llama 4 Maverick all share the same architectural family tree.

The router takes each token’s hidden representation and produces a probability distribution over all available experts. The token gets sent to its top-scoring experts, and their outputs are combined. This description hides most of the difficulty. Three routing families dominate production designs, each with a distinct failure mode.

• Fixed top-k routing is the most common. Each token selects its k highest-scored experts. Mixtral 8x7B uses top-2 across 8 experts. DeepSeek-V3 uses top-8 out of 256 routed experts plus one shared expert that handles every token. Kimi K2.6 uses top-8 out of 384 routed experts plus one shared. Kimi K3 pushes this further: top-16 out of 896 experts. Google’s 2021 Switch Transformer simplified routing to top-1, cutting communication cost but requiring careful load balancing to avoid expert collapse.
• Expert choice routing, proposed by Google in 2022 (Mixture-of-Experts with Expert Choice Routing), flips the direction: each expert holds a fixed buffer and selects its top-k preferred tokens. This guarantees load balance by construction, and Google reported roughly 2x faster training convergence on their 8B/64-expert testbed. The trade-off is that some tokens may be dropped or processed by many experts, creating quality variance at inference.
• Hash routing assigns tokens to experts with a deterministic function. It keeps perfect balance with zero learnable router parameters, but it ignores token content entirely, so experts overlap instead of specializing. A Cerebras analysis found that at 128 experts, learned routing delivers roughly 3x bigger quality gains than hash routing, with the gap widening as expert count increases.

The load balancing problem underpins all of this. Left alone, the router collapses: a few experts get selected repeatedly and the rest starve, effectively turning the model into a dense network with wasted parameters. Routing collapse is the single most common training failure in sparse architectures. The standard fix is an auxiliary loss that penalizes uneven expert use during training, but a coefficient too large interferes with the primary objective and too small invites collapse.

Auxiliary-Loss-Free Load Balancing: The DeepSeek Breakthrough

The auxiliary loss approach creates a fundamental tension. The balancing term is not the main task. If you turn it up too aggressively, its gradient distorts the model’s real learning objective. If you turn it down, experts collapse. The langcopilot MoE post-training guide notes that this is a day-to-day hyperparameter trade-off where smaller aux-loss weights often improve main task loss but worsen expert imbalance, and larger weights improve use but can hurt convergence quality.

DeepSeek-V3 sidestepped this with an auxiliary-loss-free strategy that adds a dynamic bias term to each expert’s affinity score. At each training step, the system monitors expert load and adjusts the bias upward for underloaded experts and downward for overloaded ones. The key insight is that expert selection for top-k routing uses the bias-adjusted score, while expert weighting for the actual output can still use the original gating score. No gradient interference with the main loss.

The technique was formalized in the Loss-Free Balancing paper, which validated it on MoE models with up to 3 billion parameters trained on up to 200 billion tokens. The method features an iterative process of token routing and bias updating: before the top-k routing decision, it applies an expert-wise bias to routing scores, then after routing, it updates the bias from recent load statistics. The paper reports that Loss-Free Balancing achieves both better performance and better load balance compared with traditional auxiliary-loss-controlled strategies.

The 2026 model releases show how thoroughly this approach has been adopted. DeepSeek-V4 retains the same auxiliary-loss-free mechanism from V3. Kimi K3 introduces Quantile Balancing, which derives expert allocation directly from router-score quantiles, eliminating heuristic updates and sensitive balancing hyperparameters. The MLCommons MLPerf Training v6.0 benchmarks, released in June 2026, added a DeepSeek-V3 benchmark specifically because auxiliary-loss-free load balancing had become a standard industry innovation worth benchmarking alongside Multi-head Latent Attention.

An alternative approach appeared at CVPR 2026: ERMoE (Eigen-Reparamized Mixture-of-Experts) reparamizes each expert in a learned orthonormal eigenbasis and routes with an Eigenbasis Score based on cosine similarity between token features and the expert basis. By tying assignments to each expert’s representation space, ERMoE stabilizes use, improves interpretability, and removes explicit balancing losses and their gradient interference entirely. On ImageNet and image-text retrieval benchmarks, it achieved state-of-the-art results with flatter expert loads. A 3D MRI variant improved brain age prediction by over 7% while yielding anatomically interpretable expert specializations.

The New Frontier: Reinforcement Learning Destabilizes Routing

If load balancing was the defining problem of MoE pre-training, reinforcement learning instability is the defining problem of MoE post-training in 2026. The pattern is consistent across labs: when models undergo RLHF or GRPO post-training, routing distributions drift significantly, and training sometimes collapses entirely.

An ACL 2026 paper from researchers at Microsoft identified the mechanism and gave it a name: router shift. In dense-model RL, small numerical differences may slightly perturb logits. In a sparse model, a small score difference can flip a top-k routing decision and send a token to completely different experts. Those different experts then produce different hidden states, the mismatch compounds across layers, and the importance ratios used in policy optimization become increasingly volatile. The paper found that bursty clipping behavior consistently precedes training collapse.

The proposed fix, Router-Shift Policy Optimization (RSPO), computes a per-token router-shift ratio conditioned on previously activated experts, applies a stop-gradient and lower-bound floor, and softly rescales importance ratios prior to clipping and aggregation. Across synthetic countdown tasks and real-world MATH and Code benchmarks, RSPO achieved better performance and greater stability compared to prior MoE-based RLVR methods.

A separate line of work, Predictive Routing Replay (PR2), identifies router drift as the root cause: expert activations can change drastically across model updates and differ between disaggregated rollout and training phases, causing a large rollout-training mismatch. The langcopilot MoE post-training guide recommends Routing Replay as a practical stabilization tool, with two variants. R2 (Vanilla Routing Replay) replays experts selected during rollout when gradients are computed later in training, targeting policy lag. R3 (Rollout Routing Replay) replays routing selected by the inference engine inside training, targeting the more severe train-inference mismatch that occurs when throughput-driven serving optimizations make rollout routing diverge from training routing.

This is not an academic concern. DeepSeek-V3 used its auxiliary-loss-free balancing to mitigate routing drift during its reinforcement learning stage, but the problem remains open for general-purpose solutions. The clean rule of thumb from the langcopilot guide is that smaller off-policy settings can often get away with R2, while larger off-policy settings more often need R3. For teams running RL post-training on sparse models, treating routing mismatch as a first-class failure mode rather than generic instability is the difference between a working training run and a wasted cluster week.

The 2026 Open-Weight MoE Landscape

The table below shows major open-weight sparse models that defined the first half of 2026. Every figure is sourced from the model’s own technical report or official documentation. Active-parameter counts are as reported by the model developer and should be treated as metadata, not a universal cost model.

Sources: DeepSeek-V3 technical report, DeepSeek-V4 technical report, Kimi K2.6 specifications, Kimi K3 launch details, DeepInfra model comparison, Swarmsignal MoE overview.

Several trends are visible in the data. Expert pools are becoming dramatically more fine-grained: from 8 experts in Mixtral to 896 in Kimi K3. The shared expert pattern, where one expert processes every token to maintain baseline quality while routed experts specialize, is now standard. Active parameter counts are dropping relative to total parameters: Kimi K3 activates only about 16 of 896 experts per token, an activation ratio well under 2%. The exact ratios are model-specific and should not be generalized across the entire field, but the directional trend is unmistakable.

The MLPerf Training v6.0 benchmarks, released in June 2026, reflect how thoroughly sparse computation has become the industry default. MLCommons added two new benchmarks: DeepSeek-V3 at 671 billion total parameters with 37 billion active, and GPT-OSS 20B at 21 billion total with 3.6 billion active. The GPT-OSS 20B benchmark was designed as an entry point for organizations that want to evaluate complex routing logic and sparse computation patterns on hardware configurations as small as a single 8-GPU node. MLPerf Training Working Group co-chair Shriya Rishab called sparse computation “dominant trend in AI right now,” noting that over the past two years, all major new generative AI models have used a sparse computation architecture.

Deployment Economics: Memory Is the Real Constraint

The economic case for sparse architectures comes down to simple arithmetic. Training and inference costs scale primarily with active parameters, not total parameters. DeepSeek-V3’s technical report describes a training run of 2.788 million H800 GPU hours at a reported cost of $5.576 million, assuming $2 per GPU hour. DeepSeek-V4 was pre-trained on more than 32 trillion tokens, and V4-Pro requires only 27% of the single-token inference FLOPs of DeepSeek-V3.2 at one-million-token context length, with 10% of the KV cache.

The catch is memory. All parameters, active or not, must be loaded into GPU memory because the router needs access to every expert to make its selection. A 671-billion-parameter model needs the same VRAM as a 671-billion dense model even though it only uses 37 billion parameters per forward pass. As the DeepInfra analysis of MoE economics puts it: the architecture shifts the constraint from compute to memory, which is a favorable trade for served API inference where memory cost is amortized across many requests, but a real challenge for local deployment.

The Spheron GPU requirements cheat sheet, updated July 2026, gives concrete numbers. DeepSeek V3.2 at FP8 precision needs roughly 700 GB of VRAM for weights alone, plus approximately 30 to 60 GB for KV cache, activations, and framework overhead. That puts minimum practical VRAM at around 700 GB, which exceeds 8x H100 80GB (640 GB total). Production deployment requires 8x H200 141GB (1,128 GB total) at roughly $29.60 per hour on cloud, or splitting across multiple H100 nodes with CPU offload.

DeepSeek V4 at 1 trillion total parameters needs about 1,000 GB at FP8. Eight H200s provide 1,128 GB total, which fits with limited KV cache headroom. For full production batch sizes, multi-node H200 configurations are the realistic path. Kimi K2.5 at INT4 needs roughly 630 GB across 8x H100 at approximately $20.32 per hour. Llama 4 Maverick at 400 billion total with 17 billion active is more forgiving: INT4 at roughly 200 GB fits on 4x H100 at about $10.16 per hour, while FP16 at 800 GB needs 8x H200 at $29.60 per hour.

There is also routing overhead at large batch sizes. When many tokens are processed simultaneously across many GPUs, the router’s decision about which expert to send each token to requires all-to-all communication across devices. For high-batch-size serving, this communication pattern can partially offset per-token compute savings. The practical implication, as the DeepInfra analysis notes, is that the efficiency advantage is most pronounced at low-to-moderate batch sizes and becomes more complicated at the extreme batch sizes of large-scale serving.

Kimi K3’s deployment guide offers the most extreme reference point. The model uses MXFP4 weights with MXFP8 activations, and Moonshot recommends supernode configurations with 64 or more accelerators. Because Kimi Delta Attention poses new challenges for prefix caching, Moonshot contributed a custom implementation to vLLM. This is the pattern for 2026: models that lead benchmarks require infrastructure engineering that goes well beyond off-the-shelf serving stacks.

The expert parallelism versus expert tensor parallelism distinction matters here. Expert parallelism distributes different experts across different GPUs, solving the problem of having too many experts to fit on one device. Expert tensor parallelism shards the weights of a single expert across multiple GPUs, solving the problem of one expert being too large for one device. The langcopilot guide offers a simple heuristic: if one expert does not fit comfortably, raise ETP; if the model has too many experts overall, raise EP.

NVIDIA’s Wide Expert Parallelism on NVL72 rack-scale systems, documented in mid-2026, changes the deployment calculus for the largest models. On NVL72, wide EP can deliver roughly 10x faster inference and one-tenth the token cost compared to prior-generation architectures for MoE frontier models. For organizations running DeepSeek-R1 or similarly large sparse models, this shift from per-GPU to rack-scale expert placement is the infrastructure story of the second half of 2026.

Trade-offs and Where Sparse Architectures Still Fall Short

Sparse architectures are not a free lunch. The limitations are genuine and underappreciated by teams that only see benchmark scores.

• Expert collapse remains unsolved in the general case. Despite the auxiliary-loss-free approach working at scale for DeepSeek and the Quantile Balancing approach shipping in Kimi K3, most training runs still require careful hyperparameter tuning to prevent routing collapse. The Cerebras guide on MoE routers puts it bluntly: the router can “single-handedly destroy MoE model.” Even with implementations that appear sound, mysterious training instabilities still occur, and the ERMoE paper at CVPR 2026 explicitly frames routing logit misalignment with expert structure as an unsolved problem.
• Memory requirements are punishing. A 671-billion-parameter model needs the same memory as a 671-billion dense model. For organizations that cannot afford multi-GPU clusters, this erases the efficiency advantage. The cost of frontier inference is falling, but the memory footprint keeps sparse models out of reach for many deployment scenarios. The Colibri engine, which disk-streams expert weights in pure C to run a 744-billion-parameter model on consumer hardware with 25 GB of RAM, is a creative workaround but not a production solution.
• Interpretability is harder. Dense models are already opaque. Sparse architectures add another layer of opacity: which experts activated, why, and whether different routing paths produce meaningfully different outputs. The ERMoE paper’s interpretability results are promising but limited to vision tasks. In language models, experts do not cleanly specialize by topic or task the way the architecture’s metaphor implies. The routing is messier than the diagrams suggest.
• The throughput advantage shrinks at long context. The efficiency advantage is most pronounced for short-context, high-batch inference. As context lengths grow and inference-time compute scaling becomes more common, the memory bandwidth bottleneck of loading all expert parameters dominates, and the gap between sparse and dense models narrows. DeepSeek-V4’s hybrid attention (CSA and HCA) addresses this directly, reducing inference FLOPs to 27% and KV cache to 10% of V3.2 at one-million-token context, but that required custom architectural innovation beyond standard MoE.
• RL post-training is fragile by design. Router shift is a structural consequence of having a discrete routing decision inside a continuous optimization loop. RSPO and Routing Replay mitigate symptoms, but the underlying tension between routing stability and policy improvement is inherent to the architecture. The “How Many Experts Are Enough” paper from December 2025 raised a deeper question: beyond a certain point, additional experts fragment knowledge without improving specialization. The optimal expert count depends on the relationship between model capacity and data diversity, and simply adding more experts is not a scaling strategy that works indefinitely.

The architecture that decoupled model capacity from per-token compute has become the default for frontier open-weight models in 2026. The reasons are straightforward: it is one of the clearest ways to scale model capacity without forcing active compute to grow in lockstep. The routing mechanisms that make it work, auxiliary-loss-free balancing for pre-training and Routing Replay for post-training, have matured from research curiosities into production requirements. The deployment economics are improving as Wide Expert Parallelism on NVL72-class hardware reshapes the cost curve. But the central tension remains: sparse architectures shift the bottleneck from compute to memory, and the routing instability that emerges during reinforcement learning is not fully solved. The 2026 story of MoE is that it works well enough to be the default, and the remaining problems are hard enough to keep research labs busy for years.

Sources and References

Sources cited while researching and writing this article:

• Switch Transformer
• Mixture-of-Experts with Expert Choice Routing
• [2412.19437] DeepSeek-V3 Technical Report – arXiv.org
• Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
• MLCommons MLPerf Training v6.0 benchmarks
• [2606.00395] PR2: Predictive Routing Replay for MoE-Based LLM …
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Kimi K2.6 specifications
• How Mixture of Experts Models Changed LLM Economics
• Mixture of Experts Explained: MoE Architecture