Mixture-of-Experts (MoE) Scaling: 2025-2026 Comprehensive Report

Executive Summary

The MoE architecture has become the dominant paradigm for frontier language models in 2025-2026. The key insight — that you can scale total parameters massively while keeping inference costs manageable by activating only a subset of “experts” per token — has driven a rapid escalation in model sizes. The largest MoE models now exceed 1 trillion total parameters, with DeepSeek-V3 at 671B and credible reports of models approaching or exceeding 1T. Below is a detailed survey of the landscape.

---

1. DeepSeek V3 and R1/R2

DeepSeek-V3 (December 2024)

• Total parameters: 671 billion
• Active parameters per token: ~37 billion (roughly 5.5% of total)
• Architecture: 256 routed experts + 1 shared expert per MoE layer, with top-8 routing (8 of 256 experts activated per token)
• Expert routing: DeepSeek pioneered an auxiliary-loss-free load balancing strategy, using a bias term added to expert routing scores during training to encourage balanced utilization without the distortion that traditional auxiliary losses cause. They also introduced multi-token prediction (MTP) as a training objective.
• Training efficiency: Trained on 14.8 trillion tokens. The widely-cited training cost was approximately $5.576 million in compute (2,788,000 H800 GPU-hours), a figure that stunned the industry for producing a GPT-4-class model at a fraction of the cost. This was achieved through aggressive FP8 mixed-precision training, pipeline parallelism innovations (DualPipe), and custom communication kernels that overlap compute and networking.
• Inference optimization: DeepSeek deployed Multi-head Latent Attention (MLA), which compresses KV-cache by projecting keys/values into a low-rank latent space, dramatically reducing memory during inference. Combined with sparse activation, inference throughput is comparable to much smaller dense models.
• Benchmarks: At launch, DeepSeek-V3 matched or exceeded GPT-4o and Claude 3.5 Sonnet on MMLU (88.5), MATH-500 (90.2), HumanEval (65.2 pass@1), and Codeforces (51.6 percentile). It set a new open-weight frontier.

DeepSeek-R1 (January 2025)

• Built on the V3 base, R1 applied reinforcement learning for reasoning (similar to o1-style chain-of-thought) on top of the MoE architecture.
• Same 671B total / ~37B active parameter configuration.
• Achieved substantial gains on reasoning benchmarks: AIME 2024 (79.8% pass@1), MATH-500 (97.3%), and competitive performance on GPQA Diamond (71.5%).
• Demonstrated that MoE + RL-based reasoning is a potent combination.

DeepSeek-R2 (Expected 2025-2026)

• As of early 2026, DeepSeek has not officially released a model labeled “R2,” though strong rumors and partial leaks suggest it is in development or internal testing.
• Expected to either scale to a larger MoE configuration (potentially 800B-1T+ total parameters) or to significantly improve the reasoning pipeline on a V3-class base.
• Some community reports reference “DeepSeek-V3-0324” — an updated V3 variant released in March 2025 with improved reasoning, coding, and instruction-following capabilities, which may be a precursor.

---

2. Mistral / Mixtral Lineage

Mixtral 8x7B and 8x22B (2023-2024 foundations)

• The original Mixtral 8x7B (46.7B total, ~12.9B active, top-2 of 8 experts) was a landmark in making MoE accessible to the open-source community.
• Mixtral 8x22B (141B total, ~39B active) scaled this up, demonstrating strong multilingual and coding performance.

Mistral Large 2 (July 2024) and Mistral Large 25.01 (January 2025)

• Mistral shifted strategy: Mistral Large 2 (123B parameters) is a dense model, not MoE. This was a deliberate architectural choice — Mistral found that for their target deployment scenarios (API serving with predictable latency), a well-trained dense model offered better quality-per-active-parameter than their MoE designs at this scale.
• Mistral Large 25.01 continued this dense approach with improved training.
• This is a notable counter-trend: Mistral, having pioneered open MoE with Mixtral, moved away from it for their flagship.

Mistral Medium 3 (May 2025)

• 73B dense model, reinforcing Mistral’s current preference for dense architectures at the medium scale.

Mixtral Legacy

• Mixtral remains influential as a community model. Many fine-tunes and variants continue to be built on 8x7B and 8x22B bases. However, Mistral itself has not released a new Mixtral-branded MoE model since 8x22B.

---

3. Qwen MoE Models

Qwen1.5-MoE-A2.7B (Early 2024)

• Alibaba’s first public MoE: 14.3B total parameters, 2.7B active.
• 64 experts with top-4 routing.
• Demonstrated that a 2.7B-active MoE could match dense 7B models.

Qwen2.5 Series (Late 2024 - 2025)

• The main Qwen2.5 lineup (0.5B to 72B) was primarily dense.
• However, community and internal experimentation with MoE variants has been ongoing.

Qwen3 (2025-2026)

• Qwen3 is reported to include MoE variants, though Alibaba’s public releases have emphasized dense models and “hybrid thinking” (models that can switch between fast response and extended chain-of-thought reasoning).
• Qwen3-235B-A22B: A confirmed MoE variant with 235B total parameters and 22B active parameters. This follows the trend of very high total-to-active ratios.
• Qwen3-30B-A3B: A smaller MoE variant with 30B total and 3B active — targeting efficient inference on consumer hardware.
• Qwen3 MoE models use fine-grained expert segmentation and reportedly incorporate lessons from DeepSeek’s auxiliary-loss-free balancing.

---

4. Llama MoE Variants

Meta’s Official Position

• Meta’s Llama 3 (April 2024) and Llama 3.1 (July 2024) releases were all dense models (8B, 70B, 405B).
• Llama 4, released in April 2025, marked Meta’s first official MoE Llama models:
• Llama 4 Scout: 109B total parameters, 17B active. 16 experts, top-1 routing. 10M token context window. Fits in a single H100 node.
• Llama 4 Maverick: 400B total parameters, 17B active. 128 experts, top-1 routing. 1M token context window.
• Llama 4 Behemoth: Reportedly ~2 trillion total parameters (the largest known MoE model if confirmed), still in training as of early 2025 announcements. 288B active parameters. Intended as a “teacher” model.
• The Llama 4 release was controversial: community benchmarks often showed weaker-than-expected performance relative to the parameter counts, and there were accusations of benchmark-specific tuning (“lmsys-contamination” concerns for chatbot arena rankings).

Community MoE Adaptations of Llama

• Various community projects have converted Llama 3 dense models into MoE configurations using techniques like expert-splitting and sparse upcycling. These are experimental and generally not frontier-competitive.

---

5. Other Notable MoE Models

Grok-1 (xAI, March 2024)

• 314B total parameters, reportedly ~86B active (top-2 of 8 experts).
• Open-weight release. Architecture similar to early Mixtral but larger scale.
• Grok-2 and Grok-3 (2025) architecture details have not been fully disclosed but are believed to use MoE or hybrid-MoE architectures at significantly larger scales.

Grok-3 (xAI, February 2025)

• Not fully open, but reported to be trained on xAI’s Colossus cluster (100,000 H100 GPUs).
• Believed to be an MoE model with total parameters potentially exceeding 1 trillion, though exact architecture is unconfirmed.
• Strong benchmark results: competitive with GPT-4.5, Claude 3.5, and DeepSeek-V3 across reasoning, math, and coding tasks.

DBRX (Databricks, March 2024)

• 132B total, 36B active. 16 experts, top-4 routing.
• Fine-grained expert design (16 experts, 4 chosen) rather than the Mixtral-style 8-expert, 2-chosen approach.
• Competitive with Mixtral 8x22B but trained more efficiently.

Jamba (AI21 Labs, 2024)

• Hybrid architecture: combines Mamba (state-space model) layers with Transformer layers, and uses MoE on top.
• 52B total, 12B active. Demonstrates that MoE can be combined with non-Transformer architectures.
• Jamba 1.5 (August 2024): scaled versions at 52B and 398B.

Snowflake Arctic (April 2024)

• 480B total parameters, 17B active.
• 128 experts, top-2 routing.
• Designed specifically for enterprise AI workloads (SQL generation, coding).
• Dense MoE hybrid: a 10B dense transformer combined with a 128x3.66B MoE residual layer.

Hunyuan-Large (Tencent, November 2024)

• 389B total, 52B active.
• 16 experts with a mix of shared and routed experts (following DeepSeek’s shared-expert pattern).
• Largest open-source Transformer-MoE model from a Chinese lab at the time.

Phi-3.5-MoE (Microsoft, August 2024)

• 42B total, 6.6B active. 16 experts, top-2 routing.
• Remarkably efficient: competitive with much larger models (Gemma-2 9B, Llama 3.1 8B) while using fewer active parameters.
• Demonstrated that MoE works well even at small scales for efficiency-oriented deployments.

---

6. Technical Deep Dive: Expert Routing Mechanisms

The routing mechanism — how tokens are assigned to experts — has seen significant innovation:

Mechanism	Used By	Description
Top-K Token-Choice	Mixtral, DBRX, Phi-3.5-MoE	Each token selects its top-K experts by router logit score. Simple but can lead to load imbalance.
Expert-Choice	Some Google models	Experts select their top-K tokens. Guarantees perfect load balance but tokens may be dropped or duplicated.
Auxiliary-Loss-Free Balancing	DeepSeek-V3	Adds a learnable bias to routing scores to balance load, avoiding the quality degradation of auxiliary losses.
Shared + Routed Experts	DeepSeek-V3, Hunyuan-Large	One or more experts are always active (shared), while others are routed. Shared experts capture common patterns; routed experts specialize.
Top-1 Routing	Llama 4 Scout/Maverick	Extreme sparsity: only 1 expert per token. Minimizes compute but requires very large expert pools (16-128) to maintain quality.
Hierarchical/Grouped Routing	Research proposals (2025)	Two-stage routing: first select an expert group, then select within the group. Reduces routing computation overhead.
Soft MoE / Token Merging	Google research (2024-2025)	Tokens are softly blended across experts rather than hard-routed. Avoids discrete routing decisions but harder to achieve true sparsity.

The trend is toward more experts with sparser activation (e.g., top-1 of 128, or top-8 of 256), combined with techniques to ensure load balance without hurting model quality.

---

7. Training Efficiency Gains

MoE models have fundamentally changed the cost curve:

Model	Total Params	Training Tokens	Reported Cost	Cost Efficiency
GPT-4 (est.)	~1.8T (rumored MoE)	~13T	~$100M+	Baseline
DeepSeek-V3	671B	14.8T	~$5.6M	~18x cheaper than GPT-4-class
Llama 3.1 405B (dense)	405B	15T	~$30M+ (est.)	Dense comparison point
Mixtral 8x22B	141B	~4T (est.)	Not disclosed	Modest scale

Key efficiency innovations driving these gains:
- FP8 mixed-precision training (DeepSeek, others): nearly 2x throughput vs. BF16 with minimal quality loss
- Expert parallelism combined with pipeline and tensor parallelism
- DualPipe (DeepSeek): overlapping forward/backward computation with all-to-all expert communication, hiding network latency
- Sparse upcycling: initializing MoE from a trained dense model, saving significant pre-training compute
- Multi-token prediction objectives: improve sample efficiency by predicting multiple future tokens per position

---

8. Inference Optimization

MoE inference presents unique challenges (large total model size, all-to-all communication for expert dispatch) and unique advantages (sparse activation):

Memory Management

• Expert offloading: Keep only active experts in GPU memory, offload others to CPU/NVMe. Enables running 600B+ MoE models on consumer hardware (slowly).
• MLA / KV-cache compression: DeepSeek’s Multi-head Latent Attention reduces KV-cache by 5-13x compared to standard GQA.
• Quantization: MoE models are particularly amenable to quantization (e.g., GPTQ, AWQ, GGUF formats) because expert weights that are rarely activated can tolerate more aggressive quantization.

Throughput Optimization

• Expert batching: Accumulate tokens for each expert and process in batches for better GPU utilization.
• Speculative expert loading: Predict which experts will be needed for upcoming tokens and pre-load them.
• Dedicated inference frameworks: vLLM, TensorRT-LLM, SGLang, and MLC-LLM have all added MoE-specific optimizations in 2025.

Deployment Patterns

• MoE models shine in high-throughput serving where many requests share the same expert pool.
• They are less advantageous for single-request latency on limited hardware, where the large total model size creates memory pressure even if compute is sparse.

---

9. Scale Comparison: Largest MoE Models (as of early 2026)

Model	Total Params	Active Params	Experts	Routing	Status
Llama 4 Behemoth	~2,000B	~288B	Unknown	Unknown	In training
Grok-3 (est.)	~1,000B+	Unknown	Unknown	Unknown	Deployed (closed)
DeepSeek-V3	671B	37B	256+1 shared	Top-8, aux-free	Released, open-weight
Snowflake Arctic	480B	17B	128	Top-2	Released, open
Llama 4 Maverick	400B	17B	128	Top-1	Released, open
Jamba 1.5 Large	398B	~94B	MoE+Mamba hybrid	Top-2	Released
Hunyuan-Large	389B	52B	16	Shared+routed	Released, open
Grok-1	314B	~86B	8	Top-2	Released, open
Qwen3-235B-A22B	235B	22B	Unknown	Unknown	Released
Mixtral 8x22B	141B	39B	8	Top-2	Released, open
DBRX	132B	36B	16	Top-4	Released, open
Llama 4 Scout	109B	17B	16	Top-1	Released, open

The largest confirmed MoE model with disclosed parameters is Llama 4 Behemoth at approximately 2 trillion total parameters. The largest released and usable open-weight MoE model is DeepSeek-V3 at 671B total.

---

10. Key Trends and Observations

1. Total-to-active ratio is increasing. Early MoE (Mixtral 8x7B) had a ~3.6x ratio. DeepSeek-V3 is ~18x. Llama 4 Maverick is ~23.5x. The trend is toward ever-sparser activation with massive expert pools.

2. MoE has won the scaling debate for frontier models. Nearly every top lab is using or exploring MoE. Dense models persist mainly at smaller scales (sub-100B) where deployment simplicity matters.

3. Open-weight MoE models have closed the gap with proprietary ones. DeepSeek-V3 and R1 demonstrated that open MoE models can match closed-source frontier models at a fraction of the cost.

4. Expert routing remains an active research area. The shift from simple top-K to auxiliary-loss-free balancing, shared experts, and hierarchical routing shows that routing quality is a key differentiator.

5. Training costs have plummeted. DeepSeek-V3’s $5.6M training cost for a GPT-4-class model was a watershed moment, enabled largely by MoE’s parameter efficiency combined with hardware-aware training optimizations.

6. Inference is the new bottleneck. While MoE saves compute, the total model size still demands significant memory. Efficient inference serving for MoE models — expert offloading, quantization, speculative loading — is where much of the engineering effort is now focused.

7. Hybrid architectures are emerging. Jamba (MoE + Mamba), various MoE + long-context designs (Llama 4 Scout’s 10M context), and MoE + reasoning RL (DeepSeek-R1) suggest that MoE is becoming a composable building block rather than a standalone architecture.

---

This report covers the state of MoE scaling through early 2026. The field is moving rapidly — by mid-2026, DeepSeek-R2, further Llama 4 variants, and potentially new entrants (Google’s Gemini architecture details, if disclosed) may significantly update this picture. The overarching trajectory is clear: MoE is the architecture of choice for scaling language models beyond 100B parameters, and the frontier is now firmly in the multi-hundred-billion to multi-trillion parameter range.