When DeepSeek V3 dropped at the end of 2024, my team spent a week trying to serve it the same way we served dense models. We had eight H100s, the right amount of total VRAM on paper, and a vLLM config we had used reliably for Llama 70B. The result was abysmal latency, frequent OOM errors, and eventually a very humbling conversation about why a 671-billion-parameter model with only 37 billion active parameters was behaving nothing like a 37-billion-parameter model.
That experience taught me that Mixture-of-Experts is not a variation on dense model architecture. It is a fundamentally different infrastructure problem. By mid-2026, MoE powers over 60% of the serious open-weight models: Llama 4, DeepSeek V3, Qwen 3 MoE, Gemma 4. If you are running self-hosted inference, you are either already dealing with MoE or you are about to be. This article is the guide I wish I had existed when we started.
What MoE Actually Is (and Why It Changes Everything)
A dense transformer model processes every token through every parameter on every forward pass. A 70B dense model activates all 70 billion parameters for every single token. That is conceptually simple and operationally predictable: memory footprint equals model size, throughput scales linearly with VRAM.
MoE breaks that assumption completely. In an MoE model, the feed-forward layers (which in a standard transformer represent the majority of parameters) are replaced by a set of parallel “expert” networks plus a lightweight routing network. The router looks at each token and selects a small number of experts, typically two out of eight or sixteen or even sixty-four, to process it. The remaining experts sit idle for that token.
DeepSeek V3 has 671 billion total parameters across 256 experts, but it activates only 37 billion parameters per token because the router selects just 8 experts at a time. That 37B active parameter count is what makes the model fast at inference time. But you still have to load all 671 billion parameters into memory, because any token might route to any expert.
This is the core tension that breaks naive infrastructure assumptions: you pay the memory cost of a 671B model but the compute cost of a 37B model. Your VRAM budget and your compute budget now live in completely different universes, and all your prior intuitions about GPU utilization and throughput go out the window.
The Memory Math Is Different
For dense models, memory planning is straightforward. A 70B model in BF16 takes about 140GB. Load it onto two H100s with 80GB each, and you are done. The relationship between parameter count and memory is linear and boring.
For MoE models, you still load all parameters, so DeepSeek V3 in BF16 needs roughly 1.3TB of VRAM. That requires sixteen H100 80GB cards just for weights, before KV cache. Llama 4 Scout at 109B total parameters with 17B active is more manageable at around 218GB, but that is still three H100s for weights alone.
The KV cache calculation also changes. In a dense model, attention heads exist once per layer. In an MoE model, the attention heads are dense (each expert layer is a different FFN, but the attention is shared), so KV cache per layer is similar. But because you need more GPUs to hold the weights, you typically have less remaining VRAM per node for KV cache, which caps your batch size and concurrency.
I have watched teams get burned by this. They spec a cluster based on the “active parameter count” math, realize they cannot actually load the model, and scramble to procure more GPUs. Do your memory planning against total parameter count, not active parameter count. The active count only tells you about compute, not memory.
Expert Parallelism: The Parallelism Strategy Dense Models Never Needed
Dense model inference has two main parallelism strategies. Tensor parallelism (TP) splits individual weight matrices across GPUs; each GPU holds a slice of every layer, and all-reduce operations synchronize after each forward pass. Pipeline parallelism (PP) assigns whole layers to different GPUs; tokens flow through stages sequentially. Most production setups use a combination of both.
MoE models need a third strategy: expert parallelism (EP). Because the experts are independent, you can assign different experts to different GPUs entirely. GPU 0 holds experts 0-15, GPU 1 holds experts 16-31, and so on. When the router dispatches a token to expert 23, that token gets sent to GPU 1 for processing, then the result is sent back.
This sounds clean in theory. In practice, it introduces a new communication pattern called all-to-all, which is categorically different from the all-reduce you see in tensor parallelism. With all-reduce, every GPU sends data to every other GPU and the result is symmetric and balanced. With all-to-all in MoE, different GPUs might receive wildly different numbers of tokens depending on how the router dispatches them. If experts 0-7 happen to be popular for your workload, the GPUs holding those experts get hammered while others sit idle. This is the expert load imbalance problem, and it is much more operationally painful than anything you encounter with dense models.
The standard mitigation is a combination of approaches. First, good MoE models include a load-balancing auxiliary loss during training that pushes the router toward distributing tokens more evenly; DeepSeek V3’s architecture includes this explicitly. Second, you can tune your expert parallelism group size to limit how far tokens have to travel. If you run EP across four GPUs within a single NVLink domain rather than across eight GPUs spanning two nodes, the all-to-all stays on fast interconnects.
For practical cluster design, this means you want your EP group to fit within a single NVLink-connected server when possible. An H100 DGX or H100 SXM server with 8 GPUs and 900GB/s NVLink bandwidth is the right unit of parallelism for EP. Crossing to a second node via InfiniBand or RoCE, even at 3.2Tbps, introduces enough latency that expert routing becomes a bottleneck. Our GPU cluster networking guide has the bandwidth numbers you need to work through these tradeoffs.
When your model requires more than eight GPUs, you combine EP with TP or PP. For DeepSeek V3 on sixteen H100s, a common configuration is EP=8 within each eight-GPU node, with pipeline parallelism between nodes. Each node holds a complete set of all 256 experts (since they are distributed across the eight GPUs within the node), and the two nodes handle different pipeline stages. This minimizes cross-node all-to-all traffic.
vLLM and SGLang MoE Support in 2026
When I first ran DeepSeek V3, vLLM’s MoE support was functional but not well optimized. The January 2025 V1 release changed that substantially, adding FlashInfer integration with autotuning specifically for MoE kernels. The custom MoE kernel selection now benchmarks your specific hardware and expert configuration at startup, which adds about 60-90 seconds to your first launch but gives 15-30% better throughput compared to generic kernels.
The key vLLM parameters for MoE are --tensor-parallel-size and --pipeline-parallel-size, which you combine with the MoE-specific --enable-expert-parallel flag. Setting EP correctly matters more than it does for dense models. I have seen configurations where specifying the wrong EP group size caused constant all-to-all communication across slow interconnects, cutting throughput in half.
SGLang has taken a different approach to MoE optimization, focusing on RadixAttention-based prefix caching combined with expert routing. For workloads with repeated system prompts (which is most production API deployments), SGLang’s prefix cache means the attention KV for the shared prefix is computed once and reused across requests. This interacts nicely with MoE because the router for shared prefix tokens is deterministic: the same prefix token routes to the same experts every time, so you can warm those experts up early. In my experience, SGLang beats vLLM on MoE throughput by 10-20% for typical API traffic patterns where system prompts dominate the token budget. The comparison in our LLM inference engines article now needs an MoE-specific footnote.
The llm-d project, which Red Hat, Google Cloud, IBM Research, NVIDIA, and CoreWeave launched in May 2025, adds Kubernetes-native distributed serving on top of vLLM. If you are running self-hosted inference on Kubernetes, llm-d is worth evaluating for MoE workloads because it handles the expert locality scheduling that vanilla Kubernetes pod placement gets wrong.
Quantization for MoE: The Rules Change
Quantizing dense models to INT4 or INT8 is well understood. You lose a little accuracy, gain a lot of memory efficiency, and the math is straightforward: a 70B model at INT4 fits in about 35GB instead of 140GB.
MoE quantization is trickier for a few reasons. First, the experts are not all equally important or equally sensitive to quantization. The shared attention layers, which process all tokens regardless of routing, are often more sensitive to quantization than individual expert FFN layers. Second, AWQ and GPTQ calibration for MoE models requires calibration data that covers all the experts adequately; if your calibration set happens to underrepresent certain routing paths, experts on those paths will be miscalibrated.
In practice, I have seen mixed-precision approaches work well for large MoE models. Run the shared attention layers at BF16, quantize the expert FFN layers to INT4, and you get reasonable accuracy with about 40-50% memory reduction. DeepSeek V3 at this mixed precision fits in about 750GB of VRAM instead of 1.3TB, which means twelve H100s instead of sixteen. For the team budget, that difference is enormous.
GGUF support for MoE in llama.cpp has improved significantly as well. For teams running inference on CPUs or consumer GPUs, llama.cpp’s MoE offloading can keep the active experts on GPU while paging inactive experts from CPU memory, which makes MoE models somewhat tractable on smaller hardware. The throughput is not going to win awards, but if you are running Mixtral 8x7B on a server with two A6000s and 48GB each, the offloading approach can actually work. The full picture of format tradeoffs is in our LLM quantization guide.
The Network Bandwidth Reality
The all-to-all communication in expert parallelism is bandwidth-intensive. For a batch of 1024 tokens routing through DeepSeek V3 with EP=8, each GPU potentially sends tokens to all seven other GPUs. The volume of data per all-to-all call is proportional to batch size, hidden dimension size, and the number of experts selected per token.
Working through the math for DeepSeek V3: hidden dimension 7168, 2 experts selected per token per layer, 61 layers. A batch of 64 sequences at 512 tokens each means roughly 32KB per all-to-all call per layer, multiplied by 61 layers. The bandwidth requirement across your interconnect is in the range of 2TB/s of aggregate all-to-all throughput for sustained inference, which is why InfiniBand HDR (200Gbps per link) with proper fat-tree topology is not optional for large MoE serving.
This changes how you think about infrastructure costs. For a dense 70B model, a couple of A100s on commodity 100GbE networking works fine because the communication is all-reduce with predictable, symmetric patterns. For DeepSeek V3 at production scale, you need high-bandwidth low-latency interconnects, which means you are looking at HPC-class networking gear or managed services that include it. The cost delta is significant.
For smaller MoE models like Mixtral 8x7B (46B total, 12.9B active) or Qwen 3 MoE variants, the all-to-all volumes are manageable on standard 100GbE if you keep EP within a single eight-GPU server on NVLink. The problems only become acute at DeepSeek V3 or Llama 4 Maverick scale, where you are definitely spanning multiple nodes.
From a GPU cost optimization perspective, the right answer for most teams is not to self-host the massive MoE models at all. Together.ai, Fireworks, and Groq have invested heavily in optimized MoE serving infrastructure and achieve unit economics you cannot match with a small cluster. The self-hosted case is strongest when you have latency requirements that mandate your own infrastructure, privacy requirements that prevent sending data to external APIs, or scale that justifies the capital investment in a proper HPC cluster.
When MoE Is Worth It (and When It Is Not)
The value proposition of MoE is clear: equivalent output quality to a larger dense model at lower inference compute cost. Llama 4 Scout at 17B active parameters competes with Llama 3 70B dense on most benchmarks, at roughly 4x higher token throughput for the same GPU count. That is a compelling economics story.
The infrastructure complexity costs are also real. Expert load imbalance can eat your theoretical throughput gains. All-to-all communication adds latency that dense model all-reduce does not. Memory requirements for the full parameter set constrain your deployment options. And debugging a MoE model that is behaving oddly is genuinely harder because you have to consider whether the routing itself is the problem.
My decision tree, built over two years of running these models in production, looks like this. If your inference budget is small (fewer than four GPUs), use a smaller dense model or call an API. If you have four to eight GPUs in a single NVLink server, Mixtral 8x7B or Qwen 3 MoE at reasonable quantization is a good fit: the all-to-all stays local, the memory is manageable, and you get better throughput-per-dollar than a comparable dense model. If you are at sixteen or more GPUs spanning multiple nodes, you are in enterprise MoE territory where you need a real HPC networking topology and dedicated MoE serving expertise. If you hit that scale, the serverless GPU platforms like Modal and RunPod now have MoE-specific endpoints that are worth evaluating before you build a cluster.
The LLM inference infrastructure basics article I wrote a while back assumes dense model architecture. The MoE era requires revisiting all those mental models. Throughput estimations based on “active parameters” underestimate memory pressure. Latency estimations that ignore all-to-all communication underestimate token generation time. And any autoscaling logic that treats GPU utilization as a proxy for model load needs to account for the fact that a MoE model with high memory pressure but 37B active parameters will look different from a dense 37B model on the same metrics.
Hardware Selection for MoE Workloads
The H100 SXM5 with NVLink 4.0 is the reference hardware for MoE in 2026, and not just because of raw compute. NVLink at 900GB/s aggregate bandwidth means all-to-all within an eight-GPU node completes in microseconds. That bandwidth ceiling is what makes EP=8 within a single node viable. The H200 with HBM3e adds memory bandwidth improvement as well, which helps throughput for memory-bound expert layers.
For the cross-node fabric when you have to span servers, Infiniband NDR at 400Gbps per port with good fat-tree topology is the standard choice. RoCE with DCQCN congestion control works but requires careful tuning; the default RoCE settings I have seen in cloud environments are optimized for storage workloads, not all-to-all MoE traffic. Our GPU cluster networking guide has the RoCE tuning parameters worth checking.
AWS Trainium 2 and Google’s Ironwood TPUs both support MoE, but the software ecosystem maturity is substantially behind what vLLM and SGLang offer on NVIDIA hardware. I have run experiments on custom silicon for MoE and the raw efficiency numbers are interesting, but the operational pain of dealing with model compatibility issues and slower optimization cycles is real. The custom AI silicon comparison article covers the tradeoffs in more depth, but my recommendation for most teams in 2026 is to stay on NVIDIA for MoE until the alternative toolchains catch up.
Practical Configuration Examples
For serving Mixtral 8x7B on a single 8xH100 node with vLLM:
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--max-model-len 32768 \
--gpu-memory-utilization 0.85 \
--quantization fp8
The FP8 quantization here reduces the weight memory from about 92GB to around 46GB, leaving substantial room for KV cache. With --gpu-memory-utilization 0.85, you get roughly 400GB of combined KV cache across all eight GPUs, which comfortably supports 500+ concurrent requests at 4k token context.
For DeepSeek V3 across two 8xH100 nodes with pipeline parallelism between nodes and expert parallelism within each node:
vllm serve deepseek-ai/DeepSeek-V3 \
--tensor-parallel-size 8 \
--pipeline-parallel-size 2 \
--enable-expert-parallel \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--quantization awq_marlin
The AWQ Marlin quantization (4-bit) brings DeepSeek V3’s weight memory down from 1.3TB to around 335GB across the sixteen GPUs, leaving room for meaningful KV cache. This configuration will achieve around 80-120 tokens per second per request on typical API workloads, which is competitive with hosted APIs for latency-sensitive applications.
The Operational Reality in 2026
Twenty years of building infrastructure has taught me that the architectures that win are the ones that are not just technically superior but operationally manageable. MoE is genuinely better than dense for inference efficiency, but the operational complexity is real.
Expert load imbalance is not theoretical. I have seen MoE models on certain workloads where 80% of tokens consistently route to the same 20% of experts, turning a theoretically balanced system into a hot-spot problem. Monitoring expert utilization distributions is a new observability requirement that most teams are not doing yet. Add it to your dashboards now, before you hit a production incident and have to figure out why throughput dropped 40% with no obvious cause.
The autoscaling logic you built for dense models needs rethinking. For dense models, GPU memory utilization and token throughput correlate well with load. For MoE, you can have high memory pressure (all experts loaded) with low compute utilization (because few tokens are active), or high compute spikes when popular experts get hammered. Use request queue depth and time-to-first-token as your primary scaling signals rather than GPU metrics.
Finally, MoE model updates are operationally heavier than dense model updates. Swapping a dense 70B model on eight GPUs takes a few minutes. Swapping a 671B MoE model on sixteen GPUs with complex parallelism configuration takes 20-40 minutes including loading time. Blue-green deployment for MoE at scale requires either significant extra hardware capacity or careful traffic draining. Build your deployment tooling with this in mind before you have to do a rollback under pressure.
The Kubernetes-based deployment patterns we cover for general LLM inference infrastructure still apply, but the scheduling decisions get more complex. You need to ensure that all pods for a single MoE model instance land on the same high-bandwidth fabric segment. Cloud Kubernetes schedulers do not do this by default; you need node affinity rules and pod affinity to pin your inference deployment to specific node groups. For AI FinOps on GPU workloads, MoE also complicates cost attribution because you are paying for 671B parameters of memory while only computing with 37B of them. Chargeback models that use “active parameter compute” as a cost basis will systematically undercharge MoE workloads.
MoE is the dominant architecture of 2026 and that trajectory is not reversing. Building the operational muscle to run these models well is not optional work for teams doing serious AI infrastructure. The teams that get ahead of the complexity now, before MoE is everywhere, will have significant advantages as these models become the default choice for every serious capability level.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.