Mixture of experts (MoE) inference: memory, routing, and GPUs

How mixture-of-experts inference works

A sparse mixture-of-experts (MoE) transformer replaces some feedforward layers with several feedforward networks called experts. A learned router chooses a subset for each token. Experts are parts of a layer, rather than separate chatbots answering the whole prompt.

• Score. The router reads a token’s hidden state and assigns scores to the available experts.
• Select. Top-k routing chooses k experts. With top-2 routing and 8 experts, one token uses 2 experts at that layer.
• Compute and combine. The selected experts process the hidden state. Their weighted outputs form the layer’s result.

The choice happens per token and per MoE layer. A request can use different experts as generation continues. Mixtral uses 8 experts and top-2 selection; its architecture paper describes the router and weighted combination.

The word expert does not guarantee a fixed subject specialty. Mixtral’s routing analysis found no obvious assignment by topic in the domains it examined. Do not assume you can retain only a “coding expert” to serve code prompts.

Active parameters estimate work; total parameters determine weight storage

Total parameters count the whole model, including every expert. Active parameters count the parameters used along one token’s selected path, including common layers. A small active count describes sparse computation; it does not tell you how much weight memory to allocate.

For a deployment that keeps weights on GPUs, all expert weights must fit across the serving GPU group. An expert unused by the current token may be selected by the next token or another request. The Transformers Mixtral documentation explicitly distinguishes the memory needed for all experts from the selected computation.

Attention still runs outside the routed feedforward block. Its key-value (KV) cache stores attention state for tokens; it is not multiplied by the expert count. Size it from the model’s attention architecture, context lengths, cache precision, and concurrency. See the KV cache guide.

A worked MoE memory estimate

Consider an illustrative model with 2 billion always-used parameters and 24 billion routed expert parameters. Every sparse layer has 8 equally sized experts and selects 2. The counts are synthetic, with the expert parameters summed across all sparse layers.

Total parameters  = 2B + 24B = 26B
Active parameters = 2B + (2 / 8) × 24B = 8B
BF16 weights      = 26 billion × 2 bytes = 52 GB = 48.4 GiB

Using 8B to budget memory would allocate only 16 GB for weights. The deployment actually needs 52 GB before adding attention state or runtime allocations. BF16 uses 2 bytes per value; GB uses decimal bytes, while GiB uses powers of 1,024.

For this example, also assume standard full-context attention with 32 layers, 8 KV heads, and a head dimension of 128. A BF16 cache needs 2 × 32 × 8 × 128 × 2 = 131,072 bytes per token. The first factor counts keys and values.

At 8,192 cached tokens per request, one request uses 1 GiB. Eight requests use 8 GiB. Adding an illustrative 4 GiB runtime reserve gives 60.4 GiB in total. The reserve is an assumption, not a measured requirement.

GIB = 1024 ** 3
shared_params = 2_000_000_000
routed_params = 24_000_000_000
experts, top_k = 8, 2
total_params = shared_params + routed_params
active_params = shared_params + routed_params * top_k / experts
weight_bytes = total_params * 2  # BF16
kv_per_token = 2 * 32 * 8 * 128 * 2
cache_bytes = kv_per_token * 8192 * 8
reserve_bytes = 4 * GIB  # Illustrative, not profiled

print(f"Total / active parameters: {total_params / 1e9:.0f}B / "
      f"{active_params / 1e9:.0f}B")
print(f"Weights: {weight_bytes / GIB:.1f} GiB")
print(f"KV cache: {cache_bytes / GIB:.1f} GiB")
print(f"Planning total: {(weight_bytes + cache_bytes + reserve_bytes) / GIB:.1f} GiB")

Total / active parameters: 26B / 8B
Weights: 48.4 GiB
KV cache: 8.0 GiB
Planning total: 60.4 GiB

Double the cached tokens and confirm that cache memory doubles while weights stay fixed. This estimates one logical model copy. Distributed layouts can replicate common weights or cache state, and each GPU also needs communication buffers. Check per-device allocations before deciding what fits. GPU memory for LLM inference explains the broader budget.

Why batch routing and expert load balance matter

Top-2 means 2 experts per token, not 2 experts per batch. Across many tokens, a batch may touch every expert. For example, 64 tokens produce 128 expert assignments at one top-2 layer. An even split across 8 experts would give 16 assignments each.

Real routing can be skewed. If one expert receives many more tokens, its device may finish later and delay the combined result. Increasing batch size changes both the amount of work per expert and the distribution of that work.

Inspect token assignments per expert, per-device execution time, and communication time. vLLM’s expert parallel deployment guide documents an Expert Parallel Load Balancer that redistributes expert placement. Redundant expert copies consume additional GPU memory, so include them in capacity planning.

Expert parallelism vs. tensor and data parallelism

TensorRT-LLM’s parallelism guide describes both expert sharding and hybrid layouts that also slice expert weights. DeepEP provides all-to-all kernels for expert dispatch and combination. Network topology and transfer time belong in the benchmark.

Data parallel attention combined with distributed experts differs from independent full-model replicas. vLLM’s data parallel deployment guide explains that MoE ranks sharing expert layers must synchronize forward passes, including ranks without scheduled requests.

Record the attention layout, expert layout, expert replicas, and GPU placement together. Tensor parallelism vs. pipeline parallelism covers the surrounding deployment choices.

How to compare MoE and dense inference

Choose models that meet the same quality threshold on your tasks. Equal active parameter counts do not establish equal quality. Then define a service-level objective (SLO), such as a limit on the 95th-percentile time to first token and inter-token latency.

• Fix the workload. Use the same requests, arrival pattern, output constraints, and sampling policy. Record each model’s token counts because tokenizers can differ.
• Record the deployment. Save model revision, precision, engine version, GPU count, topology, and parallelism layout. Verify memory at the intended context and concurrency.
• Sweep traffic. Test low concurrency, sustained load, and bursts. Separate prompt processing from token generation and check long prompts as well as short chat.
• Compare useful work. Measure request throughput and cost per successful task while meeting quality and latency targets. Include every GPU and any expert replicas in the cost.

Use LLM inference metrics to define timing boundaries. The inference optimization guide gives a broader experiment order. Continue with the vLLM course or SGLang course for serving practice.

Mixture-of-experts inference FAQ and common mistakes

Does an 8B-active MoE need the memory of an 8B dense model? No. Budget all resident weights, KV cache, and runtime allocations. An out-of-memory error at startup often means the active count was mistaken for total storage.

Is MoE always faster or cheaper than a dense model? No. Sparse expert computation must compensate for routing, memory access, and communication costs on your workload. Compare models that meet the same quality and latency requirements.

Does adding GPUs automatically improve MoE inference? No. More devices change placement and communication. If throughput rises but token streaming slows, compare inter-token latency and transfer time before keeping the wider layout.

Why are some GPUs busy while others wait? Skewed expert routing is one possibility. Inspect per-expert token counts and device timings, then evaluate supported load balancing. Also check attention work, communication, and request scheduling before assigning a cause.