Compare MoE and SLM serving under per-class quality and latency constraints

Compare model-serving configurations only after fixing quality and latency requirements for each request class. Output-space size, answer length, and difficulty of producing a correct answer are different: a binary decision can be difficult, while copying varied text can be easy. These distinctions apply to large sparse MoE versus small dense models, but architecture alone predicts neither quality nor fleet efficiency.

Measure three quantities separately: output tokens per GPU-second at a stated workload; maximum sustainable request rate satisfying latency targets; and useful completions per provisioned GPU-second at expected demand. Include prefill, draft-model and fallback allocations. Aggregate accuracy can hide failure on an important rare class. A low fallback rate can also hide incorrect answers accepted by the small model.

For MoE, per-token sparsity does not equal per-batch sparsity. Broader expert coverage can increase aggregate weight traffic while improving parallel bandwidth use and device balance; local expert batch sizes and communication determine the result. Semantic task categories are not reliable proxies for internal expert routing.

Evidence/example — 2026-09-20: An arithmetic illustration with 90% of requests producing 20 tokens and 10% producing 2,000 gives the long-output group 91.7% of generated tokens. This is not a GPU-time estimate or difficulty measure. Primary sources inspected: Mixtral routing analysis (https://arxiv.org/html/2401.04088v1), DistServe's SLO-constrained goodput definition (https://arxiv.org/abs/2401.09670), and NVIDIA's configuration-specific report of improved per-GPU throughput with wider expert parallelism (https://developer.nvidia.com/blog/scaling-large-moe-models-with-wide-expert-parallelism-on-nvl72-rack-scale-systems/). This synthesis is conceptual, not a matched MoE-versus-SLM benchmark.