The distinction is sparse activation. A dense transformer architecture runs every layer per token. An MoE layer has many expert feed-forward networks and a router that chooses a few of them. It can have a large parameter count while using fewer active parameters per step.
How does mixture of experts work?
The router scores available experts for a token, selects the highest scoring few, and combines their outputs. The routing decision happens token by token, not once per request. Training must balance traffic. If one expert receives most tokens, it becomes a bottleneck while the others learn too little. Inference servers must account for that uneven traffic, particularly when batching requests.
What does MoE change for a buyer or builder?
It changes the capacity and serving trade, not the need to test the model on the task. More total parameters do not mean every request receives more computation. MoE changes the cost profile, not the requirement to measure task quality. Routing, communication between devices, and memory layout can make an MoE model harder to serve than a similarly active dense model. Model routing is a separate application decision that chooses among whole models, while MoE routing happens inside one model.