MoE routing · Debug Foundry

What it's for

A dense model spends its whole brain on every token. MoE grows the brain without growing the bill: many experts, but each token wakes only a few.
Experts specialize as a side effect of training — the router learns to send similar tokens to the same few, and they get good at that niche.

Good to know

The router is a tiny learned layer: one score per expert, softmax into shares, then a top-k gate — the same scores-to-shares move as decoding, one chapter back. Only the k winners run; their outputs combine weighted by those shares. Training adds a load-balancing loss so tokens spread across experts instead of crowding the same one, and real systems cap how many tokens each expert accepts per batch (its capacity) and drop or reroute the overflow. Six experts with toy specialties and hand-set shares here; production models route over far more — 8 in Mixtral 8x7B, 128 per layer in some Switch variants. The combine weights shown are the router's full-softmax shares over all six experts (52% + 31%, the rest scattered over the dark four); some real models renormalize them over just the winners. A real token's route depends on its layer and context, not just the word itself.

How it works, step by step

Why the steps go in this order

The dense block is the baseline cost: every token pays all of it, every time.
Splitting it into experts only helps if something can choose — so the router comes next.
Scores become shares through softmax, so they can both rank and later weigh.
The top-2 gate is where the savings happen: four experts stay dark and their compute is simply not spent.
Weighted combining keeps the router's confidence in the answer, not a coin flip between experts.

Before you rely on it

This is a toy: six experts, fixed toy specialties, hand-set router shares, one token at a time. "A third of the FFN work" is this toy's arithmetic (2 of 6), not a benchmark — real savings depend on k, expert count, capacity limits and routing overhead, and the cited papers report their own measured results. Two lit experts is not two-sixths of the model's total cost: attention, the router and dispatch still run for every token, and every expert's weights still need to be stored and reachable. Expert specialties in real models are learned, never labeled, and a token's route can change between model versions. And the dense block splitting into experts is a teaching metaphor — MoE models are trained sparse, with their router, from the start; nobody converts a dense model into one.

References