Router Domain Maps
Gemma 4 26B-A4B · 30 layers × 128 experts · expert packs

Where the router sends each domain

Code, data and results on GitHub →

Every subject has a routing fingerprint: the router sends its text to a consistent set of experts. Keep the routers, drop the experts a subject never reaches, and Gemma 4 26B becomes a smaller model that still runs in stock llama.cpp: 7.4 GB at 64 of 128 experts per layer, where the full model squeezed to 10.6 GB no longer works.

The router sits at the centre. Each ring going outward is one of the 30 layers, holding its 128 experts in numbered slots. Bolts trace the experts a subject's text is routed to, layer by layer: its routing fingerprint. Compare what the full model uses with what a pack keeps, for one subject or up to three sharing one pack.

—

View

Pack size, experts per layer

Trade-off at 64 experts per layer

—weights in memory
—experts kept per layer
—vs the full 26B
—of this subject's routing lands on kept experts
—loss on held-out text

    Pack experts in common with

    Domains · select up to three for a shared pack ·