DeepMind's PEER scales language models with millions of tiny experts

Parameter-Efficient Expert Retrieval (PEER) is a technique that allows LLMs to scale to millions of experts and remain resource efficient.