For a few years the recipe for a better language model seemed to be simply this: make it bigger. Model sizes ballooned into the hundreds of billions of parameters. In 2022 a DeepMind paper, "Training Compute-Optimal Large Language Models," argued that this had been a mistake. Given a fixed budget of computation, the field had been building models that were too large and feeding them too little data, leaving performance on the table. To prove it, the team trained a 70-billion-parameter model called Chinchilla that outperformed models several times its size.