Making an AI better has meant making it bigger for about six years now, and it has worked well enough that most people treat it as a rule, but a recent paper from MIT found the point where it stops. Every LLM has to hold more concepts than it has room for, so it stacks them into the same directions and lets them overlap. That overlap is called superposition, and it is the reason these models work as well as they do, though it is also the reason they get things wrong. The paper worked out the exchange rate. Double the width of a model and you halve the errors, which turns out to be most of why bigger models have been better all along.
In this video, I break down what a model’s internal space actually looks like, why holding more concepts than it has dimensions is what makes models wrong in the first place, and the three separate things that stop this strategy from running forever, only one of which is money.