I think this leads to an interesting question. What happens when the money runs out?
Right now, a lot of money is going to train new models. And we need to train new models because they get gated by their training data. And models are only as useful as their training data.
So let's say the money stops.
Do we stop training models? Do we train them slowly? Do we accept the then current models as the limit?
The money is never going to stop. Itβs basic economics.
Well, the money will stop when the value of problems the LLM can solve is not increased by adding compute. Since current LLMs are getting quite good at solving problems, that might be a while.
If you were in 1999 you'd be saying pets.com = internet.