> I believe that’s post-training, not pre-training
Aren't AI companies already butting against the limits of usable training data?
Outside of undiscovered techniques, additional costly pre-training will likely yield diminishing returns if you've already scraped the whole internet and destructively scanned as many 20th century books as you could lay your hands on.
most training data is not of that nature, that's "pretraining" input, it's now much more comprised of agent traces in mid/late phases, which are expanded each generation as we push what we do today with the last generation into the model itself, expect more team training in the next round
Aren't AI companies already butting against the limits of usable training data?
Outside of undiscovered techniques, additional costly pre-training will likely yield diminishing returns if you've already scraped the whole internet and destructively scanned as many 20th century books as you could lay your hands on.