You don't need more data when the data you have characterizes a problem well. More data is simply redundant and resource wasting. In this case, talking like people about things people talk about is covered well by current data sets. Saying we can't get more data is really saying we have collected at least enough data. Probably more than we need.
Lots of room to improve models though:
Using convolution for vision learning didn't create/require more data than training fully connected matrices. And it considerably increased models efficiency and effectiveness on the same amount of data. Or less.
Likewise, transformers have a limited window of response. Better architectures with open ended windows will be able to do much more. Likely more efficiently and effectively. Without any more data. Maybe with less.
Maybe in a few decades we will reach a wall of optimal models. At the rate models are improving now that doesn't appear to be anytime close.
Finally, once we start challenging models to perform tasks we can't, they will start getting data directly from reality. What works, what doesn't. Just as we have done. The original source of our knowledge wasn't an infinite loop of other people talking back to the beginning of time.
I believe this is part of the argument in the post - the "architecture" of the nervous system (and the organism it is an inseparable part of) is itself largely a product of evolution. Its already optimized to deal with the challenges the organism needs to survive/reproduce, and depending on the organism, with little or even no data.
I wonder if we'll reach the physical nanostructure wall of silicon long before that, and then all progress will have to be algorithmic efficiency gains. The era of Metal Muscle will end and we will return to the era of smart people pondering in coffee shops.
Even if transistors reach physical limits, there’s always different materials and architecture optimizations. We also know the human brain has far more intelligence per watt than any transistor architecture I know of. The real question is if those will be commercially worth researching.
You don't need more data when the data you have characterizes a problem well. More data is simply redundant and resource wasting. In this case, talking like people about things people talk about is covered well by current data sets. Saying we can't get more data is really saying we have collected at least enough data. Probably more than we need.
Lots of room to improve models though:
Using convolution for vision learning didn't create/require more data than training fully connected matrices. And it considerably increased models efficiency and effectiveness on the same amount of data. Or less.
Likewise, transformers have a limited window of response. Better architectures with open ended windows will be able to do much more. Likely more efficiently and effectively. Without any more data. Maybe with less.
Maybe in a few decades we will reach a wall of optimal models. At the rate models are improving now that doesn't appear to be anytime close.
Finally, once we start challenging models to perform tasks we can't, they will start getting data directly from reality. What works, what doesn't. Just as we have done. The original source of our knowledge wasn't an infinite loop of other people talking back to the beginning of time.