> ChatGPT, if asked correctly, will instead generate one perfect document based on millions of the documents
This is the "self-driving cars will obviously be better than human drivers given that they're trained on millions of humans' driving behavior" of chatbots.
In my experience, this is a huge stretch and wishful thinking.
I mean, it happens in stages - I’m already using chatgpt for plenty of useful tasks that I would otherwise google (and then some). Similarly self driving cars are better than humans at plenty of specific things (collision detection/aversion for instance) but not a wholesale replacement of humans just yet obviously.
I was willing to give ChatGPT the benefit of the doubt after being initially impressed with its results, but I've also had it confidently tell me blatantly wrong information when the correct information is a Google search away at most or no human would confidently make such a mistake.
What makes it worse is that it spits out paragraphs confidently, where some of it might be right or close to right, but the rest might be subtly incorrect. Without domain knowledge, you might not pick up on it. Hell, you might even be convinced of the false information based on the arguments for it that the AI writes out.
At least with search engines you can cross reference multiple sources if you aren't confident in your ability to assess the first thing you read.
Self-driving cars are much more difficult than one might think because the AI has to interface with the real world, something that humans have evolved for over millions of years.
Like others have said, worry more about every job that primarily interfaces with a computer.
Computers don't exist in a vacuum, what goes on in the computer is often determined by interfacing with the real world, reflecting real world intentions, observations, input, predictions, etc.
This generation of AI is great when the universe of values can be generalized. When the universe of values is literally the universe, it shows its limitations.
This is the "self-driving cars will obviously be better than human drivers given that they're trained on millions of humans' driving behavior" of chatbots.
In my experience, this is a huge stretch and wishful thinking.