But it's much worse than that for ChatGPT. A person would know that their mental math division of large numbers is unreliable, but if you ask ChatGPT about the reliability of its answers, it is uniformly confident about almost everything. It has no grasp of the difference between the things it "knows" which are true and the things it "knows" which may be false.
And this is the error I notice people very consistently making when they evaluate the intelligence of ChatGPT and similar models. They marvel at its ability to produce impressive truths, but think nothing of its complete inability to distinguish these truths from similar-sounding falsehoods.
This is another form of the rose-colored glasses, the confirmation bias we are seeing at the peak of the current hype cycle, reminiscent of when Blake Lemoine convinced himself that LaMDA was sentient. A decade ago, techies were dazzling executives with ML models that detected fraud or whatever as if by magic. But then, when the dazzling demos were tempered by the brutal objectivity and rigor of the precision-recall curve, a lot of these models didn't end up getting used in practice. Something similar will happen with ChatGPT. People will eventually have to admit what it is failing to do, and only then will we start up Gartner's fabled Slope of Enlightenment to the Plateau of Productivity.
The failures don't refute the successes. Anything and anyone can fail, you don't get intelligent output by chance. If it makes mistakes we wouldn't make, on obvious things, it is because it is an alien form of intellingece.
RLHF and the tokenizer together explain many of the more common failure modes.
Nobody is saying the intelligent output is by chance. This is a machine that is fed terabytes of intelligent inputs and is able to produce intelligent outputs. So one explanation of its producing intelligent outputs is that it's basically regurgitating what it was fed.
The way to test that, of course, is to give it problems that it hasn't seen. Unfortunately, because GPT has seen so much, giving it problems it definitely hasn't seen is itself now a hard problem. What's been shown so far: OpenAI's benchmarking is not always rigorous or appropriate, and GPT's performance is brittle and highly sensitive to the problem phrasing [1].
I agree with the article that GPT's training enables it to access meaningful conceptual abstractions. But there is clearly quite a lot that's still missing. For now, people are too excited to care. But when they try to deliver on their promises, the gaps will still be there. Hopefully at that point we will embark towards a new level of understanding.
It's not that hard to give it problems it hasn't seen - you can take a classic description of a logical thinking exercise the text for which does occur online, then mix it up in ways that doesn't change the underlying pattern of reasoning necessary to solve it, and at least from the tests I've done it will confidently tell you the incorrect answer (along with some semi-plausible but fatally flawed description of the reasoning it used to come up with the answer). In at least one case it was certain the answer was exactly the answer given in the common online statement of the problem, despite the fact I'd changed all the inputs such that the given answer was obviously not an option (it was a guess so-and-so's birthday type problem).
Even for simple arithmetic just choosing sufficiently large numbers will bring it unstuck.
If a human mathematician said the things ChatGPT said in this dialogue, you would wonder if the person had recently suffered a severe traumatic brain injury.
This is a failure mode of people as well. Rewriting so it doesn't bias common priors or at least in the case of Bing, telling it it's making a wrong assumption works.
Sure, but the system should then be able to show its working and explain how it derived the incorrect result. If there is other evidence that ChatGPT 'understands' the concept of a prime number, then let's see it.
If wrong answers still count because humans sometimes make mistakes, then I guess it won’t be too difficult to construct an impressive mathematical AI.
It's very tempting to give these systems the benefit of the doubt, but that tends to lead to hugely inflated conclusions about their capabilities. Remember that something as simple as ELIZA was perfectly capable of fooling humans who were predisposed to believe it was intelligent.
> the system should then be able to show its working
The system fundamentally cannot do this. You can make it generate text that is like what someone would say when asked to show their working, but that's a different thing.
> It's very tempting to give these systems the benefit of the doubt, but that tends to lead to hugely inflated conclusions about their capabilities.
I agree. I am seeing a bit too much over-optimistic predictions about these things. And many of these predictions are stated as fact.
> then I guess it won’t be too difficult to construct an impressive mathematical AI.
Yes, we've had Wolfram Alpha for ages. For me, the biggest problem Wolfram Alpha is that it often doesn't understand the questions, and while I also sometimes get that with ChatGPT, the latter is much much better.
I've not had a chance to play with the plug-in that connects GPT to Wolfram Alpha.
> It's very tempting to give these systems the benefit of the doubt, but that tends to lead to hugely inflated conclusions about their capabilities.
This is an excellent and important point.
I think people who treat it as already being superhuman in the depth (not merely breadth) of each skill are nearly as wrong as those who treat it as merely a souped-up autocomplete.
I've only played with 3 and 3.5 so far, not 4, my impression is that it's somewhere between "a noob" and "genius with Alzheimer's".
A noob at everything at the same time, which is weird because asking a human in English to write JavaScript with all the comments in French and following that up with a request written in German for a description of Cartesian dualism written in Chinese is not something that any human would be expected to do well at, but it can do moderately well at.
Edit: I should probably explicitly add that by my usage of "noob", most people aren't even that good at most things. I might be able to say 你好 if you are willing to overlook a pronunciation so poor it could easily be 尼好 or 拟好 (both of which I only found out about by asking ChatGPT), so I'm sub-noob at Chinese.
This is a good point. If you ask GPT about much more conceptual advanced mathematics it's actually very good at conversing about this. That said, it does 'hallucinate' falsehoods and it will stick with them once they have been said. etc. You have to double check everything it says if you are on unknown ground with it.
Right, over the totality of things that it reasons about, to some degree it will make inroads to correctly answering these kinds of questions, and in some ways it'll make errors, and what's interesting, is it'll make errors because the way in which it's attempting to answer them bears a lot of the hallmarks that we associate with conceptual understanding, rather than the mechanical operations of a calculator which truly is blind but always correct. In a way, being wrong can be a better signal of something approximating understanding under the hood. It's like if it was shooting a basketball, and it takes numerous shots, and most of them go in but some of them go out, but even the ones that miss bear the hallmarks of proper shooting form that lead to correct answers.
I do think that this specific moment we're going through, in early 2023, is producing some of the most fascinating, confidently incorrect misunderstandings of chatgpt, and I hope that someone is going through these comment sections and collecting them so that we can remark on them 5 to 10 years down the line. I suspect that in time these misunderstandings are going to be comparable to how boomers misunderstand computers and the internet.
I recently asked ChatGPT to write me some Go code. What it wrote was mostly fine, but it incorrectly used a method invocation on an object, instead of the correct code which was to pass the object to a function at the package level (aka static method).
I think it's a real stretch to suggest that this could happen as a result of simply regurgitating strings it's seen before, because the string it emitted was not ever going to work. To me, it looked for all the world like the same kind of conceptual error that I have made in Go, and the only way I could see this working is if GPT had a (slightly faulty) model of the Go language, or of the particular (and lesser known) package I was asking about.
It's felt more like it "forgot" - like me - how the package worked, so it followed its model instead. That error was WAY more interesting than the code.
Just on the point of forgetting, to pass the time while bored at work I told Chad GPT about my fantasy baseball team, and over the course of the conversation it forgot who my first baseman was but input the name of an entirely different first baseman who I had never mentioned, but who is a real person and who in fact was similar in a lot of ways to my actual first baseman. And sometimes when it would attempt to recall my lineup it would seem to 'forget' certain players even if it remembered them later.
