My experience was different than yours. I had an existential crisis, and it only got worse the more I used it. To be clear, I did not think ChatGPT was ready to replace me now (and, mind you, I only used the 3.5 version of ChatGPT). But it was so easy for me to see how in just a few years it can accelerate its pace of learning.
I was discussing a bug with a colleague, so for curiosity's sake I decided to plug a similar question into ChatGPT. I was quite impressed with the solution it gave, and interestingly, it had the same subtle bug that our code had. What blew me away is that when I pointed out the bug, ChatGPT fixed the code by itself. On one hand I felt "phew, at least it needed me to point out the bug", but then I thought "I just (perhaps stupidly) provided training data so that down the road ChatGPT would get it right the first time."
You can also just ask it to check if its result matches the spec or check it for bugs. I've done that and had it find things without me telling it what was wrong.
I don't see how people can see stuff like that and say "oh, it's just a fancy Markov chain generator" or "it can't reason". Even if that stuff is nominally true, how can people not be totally blown away by this? Just a couple years ago I think people would have been amazed that it can have totally natural, grammatically correct conversations. Moreover, for nearly 3/4 of a century the scientific community has pretty much coalesced on only using the output to define intelligence (aka the Turing Test). While I understand that ChatGPT may not 100% be there yet, I see no reason to believe that all this interaction people are having with it won't be fed back into it to drastically improve its responses over time.
> Moreover, for nearly 3/4 of a century the scientific community has pretty much coalesced on only using the output to define intelligence (aka the Turing Test).
It's more like the scientific community has spent the last 50+ years criticizing the Turing Test. Passing as a human is a nice engineering goal, but there has been a lot of doubt of using input/output behavior as the only measure of intelligence. If you took a basic AI class before machine learning became popular, the chances are the class spent more time on the criticism than on the test itself.
I watched the video. Chat GPT is impressive, but exactly as the person creating the video mentioned, the deeper you get into the project, the easier it is for Chat GPT to get confused. And Chat GPT has no idea it's confused. You have to point out each individual error, and the Chat GPT scans for it, makes a correction, and potentially introduces more errors. I don't see how you could use it with any confidence on a project with sufficient complexity.
EDIT
And here is something else that just struck me. The errors in the code are obvious the minute you run the program, particularly in the beginning when the population dies off after day 1. Chat GPT is apparently incapable of running the simulation to check if its code actually makes sense. It needs someone to tell it. Think about that for a second. Can you imagine a fellow programmer handing you a piece of code without bothering to try to run it first?
I can see Chat GPT as an advanced assistant who can save a programmer a LOT of time right now, but definitely nothing more than that.
The thing is, that's a capability which is likely to get added into AI harnesses very soon in the future. Look at the difference between Bing Chat and ChatGPT. Now consider when we have not a 32k context length model, but a 1m CL model, or even 1g CL, and that model can search the web for information, act on it, compile and test the code, diagnose the errors by again searching the web for information, etc. ChatGPT is the simplest possible use of the model, with nothing but user input -> model output. The thing is, tests have already been done with having these models act as agents, and they perform exceedingly well when appropriately instructed. It's not about right now. It's about 1 year from now, or maybe even merely 6 months from now.
I understand the expectation, but I am definitely in the "I'll see it when I believe it camp." Which is not to say that I am not impressed by what it can do now. I am. There is just no way I would trust it without first thoroughly examining the code it produced.
While I agree with you about the status of ChatGPT right now, it seems pretty obvious to me that all of the problems you state will be easily surmountable in time, and I've just been shocked at the pace of advancements in AI in the past couple years. I think it's totally plausible that the AI community hits a "brick wall" eventually in some number of years with regards to their pursuit of AGI with LLMs, but the problems you point out seem to me exactly the type of things LLMs can solve with more training data/parameters/dimensions, and added functionality to test code.
Also, I see lots of comments downplaying the potential impacts of LLMs because they can hallucinate or they have errors, and though I agree with all these points, I also want to shout "Gang, we're having a fully natural, back-and-forth conversation, with a computer. It speaks English and French and TypeScript!" This is stuff that seemed fully planted in the realm of science fiction only a decade or so ago. For example, I'm not sure I've ever seen ChatGPT make a grammatical mistake, or even generate code that can't compile (though I have seen ChatGPT "crap out" if the program it's writing gets too long).
I think I'm naturally a pretty skeptical person when it comes to tech hype, to the point of usually being over-conservative about potential impacts (I drastically underestimated the impact mobile would have on society, for example). But with ChatGPT, I feel like I need to take breaks just from the constant mind-blowing nature of it.
Is it not possible to write a script which repeatedly attempts to run the code, then sends the errors back to GPT to be fixed? Then it changes the code based on GPT’s requests and runs it again, refining it until some test condition is met
I was discussing a bug with a colleague, so for curiosity's sake I decided to plug a similar question into ChatGPT. I was quite impressed with the solution it gave, and interestingly, it had the same subtle bug that our code had. What blew me away is that when I pointed out the bug, ChatGPT fixed the code by itself. On one hand I felt "phew, at least it needed me to point out the bug", but then I thought "I just (perhaps stupidly) provided training data so that down the road ChatGPT would get it right the first time."