Ask HN: Is ChatGPT just kinda shit for programming help?

[jascination]

I've tried using ChatGPT for help with a React Native (JS) project, and it continually seems to just make up answers and confidently display them as correct.

For instance, I'm trying to use AWS Amplify's Auth service to authenticate users, and when I sign them out it's trying to auto-log them in again.

This is a known bug and there are a few github issues about it. ChatGPT confidently tells me that the way to solve this problem is to use some methods which, when I try them, don't exist at all on Amplify Auth, and when I search for them there's no record of them whatsoever.

Similarly I tried to get its help in making an animation using the RN Animated API, and its suggestions were all from React Native Reanimated (a different project), although the code sample imported the Animated API from RN instead. When I corrected it, it gave some suggestions with methods that didn't exist for either.

These are just a few examples but I'm finding it fairly useless for coding help so far. Am I expecting too much from it? Am I using it wrong?

[skilled]

It's good for simple things, but nothing else. If you gave someone ChatGPT who never did programming before, that person would be stuck in a limbo for 10 hours trying to get a simple program to work because that's how the model is.

In many ways it is a glorified copy/paste machine that's faster (in some cases) than using traditional search. And for bigger projects you're going to get obliterated because at some point the model will either forget the conversation or it will start giving you some random suggestions, all of which lead to frustration and lost time.

But for small projects and snippets, I definitely use it a lot. JavaScript functions, CSS boilerplates and stuff like that - even if it's 50 lines long, I can just do some other stuff while it generates the basic outline and I am on my way.

[mindcrime → skilled]

In many ways it is a glorified copy/paste machine that's faster (in some cases) than using traditional search.

That sounds like the most apt description I've heard yet. I have a feeling that's pretty close to spot on. I don't think ChatGPT is going to do much that's very creative or elegant in and of itself, but it can find and c&p a useful snippet for you very quickly.

Maybe the biggest advantage of this over traditional c&p though, is that it does have at least some memory and ability to revise the generated code in a "conversational" fashion. IME (which to be fair is limited, see my other comment) that actually does work, at least to a point.

[mindcrime]

Am I expecting too much from it?

Hard to say.

Am I using it wrong?

Maybe? I think we're all still trying to figure out the best way(s) to use this stuff right now.

Just for the sake of illustration, here's a recounting of my recent experience using ChatGPT for code:

I've only tried it once on a coding project of any substance, but it did work well enough that it definitely saved me a meaningful amount of time. Probably on the order of an hour or two.

The project was concerned with transferring a bunch of my Github repos en-masse from one organization to another, using the REST API. I could have code that up mostly in my sleep, but I decided to try ChatGPT as an experiment. My initial prompt was something like:

"Write me some Java code to transfer a Github repo from one organization to another, using the REST API. Please use Apache HttpClient."

The code I got was pretty close to what I needed, although I think I did have to prompt it to tweak the code once or twice. One was asking it to add code to use a Personal Access Token as an Auth header.

So in the end, it generated most of the code I needed, and it represented a meaningful time savings, just based on the time it would have taken me to type the code in, plus I'd have had to consult the GitHub REST API docs at least once or twice to lookup endpoints, parameters, etc.

Can it do much more than something like that? I have no idea. But for dealing with fairly generic, boiler-plate'ish code like that, I find it useful enough that I'll probably continue to experiment with it.

YMMV, of course.

[PaulHoule]

(1) Even the advocates say that ChatGPT needs a lot of reprompting and the code needs a lot of editing.

(2) Huge amounts of fraud and misconduct have been widespread in the text generation field, in the RNN era I'd be trying to train a model to write fake clinical reports and get results like

   "A 57-year old male was admitted to the hospital for a heart attack attack attack attack attack attack.."

and found out that the people who were showing good results on their web pages were tweaking their models hard in irreproducible ways, not least generating 100's of examples and only showing the best.

[version_five]

It's funny how quickly opinion shifts. Read through articles like https://news.ycombinator.com/item?id=35469908 from last week, the second comment compares the importance of "AI" for programming to the invention of the GUI

[mindcrime → version_five]

To be fair, some of that might be based on extrapolation, eg people thinking "If GPT 3.5 / GPT 4 can do this today, what will GPT 17 be able to do in 10 years?" and that kind of thing. And that's an interesting question to think about, IMO.

[InfamousRece]

I use GPT to generate silly poems for my kid. It works great for this purpose. But for programming it feels like interacting with a very obtuse human. I find it emotionally draining. Yes it can write a Python function to compute edit distance, so maybe I can use it for coding interviews?

[youshouldsayit]

I kind of agree. I find that GitHub Copilot is more useful, especially since in the JetBrains plugin you can list various autocompletion alternatives

[ssd532 → youshouldsayit]

I like ChatGPT more because of its conversationial interface. I can talk to it and refine the code, it can suggest alternatives, I can fix its mistakes, and so on -- we both help each other and develop the solution. This kind of two way communication is not possible with Copilot.