The other day I made a change to one of our iOS builds such that the ipa (a zip file) now contained a subdirectory called "Symbols/". I wanted to know how much space the compressed symbols grew the overall ipa by. I immediately typed this on the command line:
I work with developers who are not CLI-fluent and wondered what they'd do. And then I realized, they might ask ChatGPT. So I asked it: "How do I figure out how much space a subdirectory of files takes up within a zip file?"
It took a lot of prompting to guide it to the correct answer:
The weird part to me is how sure it is of itself even when it's completely wrong. (Not too different from humans I work with, self included, but still...) e.g.
> "The reason I moved the subdir filtering to 'awk' is that the 'zipinfo' command doesn't support filtering by subdirectory like the 'unzip' command does."
Where does it come up with that assertion and why is it so sure? Then you just tell it it's wrong and it apologizes and moves on.
Now, the addendum here is that if either of us had read the zipinfo man page more closely, we would've realized it's even simpler:
zipinfo -lt foo.ipa "Symbols/*" | tail -1
---
I also tried to use it to translate about 10 lines of Ruby that was just mapping a bit of json from one shape to another to jq. It really went off into the weeds on that one and initially told me it was impossible because the Ruby was opening a file and jq has no I/O. So I told it to ignore that part and assume stdin and stdout and it got closer, but still came up with something pretty weird. The core of the jq was pretty close, but for example, it did this to iterate over an object's values:
to_entries | .value.foo
Instead of just using the object value iterator:
[].foo
---
One of my pet peeves is developers who never bother to learn idiomatic code for a particular language, and both of these turn out to be that sort of thing. I guess it's no worse than folks who copy/paste from StackOverflow, but I kind of hoped ChatGPT would generate things that are closer to idiomatic and more based on reference documentation but I haven't seen that to be the case yet.
---
Has anyone here tried using it for code reviews instead of writing code in the first place? How does it do with that?
I think part of the problem is that they've wrapped all the responses in a positive language framing to make it feel more human.
It's always "Sure thing!" or "Yeah I can do that!", and it responds so quickly that you equate it to when a human responds instantly with a strong affirmative - you assume they're really confident and you're getting the right result.
If it took 5-10s on each query, had no positive framing and provided some sort of confidence score for each answer, I think we'd have different expectations of what it can do.
I am not familiar with these command line tools so I would probably just compare the file sizes of the 2 artefacts, or if there are other differences in the artefacts and I has to hone in, just open it up in 7z GUI and take a look.
As for ChatGPT i have jokingly thought to always tell ChatGPT “that didn’t work” until it settles on something it is really confident on (no more alternatives) then try that.
Note that this is GPT-3.5. GPT-4 solves this problem correctly with Python, at least, although it still struggles with the shell. I think that's a function of the training data - it is known that Python is the most represented language in the training set.
It took a lot of prompting to guide it to the correct answer:
https://imgur.com/a/kAaMova
The weird part to me is how sure it is of itself even when it's completely wrong. (Not too different from humans I work with, self included, but still...) e.g.
> "The reason I moved the subdir filtering to 'awk' is that the 'zipinfo' command doesn't support filtering by subdirectory like the 'unzip' command does."
Where does it come up with that assertion and why is it so sure? Then you just tell it it's wrong and it apologizes and moves on.
Now, the addendum here is that if either of us had read the zipinfo man page more closely, we would've realized it's even simpler:
---I also tried to use it to translate about 10 lines of Ruby that was just mapping a bit of json from one shape to another to jq. It really went off into the weeds on that one and initially told me it was impossible because the Ruby was opening a file and jq has no I/O. So I told it to ignore that part and assume stdin and stdout and it got closer, but still came up with something pretty weird. The core of the jq was pretty close, but for example, it did this to iterate over an object's values:
Instead of just using the object value iterator: ---One of my pet peeves is developers who never bother to learn idiomatic code for a particular language, and both of these turn out to be that sort of thing. I guess it's no worse than folks who copy/paste from StackOverflow, but I kind of hoped ChatGPT would generate things that are closer to idiomatic and more based on reference documentation but I haven't seen that to be the case yet.
---
Has anyone here tried using it for code reviews instead of writing code in the first place? How does it do with that?