Pretty amazing interaction. Every time I've tried ChatGPT I get two questions in, an error happens and then all subsequent requests are rejected and I'm told to slow down, even if I wait quite a bit.
This is very scary human-like. I was a teaching assistant and asked students to fix issues in their code. They acknowledged these issues but submitted again the same erroneous code.
It’s very clever in how you can tell it that it was wrong and it admits this mistake. Has it been written in such a way that the operators instructions are given a higher weight compared to what it knows?
I’ve asked it to generate python code and then format it with black. It reproduced the original code and then what was supposed to be the formatted code, but it was identical.
When told of this it admitted it had made a mistake and this time correctly outputted the formatted code.
The operator prompts becomes part of the input vector (it has a certain context depth it can accommodate), whereas the data it was trained on actually affects the models weights and biases. You can think of it like “evaluating this statement in light of the prior context against the training set”.
I actually got it to work through debugging its own code by running it and giving it error messages (which reference the line), or describing the thing that wasn't working. I didn't try asking it to add line numbers because I figured it has a tenuous understanding of the relationship between tokens as it is, but it seemed to understand what needed to be fixed from the error messages, the output (which was generated by log commands it added upon my prompting), and a description of the problem. See https://github.com/HartS/gpt-hacker-news-extension
In the situations where it gave me partial output, I copied the last line and said "continue from "...""
Maybe it's just under high demand right now.