I've noticed if I give chat GPT an algorithm it gives me consistent results when it shows it's work but if I ask it not to show it's work and just give me the answer. It consistently gives me incorrect answers even when I ask it to follow an algorithm that I gave it.
I suppose this is similar to humans and probably why my school teachers always told me to show my work, but I'm curious if this has been documented and if there are any explanations for why it works this way with LLMs.
By their very nature they only "know" what they have written down and must infer the final answer from that token by token.
They fundamentally can't do certain things such as complex iteration or track back.
When you ask for chain of thought thinking, you allow the LLM to create a "buffer space" and break down the task into more manageable substeps thereby improving the quality of the results.
The Bing LM, or rather the service, did have "inner monologue" in the sense of text that it would generate, but not show to the user, and treat as "thoughts" to guide the generation of an actual reply that the user would see.
We know this because it happily told us, including the json format it uses internally.
No, but the reconstructed examples have "im_start" and "im_end", which strongly implies that it is, if not verbatim, then a close enough restatement of the real deal. Take a look:
First you wrap the user query with "the user asked you: ... . What are the reasoning steps you need?" and then you prompt with "considering `<previous answer>` now answer <user prompt>"
Obviously this is clearly hackable so it would need improvements.
GPT is auto regressive. That means each output token becomes part of the new input sequence. Which is to say, the beginning of the model’s answer becomes part of your prompt.
If the model makes some mistake in the beginning, it now needs to explain / make sense of that mistake.
Kind of like a split-brain patient whom you ask why they got up, and they then say, to get a Coke. [1] In psychology, that is called confabulation. In machine learning, they use “hallucination“, probably so they can use the term across several disciplines, like language, audio, vision, etc.
The psychology of split-brain is a nice gateway to 'The Bicameral Mind', a major theme about how the conception of the AI's works in the series Westworld. Nice!
I just watched a video where the guy touches on GPT-4 limitations and one of those is simple math. He asks it some order of operations question and it outputs the correct answer but only if it does it step by step. It then apologizes and says its original incorrect answer was “a typo.”
I suppose this is similar to humans and probably why my school teachers always told me to show my work, but I'm curious if this has been documented and if there are any explanations for why it works this way with LLMs.