Fascinating. They gave various reasoning tasks to GPT-3. The model was helped by adding in the question the phrase 'Let's think step by step'. As a result performance increased to above human level on 17 of 23 tasks.
For example, this was one the questions
Q: If you follow these instructions, do you return to the starting
point? Turn left. Turn right. Take 5 steps. Take 4 steps. Turn
around. Take 9 steps.
Options:
- Yes
- No
A: Let's think step by step
And this was the model output:
We start at the origin (0, 0), facing the positive y-axis.
(1) Turn left: (0, 0), facing the negative x-axis.
(2) Turn right: (0, 0), facing the positive y-axis.
(3) Take 5 steps: (0, 5), facing the positive y-axis.
(4) Take 4 steps: (0, 9), facing the positive y-axis.
(5) Turn around: (0, 9), facing the negative y-axis.
(6) Take 9 steps: (0, 0), facing the negative y-axis.
Since (0, 0) is (0, 0), we are indeed where we started. So the
answer is Yes.
For example, this was one the questions
Q: If you follow these instructions, do you return to the starting point? Turn left. Turn right. Take 5 steps. Take 4 steps. Turn around. Take 9 steps. Options: - Yes - No A: Let's think step by step
And this was the model output:
We start at the origin (0, 0), facing the positive y-axis. (1) Turn left: (0, 0), facing the negative x-axis. (2) Turn right: (0, 0), facing the positive y-axis. (3) Take 5 steps: (0, 5), facing the positive y-axis. (4) Take 4 steps: (0, 9), facing the positive y-axis. (5) Turn around: (0, 9), facing the negative y-axis. (6) Take 9 steps: (0, 0), facing the negative y-axis. Since (0, 0) is (0, 0), we are indeed where we started. So the answer is Yes.