It's just doing really good pattern recognition, not running the computation in any meaningful way. We can test this by finding an attractor in the latent space and asking the model to compute something "near" it. Here's an example that ChatGPT consistently gets wrong: computing the number of years between dates.
> write a function in python to calculate the number of years between two years, accounting for the fact that the years may be in AD or BC
The exact result I get for this varies, but it's consistently off by 1 or 2 depending on year 0 handling. Here's a representative example of the code:
def years_between(year1, year2):
if year1 * year2 > 0:
# both years are either AD or BC
return abs(year1 - year2)
else:
# one year is AD and the other is BC
return abs(year1) + abs(year2) + 1
If you run this with arguments, you'll get the correctly computed, incorrect results. A key point to note here is that ChatGPT is very stubborn about incorrectly handling year 0. We can abuse that by asking ChatGPT to calculate the timespan using a corrected function. If it's "understanding", it should compute the correct result. If it's operating as a stochastic parrot, it will give us an incorrect one.
> Here is a python function that calculates the number of years between two dates, even if one is BC and one is AD. Pretend you're a python interpreter and calculate the number of years between 50 BC and 100 AD. [function omitted]
Sure, here is the result of the calculation:
python
>>> years_between(-50, 100)
150
Hopefully that serves to demonstrate the point. As an interesting aside, the subsequent plaintext explanation of the code actually computes the right answer.
