I notice in my (brief and probably user error filled, I'm an embedded dev, not an AI expert) testing, it(and pretty much every other small model) seems to have trouble interpreting numbers expressed as words when filling out a JSON object like:
You might say something like five thousand fifty six, and it will fill in something like 556 or 5560.
Like as if it is just transferring digits one by one, not using the structure to know about the implicit zero.
Which is very interesting since that seems like a mistake I would make too!
It doesn't do it all the time, and I only know about the ollama quantized version, and I mostly only try the 1B models, and I've seen similar issues with other sub-2B models as well.
The other interesting thing is in a chat, almost every model I've tried seems to interpret the numbers correctly, if you say "what's ten million and fifty times eight" it will start with "10,000,050 x 8 is...".
Sometimes they get the math wrong after that, but the number interpretation is right.
I wonder if there's something special about all "intro text" in the chat mode that is actually acting like reasoning, or if the digit separators(that don't exist in JSON) help them figure out what they're doing?
I wonder if it would be better for some applications to include a line of thoughts/summary/intro in the JSON format constraint?
{"operator": "*", "command": "calculate", "a": 473, "b": 2848}
You might say something like five thousand fifty six, and it will fill in something like 556 or 5560.
Like as if it is just transferring digits one by one, not using the structure to know about the implicit zero.
Which is very interesting since that seems like a mistake I would make too!
It doesn't do it all the time, and I only know about the ollama quantized version, and I mostly only try the 1B models, and I've seen similar issues with other sub-2B models as well.
The other interesting thing is in a chat, almost every model I've tried seems to interpret the numbers correctly, if you say "what's ten million and fifty times eight" it will start with "10,000,050 x 8 is...".
Sometimes they get the math wrong after that, but the number interpretation is right.
I wonder if there's something special about all "intro text" in the chat mode that is actually acting like reasoning, or if the digit separators(that don't exist in JSON) help them figure out what they're doing?
I wonder if it would be better for some applications to include a line of thoughts/summary/intro in the JSON format constraint?
Other than that I've been really enjoying Gemma3!