* unless temp = 0 and all previous messages in the chat are the same. On ChatGPT, you can't even see all the messages in the chat System messages aren't visible
Very likely. I asked it about a programming language I work on (which it doesn't know the first thing about) and it hallucinates features based on its name, sometimes accurately. "CollegeAI" has some obvious potential use cases.
Very unlikely. That's a pretty accurate description of the startup. "CollegeAI" doesn't suggest at all it's an admission matching platform. The hot take would've been it's a college driven by AI, or college specializing in degrees related to AI or god knows what.
Everything is possible. We're discussing what's likely and not likely. I don't buy the hallucination theory. BTW, technically it "hallucinates" everything, because it has no connection to the outside world, only memories to base its answer on. But if the hallucinations are correct, then being less correct is still a problem. If you can show it's more correct in other ways, be my guest.
People are familiar enough with LLMs to understand the nuance in their limitations. Many of these only become obvious when you transition from using it as a toy to using it as a tool.
We a novel tool like this comes out people take wild guesses about how to use a tool, jumping well past its capabilities. When they see limits, they jump back to zero (or close to zero) and try again.
Once your familiar with a tool, you know how to use it near its limits. Naturally, you’re going to feel more pain when you cross the limit. It won’t feel like you’re far away from a known good.
I suspect this is essentially all that’s going on. P
We trained a large language model and had it deployed for multiple month last year (Cedille.ai, largest open source LLM for French at the time). We had multiple companies experimenting with it to use it in production. There were multiple occurrences where we got feedback about "the model having gotten significantly better". But the model in prod was never altered.
I have had a significant decrease in usable GPT-4 via Chat. Anyone to say otherwise hasn't been using it daily as a plus user since first access. It's absolutely hindered in comparison. But also twice as fast.
Did it forget? I actually checked this out recently and generics were introduced in 1.18 while chatgpt is only trained on golang up to 1.17. I realize it could also be trained on alpha release for 1.18 but it doesn’t seem to be and that’s a plausible explanation.
Quick pass:
- questions are different.
- answer = function of several variables, most importantly, entire chat. then temperature, top-p...
- models haven't changed.
- entirely possible chatgpt's temperature, top-p etc. changed but model didn't
see images of gpt3.5 at 0 temp and 0.3 on this tweet, 0.3 makes up an answer like february, 0 gets "sorry, as an AI..."
https://twitter.com/jpohhhh/status/1665007972114743297?s=20
* unless temp = 0 and all previous messages in the chat are the same. On ChatGPT, you can't even see all the messages in the chat System messages aren't visible