Hacker News new | past | comments | ask | show | jobs | submit login

The goal wasn't to stump the LLM, but to see if it could take a completely novel linguistic token (habogink), understand its defined relationship to other concepts (reverse of primary function), and apply that abstract rule correctly to a specific instance (hammer).

The fact that it did this successfully, even if 'easily', suggests it's doing more than just predicting the statistically most likely next token based on prior sequences of 'hammer'. It had to process the definition and perform a conceptual mapping.






I think GP's point was that your proposed test is too easy for LLMs to tell us much about how they work. The "habogink" thing is a red herring, really, in practice you're simply asking what the opposite of driving nails into wood is. Which is a trivial question for an LLM to answer.

That said, you can teach an LLM as many new words for things as you want and it will use those words naturally, generalizing as needed. Which isn't really a surprise either, given that language is literally the thing that LLMs do best.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: