Indeed. The reward function we're using in RLHF today induces AI models to behave in ways that superficially seem better to human beings on average, but what we actually want is to induce them to solve cognitive tasks, with human priorities.
The multi-trillion dollar question is: What is the objective reward that would induce AI models like LLMs to behave like AGI -- while adhering to all the limits we human beings wish to impose in AGI behavior?
I don't think anyone has even a faint clue of the answer yet.
You can't just take an arbitrary neural network architecture, and make it do anything by giving it an appropriate loss function, and in particular you can't take a simple feed forward model like a Transformer and train it to be something other than a feed forward model... If the model architecture doesn't have feedback paths (looping) or memory that persists from one input to the next, then no reward function is going to make it magically sprout those architectural modifications!
Today's Transformer-based LLMs are just what the name says - (Large) Language Models - fancy auto-complete engines. They are not a full blown cognitive architecture.
I think many people do have a good idea how to build cognitive architectures, and what the missing parts are that are needed for AGI, and some people are working on that, but for now all the money and news cycles are going into LLMs. As Chollet says, they have sucked all the oxygen out of the room.
> The multi-trillion dollar question is: What is the objective reward that would induce AI models like LLMs to behave like AGI
No, the reward for finding the right objective function is a good future for all of humanity, given that we already have an algorithm for AGI.
The objective function to acquire trillions of dollars is trivial: it’s the same minimization of cross-entropy that we already use for sequence prediction. What’s missing is a better algorithm, which is probably a good thing at the moment, because otherwise someone could trivially drain all value from the stock market.
The multi-trillion dollar question is: What is the objective reward that would induce AI models like LLMs to behave like AGI -- while adhering to all the limits we human beings wish to impose in AGI behavior?
I don't think anyone has even a faint clue of the answer yet.