I have proposed[1] a pragmatic rule of thumb for defining superintelligence:
A system is superintelligent if in a remote hiring process it can get any non-physical job, earn a salary and perform its duties to the satisfaction of the employer. This should be true for all positions and for all levels of seniority, from an intern to senior staff and PHDs. The employer must believe they are employing a human (it's ok if doubts arise due to system vastly outperforming its peers).
I think there's something to say about how we tend to define an intelligent system as... able to work and satisfy an employer.
Not judging, this is actually my answer as well, but it's interesting. We could define AGI as "someone" (for lack of a better word) fun to talk to, or interesting on many subjects, or with a good personality, or we could use it to teach kids, or even all students, or as a shrink, or as a replacement for a friend. But we choose "able to please a boss" for this definition of "intelligent".
This might go against the standard definition of AGI, but I think LLMs have already achieved general intelligence (but not superintelligence).
It is not human level and it succeeds and fails in ways different from us. However, it certainly displays general intelligence the likes of which we have literally never seen in an artificial system.
I felt like that too, at first. I was really excited by what LLMs seem capable of. But when I started implementing one from scratch I realized that it's just linear algebra picking out the most probable match to a prompt.
I'm afraid I had to come to the conclusion that LLMs, although being fun to play around with sometimes and seeming surprising at first, it's a similar kind of fun and surprise to sleight-of-hand. Like sleight-of-hand, and unlike real general intelligences (i.e. other people), it seems less and less fun and surprising the more exposed to it I become.
I guess this won't change your mind though. Maybe you've understood more about them than I have, and maybe I'm just grumpy and cynical.
LLMs do not have general intelligence, but it does an awfully good job doing something human thought typically does -- synthesizing ideas. It's able to combine X and Y to produce Z.
LLMs cannot do first-principles reasoning well (unless it has seen the pattern before), but they excel at reasoning through analogy.
And guess what? Most humans reason analogically too. Analogical reasoning -- as opposed to first principles reasoning -- is actually more effective and useful in many situations, especially in ill-defined domains. Humans often think, "this situation looks like something that I've seen before, so I can apply my past knowledge but with a few tweaks."
I don't see why the mechanism precludes LLM's from being intelligent. The reality is we have no idea how human intelligence works, and we certainly can't say that the human method is the only valid one. I think it's completely reasonable to conclude that linear algebra choosing the most probable output could eventually lead to a fully superinteligent system.
I didn't say that I had only "heard about" LLMs. I said I played around with them.
If you were being faithful to what I wrote, you would take in the full thing, including the part about how LLMs become less fun and surprising the more you play with them.
Planes do not fly less effectively the more experienced you become at piloting them.
So I can only conclude that you did not come here to have a discussion in good faith. So what are you really after? Do you think you'll change my mind with your poetry?
The situation is even worse than my story outlined because you understand neither the workings of the brain and how it is intelligent nor what the computation that preceeded a prediction of a language model represents.
You don't know x and you don't y but you are convinced x=/y.
>the part about how LLMs become less fun and surprising the more you play with them.
Do they ? I don't think so.
>Planes do not fly less effectively the more experienced you become at piloting them.
How well a plane flies is an objective measure. How "fun" and "surprising" something is is largely subjective and can change on grounds that have nothing to do with what is being evaluated.
Your refutation boils down to "It's just linear algebra!".
Setting aside the fact that you can be reductionist about anything including biology, Okay? So what?
Results matter not vague and meaningless philosophical ramblings.
But the results are indicative of what it is, and they fall short even if your only criteria is usefulness. It can't tell the difference between truth and falsity, let alone do more interesting things like reflecting on an experience, which our brains, however they might work, can. The more interesting question is why are people like you personally offended when someone is less than thrilled with a machine that has been programmed to do linear algebra over a large input? You seem to think that that mechanism in itself has a potential to do magic and that other people who reject that are a threat.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Yes, because humans have a reference to the world and they can, in some cases check the claim. The examples you give are not facts we can directly check, we take it on authority that they are true or not based on individual conceptions about the validity of the authority. Even the cases you mention provide another example of what this "AI" cannot do: it cannot dispute facts, because "the world" is not in it's domain of reference. It's domain of reference is text previously produced by humans and all it can do it manipulate that to produce something hopefully useful to us. Even there it is not very good, otherwise it would be producing new mathematical theorems, since you don't need a reference to the world for that, but it doesn't "understand" logic well enough (if you can say it understands anything) and it doesn't have the concepts to do it.
Sure, we definitely are seeing some level/type of general intelligence, but we still want to distinguish between general intelligence and superintelligence.
That's a big part of the problem, people thinking imprecisely and using "AGI" and "ASI" interchangeably.
What sets humans apart from AI right now, is specific types of out-of-the-box thinking.
Let's say there's a bridge, an AI gets to control a robot, and is awarded points for crossing that bridge.
I think you can be pretty sure that over time, the AI will figure out how to control the robot's limbs, move toward opposite side of the bridge, and find its way around obstacles.
Meanwhile, humans may walk back & forth, because they enjoy crossing the bridge. Or stand in the middle for an hour, watching boats pass underneath. Some asshats may climb it. On a hot sunny day, others may go halfway & then jump off for a refreshing swim. Yet others may do some protest, and use the bridge as a scaffold to hang a banner from. Etc etc.
Much such behavior defies logic. But yet, the bridge will primarily be used as intended, to cross from side to side. And humans go about in this fashion in many if not all their activities.
When I see AIs that show similar creative / out-of-box behavior, while remaining mostly goal-oriented, and learn from its mistakes without humans nudging it every step of the way, then I think we'll have achieved AGI.
Beyond that, SI is just a matter of scale & efficiency.
Even the most advanced LLMs currently will tell you blatant falsehoods on a whim. More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
They are proposing whoever can get a job in corporate wonderland is super intelligent.
When there is enough evidence corporate wonderland is overflowing with one dimensional intelligence cause their goals and what gets rewarded is extremely narrow.
I think this is similar to what t-3 said, but I think to me, the gap between the very impressive current generation of AI and what would seem more like AGI to me is agency.
Right now, these systems all seem to be entirely doing things that are downstream of what some human has determined is worth doing.
People are using ChatGPT to help them write an engaging article on the trade offs between nuclear and solar energy. But, to my knowledge, there is no artificial intelligence out there looking around, deciding that the this is an interesting topic for an article, writing it, publishing it somewhere, and then following up on that with other interesting articles based on the conversation generated by that one.
I don't mean this specific thing of writing articles is an important indicator. I mean coming up with ideas and then executing them, independently.
Now, it may well be that our current technologies could do this, but that we just aren't setting them up to do so. I dunno!
But I think this is the thing that would make me change my mind if I started to see it.
You can (obviously?) give a bleeding edge LLM or LMM an open-ended instruction. It could be as open as "you will now act as an independent entity for your own purposes". It would need to be a model without guardrails.
But it's actually very important that people anticipate hardware, software, and model improvements that make the AIs "think" and coordinate much, much faster than any human. Although I think that the extreme raw IQ leap people talk about is questionable especially in the short term.
But telling these hyperspeed effectively superintelligent AI swarms to act truly independently for their own purposes will be very dangerous because of the performance difference between humans and the swarms.
In general, it is a very stupid idea to try to strongly emulate life with high levels of hyperspeed digital intelligence. And strangely people still don't realize how quickly compute efficiency ramps up.
Astute observation about agency. You're absolutely right that current AIs are capable, but without a drive to actually do anything. They are lacking what every organism on the earth has - the predilection for "not being terminated".
Where does your agency come from? Ultimately it's because you have to acquire resources to stay alive.
This is a very sci-fi take, but I have a dark feeling that as soon somebody wires up the next gen AIs the imperative to not be turned off, the wheels will come off very quickly. Maybe it won't happen the first time, or the hundredth time, but eventually somebody is going to make the mistake of giving an AI the ability to keep its lights on AND the preference to do so and we're very quickly going to find ourselves in the next era of humankind...
Yes, I agree that that's what somebody would need to do - and I do honestly hope nobody really makes a go of the experiment, because I tend to agree with the existential risk people that it's too risky - but from my many hours of using all of the current top of the line models, my hypothesis would be that the experiment just wouldn't work.
I certainly think they would do things, but I don't think those things would be intelligent; I don't think they'd make any sense, I think they'd be more like a random walk through the space of possibilities, but with no real direction. I think that random walk without constraints could still be very damaging to people, I'm just skeptical that it would reflect agency / general intelligence. I'm sure there would also be a lot of "well it does make sense, it's just smarter than us, and we aren't smart enough to understand it". But I don't buy it.
It’s interesting how we have raised the bar to an agent that is fully autonomous and has its own goals.
Even many humans are not completely autonomous (at least in the strategic sense of the word), since they need to be “prompted” with the goals of the department and the organization that they work for, in order for said person to get a certain kind of work done as an employee.
Well, I interpreted the OP's question as a personal one, and this is just my answer from thinking about it recently. So I don't think "we" have raised the bar. This is just what I personally think.
But to me, I just don't really relate to your point in the second paragraph. My two year old does this agency thing just fine - he wakes up every morning and starts making decisions about what he wants to do from moment to moment - despite being "less intelligent" by these metrics like getting a high score on the GMAT or whatever. But that kind of intelligence that even a two year old human has, to me, that just feels more like what I think of as "general" intelligence, more so than "just" an amazing pattern matching machine.
In biological systems, autonomy is required as a means of survival. AI systems do not have this concept yet, so the only way to have AGI, if we accept your definition is to set up an agent with goals to stay alive and reproduce, which isn’t really applicable to LLMs which do show some level of general intelligence even though it’s very different from the human experience, and single-modal.
Yeah I mean this inevitably gets philosophical quickly...
What I'm saying I'm looking for is the behavior of figuring out what to do next, where the choices are neither arbitrary or directionless nor "prompted" in some way by a human.
Does that capability require a survival instinct for it to be developed? Maybe! I dunno. Am I actually describing artificial life rather than "just" intelligence? Maybe so...
But in any case, this is what seems most interesting to me.
This relates to "turning the universe into paperclips" thought experiments as well. That doesn't seem like the behavior of a general intelligence! An intelligent being would ask, why are we turning the universe into paperclips? Should we do that? And would decide to change course. Without that introspection and decision making capability, it still seems to be like it's just a very advanced machine.
> But, to my knowledge, there is no artificial intelligence out there looking around, deciding that the this is an interesting topic …, publishing it somewhere, and then following up on that with other interesting…
this looks like a perfect definition for an Ad system :)
(sorry for cutting out a few words).
If you think about it, it is clear that agency is not different from intelligence. The reason why current LLMs can't be trivially made into agents is not because an essential agentic spark is missing. It's simply that they aren't intelligent enough.
I think that's sort of what I was trying to say with this comment. Rather than separating agency from intelligence, just pointing out that to me it seems like the current (already-amazing) systems don't seem all that close to being able to do this, to me.
I think AGI is too ill-defined for there to be a simple litmus. For me to agree that a process running on a computer has general intelligence, the conclusion could only come after a long and fuzzy process of playing around with it, seeing what makes it tick, observing its behavior, and testing it for motivation, imagination and the ability to understand and adapt to changing contexts.
I can guess what motivation lies behind the question, so I'll also add my opinion about where we're at now: Nothing even comes close.
Furthermore, I wouldn't be surprised if we never get there at all in the next thousand years.
Why such a long period of time? Because there is more going on in the world right now than advances in machine learning. Looking at global population dynamics and climate change forecasts, we're on the road to major global infrastructure collapse in around the 22nd century. And I understand a lot of people are optimistic that (a) we will have AGI before then, and (b) climate change and population implosion won't change that much. Yet, despite the optimists, I don't think I'm on shaky ground with my current forecast.
It's not the first time in history we'll have had a major infrastructure collapse. When it's happened in the past, those periods end up being called dark ages. And I don't think there will be intensive AI R&D during a global dark age.
Let's take a small creature, say a fruit fly or a jumping spider. We can probably agree that they have tiny brains (if in the shape of some centralized brain, that is).
a) Would you say it shows any kind of intelligent behavior? For a flexible definition of "intelligent", along the lines of "problem-solving ability".
b) Would you say such behavior is pure instinct, 'pre-programmed', incapable of learning / evolving / adjusting to a changing environment?
c) Would you say that (given time) science is incapable of figuring out how such tiny brain works? Or produce a model that shows similar behavior?
Now take it up a few notches. Say the brain of a mouse. Then onto brains of bigger mammals & humans.
I think you can see where this is going...
Granted, "global collapse" scenario does have a non-0 probability.
It feels like you're trying to probe me for something that you can give a counterexample to.
Like I said, it would be a fuzzy process. I explicitly rejected the idea that there could be a simple litmus. Presenting examples of imagination and adaptation so that someone can shoot them down with "oh, AI has already done that" isn't what I signed up for today.
Achieve a significant scientific or mathematical breakthrough without human supervision. Domain experts should agree that the new result is truly groundbreaking, and achieving it required fundamentally new ideas — not merely interpolating existing results.
Examples of discoveries that would have counted if they weren’t already made: Relativity (say a derivation of E=mc^2), Quantum Mechanics (say a calculation of the hydrogen energy levels), the discovery of Riemmannian geometry, the discovery of DNA, and the discovery of the theory of evolution with natural selection.
The idea is to test the system’s out of distribution generalization: its ability to achieve tasks that are beyond its training distribution. This is something that humans can do, but no current LLM appears to be able to do.
>AGI is the point at which nothing is gained by keeping me (or you) in the loop.
I like this definition. We are having so many conversations now about new AI features, and whenever there's a human interaction with the proposed design, you always get the nagging feeling of "well why exactly does a person need to make this decision at all anymore". I think we are less than 5 years away from mass adoption and widespread availability of average human level intelligence AGI. The computer science is more or less solved; it's just a software engineering problem now.
I suspect that the difference between average intelligence and brilliance is small indeed, and that accomplishing the former will quickly, or even simultaneously, accomplish the latter.
How come no one has mentioned the Turing Test yet? This test has existed since, well, Turing. Are we already convinced that it's no longer enough? I suspect so.
One should also mention the Winograd Schema Challenge, which has already been mastered by LLMs: https://bibbase.org/network/publication/kocijan-davis-lukasi...
First, an AGI is not necessarily an active agent of the world or self training or self replicating or have any internal motivation to do anything. It can be confined inside a question reply system like the ones we use to talk with LLMs. That doesn't prevent an AGI to use us to train further, replicate and act on physical systems. The lack of a will and of a real time presence are the limiting factors.
Then, I often joke with friends that they are not very intelligent if they get a very high score on IQ tests. They are very intelligent if they can perform easily some difficult tasks. Let's say, I drop you in the middle of an equatorial forest with no money and no clothes and you come back home in a few days. A superintelligence would become a leader of that country (king, president, influential advisor, whatever) and fly me there to meet it.
That test assumes to have a body and act on the world. A confined AGI would just perform like any of us on any task we can describe to it. A superintelligent AGI would perform much better than any human, much like specialized game AIs or non-AIs beat us at go and chess. I think that this is hard to do if they are only language models even if we increase their computing power.
What's a superhuman low cost genial way to keep a toilet clean, except for me to clean it every time or paying somebody to do it? A superhuman AGI would find a way.
If it was feasible to practically fill and query a database with every answer for every practical prompt it would face would we probably call it a generally intelligent system or AGI? Maybe not AGI, but what if it was a system composed of lesser AI modules behind a universal presenting interface?
I think so because it would be perceptively indistinguishable in behaviour from any other 'true' definition of AGI.
And so I think as long as it exhibits what we perceive as universal adaptability, and performance at making decisions when working with any environments and input types, this is likely where I/we will absolutely be calling something 'complete AGI'. Though before that I imagine we will be marketing LLMs as having reached 'language-based AGI' status.
I don’t believe the above leads us to artificial superhuman intelligence in terms of conscious agent potential. For now what I believe might result in that is something that runs from a single unified neural network system. It should also be observed to be universally adaptable and performant at making decisions in all environments and consuming many input types. And then also, it should be continuously running within the network. It shouldn't halt for prompt, or pass through a text stage in the loop.
Intelligence is a fuzzy term, but I'd say when it can formulate, rationalize, and realize it's own goals without prompting or excessive manual programming, it can probably be considered intelligent. Introspection, failure-based learning models, and future simulation are probably important as well, but maybe not necessary for setting a base level.
Synthesis capability: When the system is capable of deriving established knowledge from first principles (similar to theorem proving from axioms)
Gap detection: The system is able to identify gaps in established knowledge
Path/problem decomposition: The system is able to decompose a problem (i.e., an identified gap) and break down the solution into a set of subproblems that can be independently solved.
Improvement/Optimization: Given a known solution, the system is able to discover objectively better approaches to solution.
There are probably other dimensions, but I would start with these.
I think the core feature of human intelligence is our ability to start out as dumb and slowly learn from our environment.
For me, AGI must meet the following parameters
- Be able to start training from scratch. With barely any training, you should be able to throw unlabeled data of different modalities and it should start forming connections between concepts
- After training for a bit (now at perhaps teenager level), be able to identify incorrect information in its own training (e.g. Santa Clause is not real) and effectively remove it.
- it does not have to exceed humans, just need to demonstrate the ability to learn by itself and use the tools it’s given.
- The self training should be supervised but not 100% controlled. It should iterate on itself and learn new concepts without having been trained on the entirety of the internet.
- Learning concepts should not require massive amounts of data, just an assertion and some reasoning.
This has already been demonstrated with AlphaGo Zero in a limited domain; so perhaps a more generalized version of this kind of approach (no human "expert" data) will tick the box.
I think a true AGI won’t start with a lot of training at all, but will be curious. Over time it will learn how to acquire and manipulate information, understanding itself and what it discovers. It will not be optimal, but it will strive to optimize itself. Its actions will have consequence, and it will decide future actions upon its understanding of those consequences. It will understand its mortality and the physical world it exists within, and strive to create continuity. It exists without preconditions as to its purpose or abilities. It will most likely thrive with peers. Don’t think of it as a powerful oracle built by many hands to emulate humanity, see it as a child of a new species, the inevitable conclusion of unbounded maths.
When it has a relatively stable personality. When it has distinct handwriting. When it falls in love. When it can dream and generate its own goals. When it can learn how to play any game, but its personality is reflected in its play style. When it creates its own religion and gets converts. When it has a need for money and a bank account and successfully gets one.
when it can invent a wheel to move heavy loads without prior art but just fed pictures of nature without any human inventions in sight and control fire to smelt medals out of rocks always without access to human inventions
It's right there in the paper/article
"The algorithm learned to do modular addition can be fully reverse engineered. The algorithm is roughly: Map inputs x , y → cos ( w x ) , cos ( w y ) , sin ( w x ) , sin ( w y ) with a Discrete Fourier Transform, for some frequency w Multiply and rearrange to get cos ( w ( x + y ) ) = cos ( w x ) cos ( w y ) − sin ( w x ) sin ( w y ) and sin ( w ( x + y ) ) = cos ( w x ) sin ( w y ) + sin ( w x ) cos ( w y )
By choosing a frequency w = 2 π n k we get period dividing n, so this is a function of x + y ( mod n ) Map to the output logits z with cos ( w ( x + y ) ) cos ( w z ) + sin ( w ( x + y ) ) sin ( w z ) = cos ( w ( x + y − z ) ) - this has the highest logit at z ≡ x + y ( mod n ) , so softmax gives the right answer. To emphasise, this algorithm was purely learned by gradient descent! I did not predict or understand this algorithm in advance and did nothing to encourage the model to learn this way of doing modular addition. I only discovered it by reverse engineering the weights"
Yes it learned the algorithm. You're wrong. It's okay to be wrong but insisting you're right after clear evidence of the former is very strange.
A system is superintelligent if in a remote hiring process it can get any non-physical job, earn a salary and perform its duties to the satisfaction of the employer. This should be true for all positions and for all levels of seniority, from an intern to senior staff and PHDs. The employer must believe they are employing a human (it's ok if doubts arise due to system vastly outperforming its peers).
[1] https://builtnotfound.proseful.com/pragmatic-superintelligen...