> By definition, a human designer cannot prevent a superhuman intelligence from modifying itself
And we can stop reading there. That's such muddled thinking it doesn't merit reading the rest of the paper.
("By definition"? What? This is something that requires evidence. Anything that's "by definition" doesn't mean you can rest your case on it, it means now your definitions are suspect so those better be well justified)
It’s a (reasonable, in my opinion) but not proven assumption. It’s not a fact and not by definition.
I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce.
The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours.
Like under this assumption the AI agents behaviour is by actual definition unpredictable.
Yes, I wasn't really addressing those points. It's a bit pedantic to state that an axiom isn't a fact, as, while that's largely correct, it hampers communication.
Regarding your other points, I don't disagree per se, though you could make the argument that there is a substrate of existence that humans have sufficient intellect to accurately model.
If so, it isn't necessarily completely implausible that we'd be able to predict the goals of a superior intellect by assuming that it has to play by the same rules, and will be merely be superior at playing by those constraints. As such, as we could determine the goals, we needent determine the mechanisms to make accurate claims about the end result of what a superior intellect might achieve.
It's also possible that we simply understand nothing and are too limited to ever grasp anything beyond a certain point, in which case this entire argument is pointless.
I beg to differ. All the might of a superior intellect will be of no use if a bear locks a human down and starts to maul them.
The mistaken assumption that GP is talking about is related to conflating intellectual capacity with physical capacity.
Another analogy is a brain in a vat. If the brain only has read capability over the world surrounding it, then no amount of processing power can enable it to change anything about its situation.
Exactly. Power is the crucial element. You may be a genius but still be just another prisoner subject to unyielding limitations of the environment structured to maintain particular controls.
If you don't have a gun and a knife and you're on Vancouver Island, I assure you that the bears will eat you no matter how keyboard macho you think you are.
Right, but the point is that with the intellect you don't show up in that situation without a knife or a gun in the first place. A competent intelligent human foresees that being a bad situation, avoids it, and shows up armed or with bear spray.
This is exactly the same thing a chess program does when it avoids your trap. Intelligence compensates for physical weakness.
Like I said, it's something that requires an argument or evidence. It's not something that comes by definition.
People seem to think arguing that something is true by definition is some airtight case. If your definition of superintelligence includes a part about not being controllable by humans, then you have a weird definition and calling that thing "superintelligence" is an attempt to be confusing on purpose, or to smuggle your argument into the definition. Either way, it's not something that someone who is thinking carefully does.
Note: I am not taking a stance here on whether humans can or can't control a superintelligence. I am saying it is not true or false by definition, and in fact I think nobody knows for sure. It's probably true for some superintelligences, and false for others.
I think that the readily apparent axiom isn't so readily apparent -- couldn't a vastly superior electronic intellect be controlled by pulling a plug out of a wall?
No, only a very marginally superior intellect could be controlled in such a way even at a cursory glance, and I'm not even convinced a 250 IQ machine could be stymied like this.
There are many creatures which lack any kind of trait we would call intelligence that can use vastly more intelligent and complex creatures to their own ends.
That isn't really a cohesive argument given that every entity you've mentioned can be and are eradicated by human-level intelligence, which is *not* an intelligence that is vastly greater than any earthly intellect.
We are simply marginally more intelligent and that has resulted in massive gains.
An entity with an IQ of 1,000,000 after six hours of recursive self-improvement would be *vastly* more intelligent.
Right but how many arthropods have eradicated Cordyceps? How many rats have eradicated Toxoplasma?
Evolution is much like that old sporting adage, "you can only beat what's in front of you". I think there are a lot of assumptions about the nature of evolution to assume that life wouldn't find a way.
Even saying that intelligence is an ability for pattern recognition but the patterns that you need to recognise to grow more intelligent are ever more complex and obscure. How much compute power is an IQ of a million? Where does it get its energy from? How does it process the complexity in the greatest patterns in a reasonable amount of time and space. What is going on that much faster than we experience time at that is worth perceiving?
A) Human prisons frequently have contraband brought in by jailers or jailers looking the other way because humans have flaws / may think they see opportunities to benefit.
B) prisoners regularly escape from physical systems. AI is already being connected to the internet to perform actions on the user behalf so not unreasonable to think that
C) look at ex machina - not an unrealistic scenario where the AI takes advantage of an opportunity to get out of jail
D) human physical systems have exploitable flaws. Physical systems are easier to secure than digital ones. Human software security mechanisms have flaws. How are you going to build this jail if the AI is just spending its time probing weaknesses in whatever system you’re running it on to escape the jail / sandbox?
Also every 'smart' person who thought they could talk their way out of a police interrogation and is now spending the rest of their life in prison secured by knuckle dragging brutes.
Idiot Australians who smuggle drugs to/from Indonesia come to mind.
>The difference in intelligence between the police and a singular criminal, especially the police as an organism, is vastly in favor of the police.
I don't agree, as that's highly dependent on that singular criminal and the resources police are willing to expend on catching them. The police definitely have an advantage, as I pointed (most recently) here[0]:
Law enforcement aren't superhuman. They're just as dumb (or smart, but the
really smart ones end up in corner offices rather than police stations like
police and more common criminals) as the next guy. Their big advantage,
especially in a circumstance like this, is that they only have to get it
right (i.e., find some evidence) once. The alleged perpetrator of a crime
needs to get it right (in covering their tracks, destroying evidence, etc.)
every single time to make sure they aren't identified and caught.
Yeah but I'm talking about scenarios where all people needed to do to avoid being locked in a little box for life was either A. Not do the obviously stupid thing that everyone is telling them is stupid and that they should absolutely not do. and B. Shut the fuck up.[0]
There are supposedly brilliant people serving life sentences right now and the only reason they're facing that situation is because they couldn't shut the fuck up when they sat down in a little room with an at best slightly above average intelligence police officer.
The officer had a higher working intelligence in that case.
If I prepare for a test and perform better than someone smarter than me, my working knowledge made me superior at that task, even though I was less intelligent at the time I performed it.
The problem with this comparison is that humans, a marginally superior intellect to the collective evolutionary "intellect" of pathogens, regularly eradicate them.
Superintelligence didn't create Heaven, a world of no suffering, first.
Superintelligence did not design biological genetics: including inquest.
Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree?
Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
> Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree?
This doesn't necessarily follow. In fact, it brings up the question of what even makes an entity distinct at all from its substrate, which is philosophical.
Anyhow, "disagreement" as a concept seems to result logically from a divergence of goals or capability to model reality. As such, I suspect that it is a uniquely human (or limited agent, an agent that can't recursively self-improve and an agent that desires aggregation of resources) mode of cognition.
>Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
I don't know how to comment on this beyond thinking it kind of misses the point. You don't really need to spin your wheels worrying about how a superintellect would be "accountable" when your ability to model reality would be essentially worthless.
Theory of Mind develops along with ideas about dependence, independence, and interdependence.
Humans tend to disagree according to strength of evidence given an epistemological fact-findimg process like "beyond a shadow of a doubt" or "reject the hypothesis due to insufficient structural power"; we disagree about sources, methods, motives, and intent.
No, if one says they are "Superintelligence" and they've solved it, they have a job for you; what should one do?
> The goal of evidence-based practice is to eliminate unsound or outdated practices in favor of more-effective ones by shifting the basis for decision making from tradition, intuition, and unsystematic experience to firmly grounded scientific research.[3]
Can you not wrap your head around the possibility of something that is smarter than any human, but is also not at the level of some kind of omniscient omnipotent being?
1. My point is that YOU can't. The idea that a human can perfectly measure ability that far surpasses their own and compare if to other abilities that far surpass their own makes no sense.
2. A recursively self-improving AI would appear omniscient or omnipotent in a very short period of time. A recursively self-improving AI converting the solar system into compute for further recursive self-improvement would be unfathomable.
3. You're operating from an anthropic lens. Your conception of a superintelligence is a machine with an intellect that is slightly smarter than the smartest human. A machine with an IQ of 1000 would be God. We have no ability to conceptualize a machine with an IQ of 1,000,000.
And we can stop reading there. That's such muddled thinking it doesn't merit reading the rest of the paper.
("By definition"? What? This is something that requires evidence. Anything that's "by definition" doesn't mean you can rest your case on it, it means now your definitions are suspect so those better be well justified)