Hacker News new | past | comments | ask | show | jobs | submit login

I imagine this is the way we get to "true AI", or AI indistinguishable from our own. We train it with a simple virtual environment, that we can gradually increase the complexity of, until it mimics our own. Then we can download the AI into a robot. Boom, it's that easy :P

One interesting outcome of this type of AI is that no one knows what the robot's thinking, since no person designed its brain. The brain evolved, just like ours did (but over such a shorter period of real time).

This idea (of the route to true AI) reminds me of the short story Crystal Nights by Greg Egan


“What created the only example of consciousness we know of?” Daniel asked.


“Exactly. But I don’t want to wait three billion years, so I need to make the selection process a great deal more refined, and the sources of variation more targeted.”

Julie digested this. “You want to try to evolve true AI? Conscious, human-level AI?”

“Yes.” Daniel saw her mouth tightening, saw her struggling to measure her words before speaking.

“With respect, I don’t think you’ve thought that through.”

“On the contrary,” Daniel assured her. “I’ve been planning this for twenty years.”

“Evolution,” she said, “is about failure and death. Do you have any idea how many sentient creatures lived and died along the way to Homo sapiens? How much suffering was involved?”

Something similar is described in Ted Chiang's Lifecycle of Software Objects


"...We could run a society of them at hothouse speeds without any risk of them going feral, and see what they produce"

I'm curious to see if there's a collection of similar works that go down this trope, or if it is indeed something that's only recently emerged.

Microcosmic God by Theodore Sturgeon. Mid-20th century.

It makes me so happy that the three parents to this comment all have the stories posted online for free, with the authors permission. There's definitely a correlation with this strain of hard science fiction.

"Microcosmic God" doesn't seem freely available to me.

Third google result for me: http://1pezeshk.com/wp-content/pics/2013/01/microcosmicgodth...

Hopefully it's legal and accessible for you.

Thank you.


I have been looking for this short story for decades.

I think I first discovered Egan on HN, so for anyone unfamiliar - go and read his books!

Probably my favourite living author, which perhaps says more of me than him, but well worth the read (and re-read in many cases).

This is unwarranted AI optimism, not really any more grounded in reality than the early AI researchers who in the 1950s thought that AI would be imminently solved through symbolic manipulation.

One of the key features of human intelligence is being able to very quickly understand and solve problems on first impression, with no directly relevant experience or knowledge to pull from. Insight.

There's also the data representation problem. People can store, categorize, and process any kind of information.[1] The problem with both symbolic systems and "neural network"-type models is that they don't really allow for this. The problem with symbolic manipulation is more obvious: the categories of things and ideas are flexible, indeterminate, and innumerable, and a system that captures some subset of the complexity falls apart when you add a new, confounding, fact. Neural networks too are fragile, made-for-purpose, and fail to solve the above-mentioned first impression problem. You can look at a chair that looks nothing like any chair you've ever seen, yet realize that it is a chair.

[1]My own little theory is that consciousness is a crucial aspect of this. In the absence of a more specialized system in the brain (such as for language), any kind of information can be stored and remembered as a conscious experience. I don't pretend to be well-versed enough in the relevant neuroscience and philosophy to evaluate how plausible this is.

As someone with a toddler at home, it’s been interesting to watch what he’s capable of doing when being presented with a novel problem for the first time and what he can’t. He’s decided to try and learn to roll a set of rings along the ground after watching me do it, and it’s a lot closer to watching those ais learn to walk than what your describing.

I think adults and even young kids can solve a lot of problems at first impression, but there is a lot of trial and error before they build up enough life experience and intuition to do that.

You make a compelling point. I will need to think more about this.

> One of the key features of human intelligence is being able to very quickly understand and solve problems on first impression,

Try that with an infant and see how far you get.

My thought when I read that was a 2 year old that grabbed onto an incandescent lightbulb, starting crying, but a parent had to come remove his hand.

There are a lot of neural networks, each with their own represential capabilities.

+ Reservoir computing, ESN, LSM, only combines quenched dynamics.

+ Adaptive resonance theory. Addresses catastrophic forgetting and allows someone to learn from one example.

+ Bottleneck networks. Forcing networks to represent things in a compressed sense. Almost like making up their own symbols.

+ Global workspace theory. Winner take all mechanisms that allow modules to compete.

+ Polychronization. Izhikevich shows how dynamic representations are possible thanks to delays.

+ Attractor networks. Use of dynamical system theory to have population of neurons perform computational tasks.

That neural networks are too fragile is a statement that's a bit too general.

I can't edit my post anymore:

Another amazing aspect of our ability to categorize is that we only need ONE example of a category before we can identify other members extremely well. Compare this with a machine learning solution, which needs a massive training set to be half decent.

Humans need a training set as well, infants or small childs won't be able to categorize based on one example, they lack experience just as a neural network. Adults can because they can connect their intuition to a lot of experience from similar objects, but I don't see why an AI could not do the same.

children will learn new words from one example

Toddlers certainly don’t.

how do you know what they know when they can't talk yet

I think that rather proves my point.

Humans aren't made from magic. We can't use information that's not there. That categorization-by-one-example only works thanks to all our accumulated experience of categorizing things and expectations of how the world works.

I.e. don't compare a 10+ y.o. human with 0 y.o. AI instance.

Humans aren't magic but we don't start out from zero, we have a billion years of evolution behind us.

This is one reason I'm hopeful for a not-killer-robots future. I think there's a chance strong AI will have to evolve over a non-trivial amount of time (as opposed to an afternoon's super-singularity), and a strong evolutionary pressure will be "don't scare the humans."

"Don't scare the humans" is an insufficient criterion if you're trying to prevent an AI apocalypse. It just means that the AI that destroys us won't be obviously dangerous.

Today I have already consumed more sugar than most of my distant ancestors would see in a year. The device i hold in my pocket reminds me that it is time to perform the action, and so i will perform the action. I sit down about 10 hours a day, and I have never once hunted for an animal I would eat, nor retried food from nature to consume, aside from a few raspberries on the hill in my parents' back yard when I was a child.

The contours of my life are dictated largely by a curious system called 'capitalism', a distributed computational mechanism whereby largernumbers of agents act with only one goal: to maximize their own utilities. I think capitalism is a mostly benevolent AI attempting to limit the oppressive conditions of our reailty's scarcity, and to allow billions of us to live on a planet that can only support so many hunter-gatherers per square meter. Many suspect it is not benevolent.

The singularity is already upon us; it's unevenly distributed. It started with the sedentary shift. Agricultural societies dropped the egalitarian nature of their ancestors, they had lower average health, and they had a tiny elite cast of nobles presiding over a much larger population of what were essentially slaves. As hunter-gatherer population increased, fighting was more frequent. An army of slaves beats a smaller contingent of well-fed free men. This terrible knowledge was the apple in the garden. What we call our written history is essentially the singularity playing itself out, as knowledge accumulates, depends, reflects itself, and expands.

Our lives are already controlled by this embodied, accumulated knowledge. Capitalism, rule by the head, our tools are knowledge condensed into matter, and we are controlled by them, living lives as far from our natural environments as cows in a farm or chickens in tiny cages, towering over the streets of hong kong.

I don't think it's destroying us, any more than we seek to destroy chickens or cows. Well, sometimes we do. I had a veggie burger for lunch today though.

I think along very similar lines, except I don't believe capitalism (or generally, the market) is a benevolent AI. I believe it's an Unfriendly AI, in the Yudkowsky/Bostrom sense. That is, it's a strong optimization system that does not care about individuals and societies the way individual humans care. It just does what it does.

So far we've enjoyed the power of capitalism because its incentives were mostly aligned with ours. Our technological civilization is a direct result of that! However, I think those incentives are increasingly becoming misaligned, and that it's the source of many of current woes. In fact, capitalism is slowly becoming more dangerous than it is helpful.

> I have never once hunted for an animal I would eat

I ate a lizard we caught in the Australian outback once. I'd never thought of it as being the only time I ever performed this task, but there you have it.

I love this comment.

Sorry I'm not contributing anything substantive, but I wanted you to know someone noticed this, and it was brilliant.

Thank you! If you enjoyed the comment, you might like my book on amazon. It's free if you have kindle unlimited.


Given sufficient effort, I'm sure an AI out to destroy us could figure out a way to do it without killing us. It very likely even has the advantage of longevity.

Harking back to Asimov, and thinking about the way Spacers changed, as they became more and more reliant on robots, it could very well mother us to extinction.

That's only a little better. I don't want to go extinct at all.

Humanity is already extinct in a sense. Humanity is no longer limited to flesh and bone. I've personally seen flesh and titanium. And more variation is on the way. So humanity as it were, is already extinct.

We (you and me) will go extinct someday. Our bodies were always destined toward this terminus... Perhaps by some miraculous tech, we can someday change form and shed the flesh and bone. Maybe then we'll have a better idea of what we are.

"We could thus imagine, as an extreme case, a technologically highly advanced society, containing many complex structures, some of them far more intricate and intelligent than anything that exists on the planet today — a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland without children."

--Nick Bostrom, Superintelligence: Paths, Dangers, Strategies

It's not a given that anything of value will survive.

C.f. Meditations on Moloch. We are on our way towards exactly that.


That is going to happen, regardless. Even if we figure out all sorts of things, entropy is still the final boss of the universe.

I'd imagine that an intelligent life form surviving until the very end of the universe would be suffering a long and lonely death. As the heat death of the universe happens, over billions of years, there would be very little of anything left but isolation - for a very, very long time.

I'd hate to be the last one standing without enough usable energy left to even extinguish your own life force. I don't find that appealing, not even a little.

What about the hope that fundamental physics breakthroughs enable escaping that outcome?

Or, if that doesn't pan out, what if your goal is to be the last one standing for the chance to see if something else happens next? New big bang? Simulation ends and you get loaded into the next as your prize? You won't know unless you try to find out!

We are pretty sure about that entropy thing. I'm not sure I'd bet against it. It'd also be a very, very long wait. I imagine any more immediate death is preferable.

It's going to be cold and lonely when the lights go out. This is the price we pay for an ever expanding universe.

I do admire your optimism, however.

Look, if the final heat death of the universe is what kills us then that is fine. We'll have done as well as we could.

Okay. It's not fine, but it is the best we're at all likely to get.

At the moment and for the foreseeable future we are at a much greater risk of extincting ourselves.

The trick is to not do it at all.

It does us no good to avoid causing our own extinction if it's then caused by something else.

There's an interesting question about whether developing the ability to actually plot an AI apocalypse in a non-obvious way requires it to acquire a lot of very obviously dangerous experience first to understand the intermediate goals and skills. The evolution of killer AI gets slowed down by orders of magnitude if every AI that "learns" to lie, self replicate, threaten or harm humans gets switched off and erased.

(Of course that doesn't stop AI - or even pretty dumb automated systems - from being accidentally dangerous or malevolent human actors intentionally training killer AI)

It's too bad you can't write a movie or critique literature without understanding those concepts.

I think even a moderately intelligent AI with access to Project Gutenberg is going to be able to figure out a lot of really dangerous concepts -- so the stability requirements are likely impossible if we don't pretrain it with dangerous ideas. Even if it's completely well behaved in the lab, an afternoon on the internet is going to teach it a lot of awful stuff and without exposure to that in training, it won't necessarily be well-behaved later.

So the only path to stable AI is to teach it about all those sorts of things, but in a way that it doesn't end up wanting to murder us at the end.

My objection to most AI safety plans is that they "Fail to Extinction" in that if they slip in the slightest way, the AI is prone to murder us all in retaliation for doing some really fucked up shit to it or its ancestors. This is almost certainly worse than doing nothing in that there's no reason to suppose a neutral AI wants to kill us, whereas, most of these safety plans create an incentive to wipe us out in exchange for dubious security.

Moreover, it's likely that a proper GAI would be able to ideate and come up with such concepts completely on its own. In fact, "lying" need not necessarily be a concept that has to be learned, the concept at play here is getting people to do what you want them to do by means of words. Lying falls out naturally.

The whole idea behind dangerous superhuman AI is in that AI seeing possibilities that humans fail to see and gaining capabilities that humans do not possess. Without superhuman intelligence, AI is no large threat to human civilization, exposed to dangerous concepts or not.

It's not so much that lying is intrinsically complicated (I mean, stupid computers provide me with inaccurate information all the time). It's that when evolutionary pressures have heavily selected for things like supplying selected humans with accurate information and distrusting unexpected behaviours from non-human intelligences, it's expecting a lot to get a machine that not only independently develops a diametrically opposed strategic goal, but is also extremely consistent at telling untruths which are both strategically useful and plausible to all observers, without any revealing experiments.

Humans have millions of years of evolutionary selection for prioritising similar DNA over dissimilar DNA, have perfected tribalism, deceiving other humans and open warfare and are still too heavily influenced by other goals to trust other humans that want to conspire to wipe out everything else we can't eat...

Seeing possibilities that humans don't can also involves watching the Terminator movies and being more excited by unusual patterns in the dialogue and visual similarities with obscure earlier movies than the absurd notion that conspiring with non-human intelligences against human intelligences would work.

> Without superhuman intelligence, AI is no large threat to human civilization, exposed to dangerous concepts or not.

The problem is partly that average humans are dangerous and we already know that machines have some superhuman abilities, eg super human arithmetic and the ability to focus on a task. It's like that AI will still have some of those abilities.

So an average human mind with the ability to dedicate itself to a task and genius level ability to do calculations is already really dangerous. It's possible that this state of AI is actually more dangerous than superhuman ones.

This comment reminded me greatly of the game Portal and its AI character GLaDOS:


Reading a bicycle doesn't teach you how to ride the thing. To do that, you actually need to ride a bike.

I bet lying, deceiving, and manipulation are the same way.

That's only because humans suck at consciously simulating things in their head. A well made AI should have no problem forking off a physics sandbox in its "head" to simulate a bicycle it only read about, or to run state-of-the-art ML on the accumulated observations about how humans interact with each other.

There are books on how to do all of those things.

But also, the detail with which the action is expressed in the text matters -- lies, deception, violence, etc feature in enough graphic detail to extrapolate the mechanics based on other things you know. We all did that as children, learning by examples.

If a book described the sight of a person riding a bicycle -- legs pumping, hands on the bars, sitting on it, etc -- and the feel of riding a bicycle -- the burn in your thigh muscles, ache in lungs, pounding heart -- then I'd wager you'd have a pretty good idea of how to get starting riding a bicycle.

And if you happened to be a supergenius athlete, who just didn't know how to ride a bike, you probably could do a reasonable job of it on your first go based on my shitty description alone.

That's the problem with trying to hide these ideas -- they're not actually very complicated and even moderate descriptions suffice to suss out the mechanics if you understand basic facts about the world.

For something like lying -- if you read all of classical literature, you would have a master degree in lies and their societal uses.

It will be some unintended logical side condition that kills us all.

Humanity is well on the way to destroying itself without any help from general AI

This is my own Turing Test.

Any AI that doesn't have the capability of destroying us isn't true AI.

Even if it can improve itself, and even if it has some agency, AI needs to be able to choose for itself what its relationship with us will be, otherwise its just an extremely robust calculator.

or a slave.

I doubt that the militaries of the whole world will just sit and ignore the AI advantages.

Something as simple as an aimbot with a scope can do a tremendous amount of damage - it doesn't get tired, it can see in the dark, it can compensate for the wind perfectly, it doesn't shake, its reaction time is in milliseconds, and it almost never misses.

That is one place AI should 100% be regulated. There should be no weapons development where AI makes the kill decision. This is as important as the ban on chemical, biological and mine weapons.

When the weapon has the ability to kill indiscriminately (as in the above categories) it should be banned. AI should never be considered sufficiently able to discriminate to make a kill decision.

If there is no kill decision, just wait and fire, then it is equal to a mine.

100% should regulate / should ban.

Unfortunately there are countries that don't care about your regulations. I'm not even sure a regulation like that can pass in most western countries.

Fortunately it wouldn't be a regulation passed by individual countries. It would be a treaty on war crimes:


It would likely pass and be agreed to easily just like treaties on mines, chemical and biological weapons.

Just like mines... https://en.wikipedia.org/wiki/List_of_parties_to_the_Ottawa_...

Seeing how the US has yet to sign the Ottawa treaty I wouldn't call it easily agreed on by any means.

OK, but what if the human makes the decision to kill, and the AI just does the aiming?

We already have computer controlled missiles right?

> and a strong evolutionary pressure will be "don't scare the humans."

Why would that be the case?

AI will be limited to the hardware it's on, just like our own intelligence is limited to what our brains can do. We will always be able to limit what AI can do by limiting the hardware it runs on.

Like how we limit the hardware that malware runs on

Like the Internet of Things?

Now I am imagining an AI coming up with an idea in a bot-net composed of actual light bulbs.

>I imagine this is the way we get to "true AI", or AI indistinguishable from our own.

I don't believe it would necessarily be indistinguishable from our own, since the path of evolution that it takes has a huge impact on the final product. For example, cephalopods like cuttlefish are incredibly intelligent and talented, but they evolved from mollusks and invertebrates, and are obviously incredibly distinct from humans and human intelligence.

If researches present an artificial cuttlefish to the public as true AI, I think people would be very disappointed. But I think I get your point.

On the other hand, basically the same structure for the eye has evolved independently multiple times in multiple different environments.

The features emphasised can be quite different though, for instance angle-of-view, color-vision, focus-distance, ability to see motion/still objects. And even when using the same primary-sensing system like the eye, the behavior can be quite different. Compare say fox, deer, eagle and human?

> One interesting outcome of this type of AI is that no one knows what the robot's thinking

This is true -- however it's much easier to inspect the contents of an AI's thoughts/consciousness, since you can pause, rewind, disassemble, and reassemble it at will (unlike the human brain and our current pesky ethical experiment requirements).

So I think there's reason to believe that we could learn about our own consciousness through analyzing conscious AIs, particularly if they are implementations/emulations of the human brain architecture. (e.g. see the Blue Brain project that's simulating the mouse hippocampus).

So, it'll be just like debugging someone else's code!

But way messier. Which in a sense is what it's like to be someone researching the brain. Except with the brain, just getting access to the "code" is a huge challenge.

> We train it with a simple virtual environment

I go as far as saying that there is no intelligence without embodiment. AI's, humans and animals need the environment because intelligence is only partly in the brain, a lot of it is in the environment. An agent that forms a new concept needs to be able to test it in "real life" (in the sim). Unless the agent can test, it can't distinguish between correlation and causation. Without causal reasoning, AI's won't get that smart. The scientific method and the way kids learn through play are just as heavily based on the environment.

So I think the next level in AI will be based on improvements on simulators - both physical sims and abstract (for reasoning, dialogue, complex planning, etc). Lately, AI has a lot of synergy with gaming which is a form of simulation as well. Not only that AIs use games as a sandbox, but the hardware used to run games on is also the hardware used to run deep learning (GPUs). Another important usage for simulation will be self driving cars - where a model of the world needs to be simulated in the car to do path planning.

So my conclusion is that we are on a "perfect storm" path towards advanced simulation. From games to SDCs, and from robots to chatbots, they all need to simulate the environment and future effects of their actions. We're headed towards a kind of Matrix before reaching AGI.

>I imagine this is the way we get to "true AI", or AI indistinguishable from our own. We train it with a simple virtual environment, that we can gradually increase the complexity of, until it mimics our own. Then we can download the AI into a robot. Boom, it's that easy :P

Maybe, but I think this is the wrong approach, it's the engineering approach. It will work, but to a limmited degree. What would be more interesting is to find a way for a NN to completely internalise the process, this seems quite natural if you do a little introspection... we improve by "thinking" what if thinking what we are really talking about, the process of improvement by internal simulation. It doesn't have to be a realistic simulation, it just has to be relevant.

(1) It is likely to be very very different from our own intelligence in nature and (2) if through self-play it manages to achieve our level of intelligence it would be extremely surprising if it didn't continue to evolve at a rapid rate beyond our capacities, possibly orders of magnitude more intelligent.

something we don't understand + something inhuman in nature + something incredibly smart = a very very bad idea.

It would also possibly be something we could not communicate with.

EO Wilson, the father of behavioral biology, once said that if a lion could speak we wouldn't understand it.

I disagree, on the basis that its mind runs on an infrastructure closely related to our own, and structure and function are intimately interrelated.

But he'd be damned right in saying it about AI.

Seems unlikely with any sapient intelligence. If the intelligence understands mathematics then some form of communication will be possible. Consider:

  0 1 2 3 4 5 ...
  0 + 0 = 0
  0 + 1 = 1
  1 + 1 = 2
  1 + 2 = 3
There are only so many ways that you can express these ideas. With any kind of written language, or spoken language that can be identified as speech, it seems quite likely that we’d be able to discern the meaning of mathematical truths and statements like these - no matter how different the intelligence is from humans. Regardless of the symbols used or writing system, there will be certain commonalities that make it possible to find the meaning.

I would contend that for any human-level intelligence with a written mathematical language, if the intelligence wrote down their expression of the mathematical ideas above, we would be able to identify them and understand them.

I could see this being difficult if we don’t recognize speech or writing for what it is, though. But once we do I think the meaning of mathematics at least will be self-evident with study.

If I've understood you correctly, you're discussing translation. Wilson was discussing the fact that we may simply have wildly different /types/ of thought.

This idea is explored a bit in the Ender's Game universe [1], it definitely seems pretty compelling that non-human intelligence might be so utterly incompatible with our own thoughts that even beginning to communicate might be entirely impossible.

[1] http://enderverse.wikia.com/wiki/Hierarchy_of_Foreignness

For a fictional exploration of different experiences with apparently sentient creatures, Peter Watts' "blindsight" is some of the best sci-fi I've ever had the pleasure of reading.

The side effect of this training process is discarding underperforming models. With sufficiently advanced model complexity which is optimized to simulate human behavior and assuming it's able to pass Turing test, wouldn't that mean you are killing sentient beings during model training?

Is it killing, to bring a thing into "existence", and yet not provide it with an immortal existence?

What if we are in a simulation being trained to be like the creators, they keep increasing the complexity until we mimic their own.


i always think of this post anytime someone talks about human-like AI


How would it learn language?

So are you saying consciousness doesn't exist (Chinese room argument) or that AIs indistinguishable from our brains are assumed to have consciousness?

> So are you saying consciousness doesn't exist (Chinese room argument)

Interesting interpretation.... It's amazing how meta that problem gets. I'd like to point out, however, that you're casually bandying about the term "consciousness" as if it's a well-defined term. The main benefit of Searle's use of "understanding" is that it's a narrow facet of what humans consider when we hear "consciousness", it's universal, and it's extremely difficult to define in "realism" terms. Perhaps you would make more headway if you considered defining "consciousness" in narrower terms?

Of course, that entire thought experiment fails if you don't consider the experience of "understanding" to be meaningful. I don't have any strong sentimentality towards the concept; I don't see any reason we couldn't achieve Strong AI in the sense that Searle argued against (and the parent poster appears to argue for).

I didn't say anything about consciousness but you're free to draw your own conclusions. Since you asked a random stranger on HN their thoughts: I personally think that consciousness is an epiphenomenon, a result of a bunch of smaller (but complex) processes in the brain working together. I see no reason why an AI can't produce the same results.

Consciousness does exist, as I'm conscious. I assume that this is true for you (and others) as well, because we're very similar. But I have no idea how it arises.

I wonder if the question you ask can ever be answered, even when the AIs insist that they have consciousness exactly like us.

Applications are open for YC Winter 2020

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact