I'm very pleased to see Numenta on HN front page, as they're doing incredibly difficult and ambitious work without deep learning's spotlight. They take a philosophically very different approach: instead of warming up biologically implausible neural models from the 60s and hope that with enough data we'll reach artificial general intelligence (AGI), Numenta founder Jeff Hawkins (way before Hinton's or Hassabis's recent declarations of DL reaching a deadend) thinks we shall understand our biological neocortex better and reverse engineer it because it's the only piece of hardware most scientists agree is at the source of intelligence. Although planes don't have wings, we had to understand wing-flapping first to find better ways to fly. If you're interested, I highly recommend you follow https://discourse.numenta.org/
Furthermore, the mechanics of flapping flight is complex, of little relevance to airplane design, and not well understood until well after airplanes were invented. What aviation owes to ornithology is mostly from the observation of gliding birds.
Depends on what you mean by wings. The word wings here has multiple meanings, with commonality that they both are involved in flying. However, they differ in their operation, i.e. one thing flaps the other doesn't.
No it doesn't. I have arms. Certain machinery has arms. They may mean different things but they both have arms, just as both birds and planes have wings
If anyone is interested in some foundational reading Jeff Hawkins and Sandra Blakeslee published a fantastic book in 2004 called "On Intelligence: How a New Understanding of the Brain will Lead to the Creation of Truly Intelligent Machines".
There is also an audiobook version if that's your thing. I enjoyed it very much. Happy to see more progress from this group.
"Deep learning is an amazing technology and hugely useful in itself, but in my opinion it's definitely not enough to solve AI, [not] by a long shot," [Hassabis] said
"I would regard it as one component, maybe with another dozen or half-a-dozen breakthroughs we're going to need like that.
To me these quotes don’t support the idea that deep learning is at a dead end. We are only just beginning to get it to really work for real world problems and I think that will take off a lot in the coming years.
Even if all you care about is general AI, there is still a lot of progress that needs to be made within DL.
I think the point is just that we should be open to other approaches because they will be needed as well
This theory reminds me of Marvin Minsky's "society of mind" theory where multiple agents/parts of a certain brain center encode the same object/concept in different ways and have a vote/suppress mechanism for processing different inputs in a heterarchial manner. But his model was far more theoretical than what I see here.
I don't really understand why this theory is attractive. I hope someone here can help explain:
Suppose we build a 1000 NNs for object detection, and let them 'vote' for which model to believe. What would be the best voting mechanism? Probably a final voting layer, right?
This architecture can be described as a single NN where we disallow connections between 1000 separate parts of the network in the first n-1 layers.
What would be the advantage of this? I appreciate that the brain - might - do it, but unless you can show any indication of this with experiments, or argue for why it would result in a performance increase, what's the point? Anyone can come up with an inferior architecture and propose that it's what the brain does.
This is how random forests work. Each takes a different subset of features and builds a model. Then models are averaged for much better accuracy. Unfortunately, averaging seems to be the crudest mechanism for combining models and far from the best. There are more sophisticated approaches like boosting and synergy (of monotonic rules).
If the input is completely separate, then it makes sense to call it 1000 different NNs.
If the input is not completely separate, then the architure I described with a high percentage of dropout in the first layer seems like a reasonable description?
This doesn't address my quarrel with the theory though; Why would that be either superior performance wise or more biologically realistic?
I agree that a "voting" mechanism sounds all wrong, not only for the reason you said.
In the scenario described, where we have some neurons coming ultimately from the hand and sense of touch, and others coming ultimately from the eye and sense of vision, and they disagree about what they're perceiving, they're not voting! First of all, it might be better described as a "negotiation", and in fairness I think their theory does envisage this (but they do use the term "vote", albeit in scare quotes).
Second, what this theory misses (as does a lot of recent work which takes neural networks as a jumping-off point) is the transition from sensory/sub-cognitive processing up to conscious/cognitive processing. If my hand feels a cat and my eye sees a coffee cup, then I'll consciously notice the contradiction and gather more evidence.
To address your last question, I think they are proposing (but have not yet demonstrated) that the human neocortex works this way, and I imagine the motivation for being more biologically realistic (beyond its usefulness in learning more about the biology) is that biology is still outperforming AI in these tasks.
The takeaway I got from OP is that Numenta is promoting more redundancy in those first [n-1] layers than might be typical in (e.g.) existing vision systems. The paper the original post is about doesn't discuss value-of-redundancy in those terms, but suggests that having different modules with similar structure is useful for learning compositional models where objects are composed out of other objects. I'm not really in a position to say how well this aligns with SOTA techniques; intuitively it seems like you could reinterpret the algebra of those modules a few different ways and have the same function, so I'd want to see more proof-of-novelty.
Separately, there's a somewhat dusty area of machine learning called "Boosting" which treats this problem of combining a bunch of different classifiers explicitly. You're exactly right that traditional DNNs can implement a pretty decent approach to boosting, but there are some interesting techniques from that community that don't fit as easily into the standard DNN data graph perspective. For example, check out BrownBoost, which lets you set parameters about believed noise in training data: https://en.wikipedia.org/wiki/BrownBoost
...one can probably try to use smth like genetic algorithms to select/refine the N-1 layers ...current neural networks take the architecture to be fixed or at least "given" (by some outside process, which might itself be an NN, but still).
We'll probably need a way to break out of this fixed/given architecture mode for next level breakthroughs. Smth like "differentiating through architecture-space too"... (guess the marketing dpts. will call it "profound deep learning" or something even cheesier :P)
(But I'm totally clueless as to whether Numenta is doing anything useful on this or getting anywhere useful with what they're doing ...no time to research this and kind of way above my current knowledge and understanding :))
I know little about AI and nothing about neurology, but this 'feels' right to me, based on what I think is happening in my own head. I often feel like there's different bits in there trying to solve the same problem and sometimes getting different answers.
This is a good theory. It’s one of the first that resonates with my understanding of the brain.
The brain is composed of two hemispheres, each with different regional faculties working together. Each region is responsible for producing different models, based on its responsibilities, sensory inputs, regulatory functions etc.
As a whole the brain is a massive neural network. But each region possess its own neural network (grid cells/columns), and I wouldn’t be surprised if there were sub-neural networks at play, some that are unique to each person, based on their neurological development, environmental conditioning, and overall personal adaptations.
It’s easy to see how these regional neural networks are responsible for specific statistical modeling, which then factor into other regional models, and so on, in order to arrive at the correct output response.
Also resonates with Gestalt theory/psychology, which makes a lot of sense from a complex systems processing perspective, I’m just not sure what current neuroscience says about it.
I think this theory is very promising for AI.
I look forward to seeing more of their applied work.
I recall reading about expirements with octopi concluding that individual organs have different "knowledge."
IE, if one eye learns to recognise an object the other one doesn't gain that recognition. When one tentacle learns a task, the others don't. This was, iircc, explained via their more decentralised nervous system.
No spine and autonomous decision making in limbs means that the brain must only control limbs mostly by sight. The brain doesn't have enough signal to control limbs by feel.
I close my eyes and extend a single finger towards the general vicinity of my keyboard. When my single finger touches the keyboard, I know that it's touching the keyboard, even what part of the keyboard it's touching.
I seem to have built a mental model of the world that doesn't need the senses to maintain it. I can use a single sensory input (iiish - there's many neurons on the tip of my finger) to verify it.
That is not explained by the thousand theory of intelligence, or any other theory of intelligence we currently have (and we don't really have that many).
And, not to put too fine a point on it, a thousand years from now, once (and if) we exit the dark ages we are very obviously hurtling towards, we will look back upon these attempts to explain intelligence in the same way that we look today upon the attempts of alchemists to understand matter. With sympathy and a little pity.
...or the brain simply needs more information to identify the cup than the tip of a single finger provides.
If we touched the cup with our finger, but this time we didn't touch it with the tip of our finger but with the whole finger, we would vastly increase the chances of recognizing the object.
This means the brain doesn't hold individual representations of objects per body part, but it fuses what its sensors say into a single model and then tests all its sensory input (the input from all the senses) against what is stored internally.
If we grab something like a cup with our full hand, we would certainly identify the object as a cup, but it might not be a cup actually.
This means all our senses participate in object identification.
Thus I don't think the theory of thousand AIs is correct.
This is similar to what I've been thinking. I usually say "the brain is more like a board than a CEO".
One additional interesting tidbit I've noticed is that when the brain decides on a belief, it seems to suppress alternative beliefs. For example, with the old-lady-young-lady picture, it's hard to see them both at the same time.
I support this theory - it's so obvious I'm amazed it isn't a widely accepted theory already. Numenta has some of the best views on creating AGI after human intelligence, and their approach could let us create a brain without the hardware limitations that plague ours.
In excited about 1k brains and in general biological research to inform AI! But I don't understand why folks doing serious work on the biological/artificial boundary push claims about which mechanisms are "necessary" to produce intelligent machines.
Hasn't our own progress towards understanding biological intelligence been a series of theories that supplant and reinterpret another? Don't we frequently find new biological mechanisms for information storage and computation that run parallel to presumed dominant paradigms in impactful way? Why need a discovery to be necessary rather than useful or inspirational? I don't think it's common for hard science communities to describe results in terms of what is required, as opposed to what's possible.
