I take your point that planes don't fly by flapping like birds, but obviously they do actually have wings.
No it doesn't. I have arms. Certain machinery has arms. They may mean different things but they both have arms, just as both birds and planes have wings
There is also an audiobook version if that's your thing. I enjoyed it very much. Happy to see more progress from this group.
"I would regard it as one component, maybe with another dozen or half-a-dozen breakthroughs we're going to need like that.
"There's a lot more innovation that's required."
I presume this is the thing - I don't think he actually said 'dead end'.
Even if all you care about is general AI, there is still a lot of progress that needs to be made within DL.
I think the point is just that we should be open to other approaches because they will be needed as well
Suppose we build a 1000 NNs for object detection, and let them 'vote' for which model to believe. What would be the best voting mechanism? Probably a final voting layer, right?
This architecture can be described as a single NN where we disallow connections between 1000 separate parts of the network in the first n-1 layers.
What would be the advantage of this? I appreciate that the brain - might - do it, but unless you can show any indication of this with experiments, or argue for why it would result in a performance increase, what's the point? Anyone can come up with an inferior architecture and propose that it's what the brain does.
If the input is completely separate, then it makes sense to call it 1000 different NNs.
If the input is not completely separate, then the architure I described with a high percentage of dropout in the first layer seems like a reasonable description?
This doesn't address my quarrel with the theory though; Why would that be either superior performance wise or more biologically realistic?
In the scenario described, where we have some neurons coming ultimately from the hand and sense of touch, and others coming ultimately from the eye and sense of vision, and they disagree about what they're perceiving, they're not voting! First of all, it might be better described as a "negotiation", and in fairness I think their theory does envisage this (but they do use the term "vote", albeit in scare quotes).
Second, what this theory misses (as does a lot of recent work which takes neural networks as a jumping-off point) is the transition from sensory/sub-cognitive processing up to conscious/cognitive processing. If my hand feels a cat and my eye sees a coffee cup, then I'll consciously notice the contradiction and gather more evidence.
Separately, there's a somewhat dusty area of machine learning called "Boosting" which treats this problem of combining a bunch of different classifiers explicitly. You're exactly right that traditional DNNs can implement a pretty decent approach to boosting, but there are some interesting techniques from that community that don't fit as easily into the standard DNN data graph perspective. For example, check out BrownBoost, which lets you set parameters about believed noise in training data: https://en.wikipedia.org/wiki/BrownBoost
We'll probably need a way to break out of this fixed/given architecture mode for next level breakthroughs. Smth like "differentiating through architecture-space too"... (guess the marketing dpts. will call it "profound deep learning" or something even cheesier :P)
(But I'm totally clueless as to whether Numenta is doing anything useful on this or getting anywhere useful with what they're doing ...no time to research this and kind of way above my current knowledge and understanding :))
The brain is composed of two hemispheres, each with different regional faculties working together. Each region is responsible for producing different models, based on its responsibilities, sensory inputs, regulatory functions etc.
As a whole the brain is a massive neural network. But each region possess its own neural network (grid cells/columns), and I wouldn’t be surprised if there were sub-neural networks at play, some that are unique to each person, based on their neurological development, environmental conditioning, and overall personal adaptations.
It’s easy to see how these regional neural networks are responsible for specific statistical modeling, which then factor into other regional models, and so on, in order to arrive at the correct output response.
Also resonates with Gestalt theory/psychology, which makes a lot of sense from a complex systems processing perspective, I’m just not sure what current neuroscience says about it.
I think this theory is very promising for AI.
I look forward to seeing more of their applied work.
I recall reading about expirements with octopi concluding that individual organs have different "knowledge."
IE, if one eye learns to recognise an object the other one doesn't gain that recognition. When one tentacle learns a task, the others don't. This was, iircc, explained via their more decentralised nervous system.
No spine and autonomous decision making in limbs means that the brain must only control limbs mostly by sight. The brain doesn't have enough signal to control limbs by feel.
I seem to have built a mental model of the world that doesn't need the senses to maintain it. I can use a single sensory input (iiish - there's many neurons on the tip of my finger) to verify it.
That is not explained by the thousand theory of intelligence, or any other theory of intelligence we currently have (and we don't really have that many).
And, not to put too fine a point on it, a thousand years from now, once (and if) we exit the dark ages we are very obviously hurtling towards, we will look back upon these attempts to explain intelligence in the same way that we look today upon the attempts of alchemists to understand matter. With sympathy and a little pity.
If we touched the cup with our finger, but this time we didn't touch it with the tip of our finger but with the whole finger, we would vastly increase the chances of recognizing the object.
This means the brain doesn't hold individual representations of objects per body part, but it fuses what its sensors say into a single model and then tests all its sensory input (the input from all the senses) against what is stored internally.
If we grab something like a cup with our full hand, we would certainly identify the object as a cup, but it might not be a cup actually.
This means all our senses participate in object identification.
Thus I don't think the theory of thousand AIs is correct.
One additional interesting tidbit I've noticed is that when the brain decides on a belief, it seems to suppress alternative beliefs. For example, with the old-lady-young-lady picture, it's hard to see them both at the same time.
Hasn't our own progress towards understanding biological intelligence been a series of theories that supplant and reinterpret another? Don't we frequently find new biological mechanisms for information storage and computation that run parallel to presumed dominant paradigms in impactful way? Why need a discovery to be necessary rather than useful or inspirational? I don't think it's common for hard science communities to describe results in terms of what is required, as opposed to what's possible.