Something about this reminds me of the 'Winning Ticket Hypothesis' in artifical neural networks: that some 'random' initializations prime a network far better for later faster learning. From https://arxiv.org/abs/1803.03635 - the abstract:
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.
We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.
We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
Are these innate patterns of activity priming mammalian brains in a similar way?
> Are these innate patterns of activity priming mammalian brains in a similar way?
A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning?
Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints, which is constrained by pelvis size which is constrained by the need to walk upright.
I have wondered about how true this is. Maybe our brains are just so much more powerful, or capable of much deeper understanding, that we require a much lower initial learning rate.
The lottery ticket could explain why some children learn to walk at 6 months, while others are closer to 24 months, with no different in intelligence or motor skills in later life.
Edit: As some people below have correctly pointed out, human babies could not walk within an hour for physiological reasons, but nor do they exhibit the basic motor skills, spatial reasoning or image processing required for walking. Many human babies even struggle to feed for the first 24-48 hours or longer.
Meanwhile baboon babies can hold onto their mother from birth while the mother climbs trees.
> this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints, which is constrained by pelvis size which is constrained by the need to walk upright
Interestingly, the EGG hypothesis postulates that the constraining factor is not pelvis size, but rather that neonates consume so much energy that we give birth just before a women's metabolic limits would otherwise be overrun (leading to both maternal and infant death). Based on this, the increased energy demands of our larger brains mean that we have to sacrifice neonatal motor skills or drastically improve the metabolic efficiency of adult women.
> In humans, maximum sustained metabolic rate is thought to be 2.0–2.5× basal metabolic rate (BMR).
> By 9 mo, metabolic demands of the fetus threaten to push maternal energy requirements beyond 2.1× BMR. Extending gestation by even one month would likely require metabolic investment beyond the mother’s capacity. Instead, the mother delivers and the neonate’s growth rate slows relative to its fetal growth rate, keeping both the offspring’s and the mother’s energy requirements in check.
> Instead, the mother delivers and the neonate’s growth rate slows relative to its fetal growth rate.
Worth noting how massive the growth rate is I think: Even after birth, the baby gains weight faster than it ever will again in its life. In absolute terms! In relative terms it's even more insane.
A 12-year-old boy going through puberty will go from 90lb to 100lb over a year, an increase of 10lb or ~10%.
The average baby boy will be born at 7.5lb and weigh 20lb at one year old, an increase of 12.5lb or ~270%. Nearly triple the size!
It's not though, that's my point. Even in absolute terms, weight gain from growth in the first year is still more than any other time in the average person's life.
How are you defining growth? Are you defining it by volume? Ie assuming the body mass density remains the same, how much does the weight increase (which means the body is volumetrically growing)?
If not--I am very lightweight right now, and I'm trying to put on body mass by eating more and exercising. While my volume will increase, my main growth is in density. My growth is not in fat, but actual muscle mass. Is this growth? If so, I have gone from 125lb to 140lb in ~2 months, which in absolute terms is obviously more than the baby (my goal is 160lb by the end of the year).
A few years ago, I went from ~130lb to 165 lb in three months--blame the free food at FAANG cafes during an internship. But this might be closer to what you were saying earlier--putting on fat.
Edit. I guess in the original statement saying that a person will never gain weight at this rate again there should be the explicit assumption that we're talking about natural growth, not growth due to external factors (ie more eating--intentional or not). In that case, that statement would be true.
I was just pointing out that a baby does not gain weight faster than it ever will again in its life, in absolute terms. That's all. Not sure why I got the downvotes and no-true-scotsman arguments in return.
Plus fat and some water, yes. And furthermore, to go from a 10 pound baby to a 100 pound adult, I will make the claim that a baby has to gain 900% body mass.
You talked about weight gain, in absolute terms. A novice powerlifter can easily put on both more absolute weight and absolute strength in a year than a newborn.
What I was trying to say is the baby is growing faster. If you stay at average weight throughout your life, like you're tracking along the 50% line on the growth chart, then your fastest weight gain is in your first year. On average that's when you gain the most weight (even in absolute terms), despite the fact that you're much much smaller than, say, a growing teenager.
But putting on 100% or even 200% of weight in a year is not normal or even possible for most adults without crippling their health. All babies do this.
> Interestingly, the EGG hypothesis postulates that the constraining factor is not pelvis size, but rather that neonates consume so much energy that we give birth just before a women's metabolic limits would otherwise be overrun (leading to both maternal and infant death). Based on this, the increased energy demands of our larger brains mean that we have to sacrifice neonatal motor skills or drastically improve the metabolic efficiency of adult women.
This seems weird. Fully breastfed babies require even more calories from the mother. (They are bigger than unborn babies, and they lose more heat.)
Having gone through it, while it's true you need more calories after the baby is born (I breastfed and pumped exclusively and was told it's about 500 calories per day vs 300 or so extra prior to birth) the strain on your body is lessened because you are producing a single item (milk) going to one part of your body that's drained periodically.
In contrast, by the third trimester of pregnancy, your total blood volume has increased by approximately 50% (3-4 pounds of additional blood), not to mention an extra 2-3 pounds of retained fluid, 2 pounds of amniotic fluid, and 1.5 pounds of placenta. All told, this is another ~10 pounds on top of the baby. Additionally, the mother / placenta is performing all respiratory and blood filtration functions on behalf of the fetus (the kidneys don't really do much prior to birth other than produce urine).
Personally, while breastfeeding and newborn care was exhausting, it felt more akin to training for an athletic event vs having another entity symbiotically tied to my system like pregnancy.
Perhaps it’s more than just calories or energy intake, but as well doing the breathing, immuno responses, digesting, filtering blood etc. for both mother and baby?
> Instead, the mother delivers and the neonate’s growth rate slows relative to its fetal growth rate, keeping both the offspring’s and the mother’s energy requirements in check.
Growth rate slows after birth, so that would account for some of that. Mother’s also would not need to carry around the weight of the baby, and would be more able to provide for the child.
Breast feeding is quite hard on women and many women struggle to sustain the child on the breast alone. There’s a history of communal feeding which adds new energy into the equation and also mixed feeding once the child is old enough. I would say it’s probably a continuation at the peak metabolic rate described in the parent post.
Most women I know who struggled with breastfeeding were due to inability to have the baby correctly latched. Once the baby is latched, women shouldn’t have a problem with supply. Mother spends at most 500 calories extra when breastfeeding which most of the time comes from the weight that’s gained during pregnancy. I myself was able to exclusively breastfeed for 6 months after which we started introducing solids.
I kind of doubt that, because if baby is after term, the risk is supposed to be childbirth itself being too dangerous. And they induce birth if the baby is too big too.
Perhaps the most relevant to this discussion is the "stepping" reflex. It isn't really stepping but if placed in a standing-like position babies will often raise one foot, put it down, then the other. The most interesting part to me is that this reflex stops by two months. Makes me wonder if it's an evolutionary leftover from our days as monkeys or something like that. It's all so fascinating.
Also the grasping reflex: if you place your finger in the palm of a newborn's hand, she will grasp it. It works almost without fail with any newborn baby. After a few months it completely disappears: grasping is no longer a reflex but a voluntary act, and as often as not she won't be inclined to do it.
I read that both failure to demonstrate these reflexes while newborn, as well as continued existence past a few months old, are signs that something might not be entirely right with the baby.
These primal reflex arcs are suppressed by pyramidal circuits from the brain as they mature, essentially overriding them. This is one set of neurological tests of damage to the higher spinal cord/brain in adults; those reflexes can re-appear due to again limited inhibition.
> Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains
Have you ever seen a new-born baby's legs?
They aren't even remotely strong enough to walk, no matter what the brain tells them. It takes those months to develop leg muscles. New-borns don't have them.
The brain size needs a huge head, so in a sense it is.
All the factors are interwined. The growth of the brain seems to have been a very fast evolution, The pelvis had no time to adapt, so giving birth early was the final solution.
With that in mind, I don't find fair to shame the pelvis, or the brain. The weakest link is the legs, but it's not their fault either since they had no time to mature.
Give one of those cocky foals a head of proportional weight and allow them to use only two limbs and see who's stumbling now!
Edit: newborns can't even balance their heads on their necks, they're extremely fragile.
The balance can also be trained relatively quickly, but newborns typically get very little practice.
If you start occasionally holding your baby up by the thighs when they are 4–5 months old (their back/abdomen muscles at that point are typically plenty strong), after about a month they will have the balance to hold their torso upright when sitting on an adult’s shoulders, for at least a while. By age 8–9 months they will be very stable and good at balancing for long stretches of time. (I recommend any new parent try this, not only for the baby’s sake but because carrying a baby on shoulders is dramatically less tiring for an adult than carrying a baby in arms, and less hassle than a stroller or baby carrier.)
People generally hold their babies from the top (e.g. with hands in armpits), in a way that they are passively stable. If you want to train someone’s balance, you want to force them to actively stabilize themselves from a default-unstable position at least a bit every day. Babies have very short torsos and small heads compared to older kids, with the effect that their torsos have a much smaller moment of inertia about the waist: it takes several times more muscle strength for a 3-year-old to lift their torso up.
> If you start occasionally holding your baby up by the thighs when they are 4–5 months...
Do you mean baby sits on my shoulder with his legs around my neck, I am holding on to his thighs, and he can hold on to my head with arms? We both face forward in same direction.
No, I mean you hold your baby by the thighs, facing you, and let him balance his torso upright by flexing his abdominal/back muscles. At first, the baby will not have the coordination to stay upright very well, but actively stabilizing himself will train that balance. Because you are holding the thighs, the baby only has to stabilize one joint (the waist).
Then after a month or two of practice (a few minutes per day, nothing too serious here) you put your baby on your shoulders, holding his ankles, and he will be able to hold his torso up with his abdominal muscles for at least a few minutes at a time. Practice this for a few more weeks and you will be able to walk around town with the baby on your shoulders.
Oh I get that. Kind of what my parents did that always pick up baby from waist(if he is sitting down) & holding him only at thighs/waist. I see some people pick up by from their underarms.
Based on my experience with our first child, these might be placebo exercises, and the babies might have met those goals on their own. Assuming you haven't done a randomized trial.
Based on which experience? My experience with now 2 young kids and their playground buddies is that whichever skills they regularly practice improve very dramatically compared to skills they don’t practice. The differences are not at all subtle.
If you get a few 3-year-olds and e.g. get one to practice kicking a ball for 10 minutes per day, another to practice riding a 2-wheeled scooter, another to practice hanging from monkey bars, and another to practice swimming, after a few months there will be a wide gulf between their abilities at those skills.
I have a 10 week old baby right now, he can absolutely stand on his legs if you hold him steady - but yeah, he has absolutely zero clue what to do from there :P Also he's got a kick that you really need to watch out for, strong little dude.
Interestingly they can also effectively communicate well before they can vocalise words. A baby can sign reliably at 7-9 months but typically won't be able to speak until 12 months.
Yes! Signing (albeit with a small vocabulary) was surprisingly successful with my infant daughter, over 20 years ago. I was initially skeptical, but it worked really well. Not sure if it reduced my infant daughter's communication frustrations, but it certainly helped her parents feel less frustrated.
Or maybe it's not the brain is to small, but that the brain is still growing and therefore subject to enough noise to make actions now impossible for the sake of the future.
>Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints, which is constrained by pelvis size which is constrained by the need to walk upright.
You have this a bit backwards- yes brain size at birth is limited by pelvis size, but this means that humans are born prematurely compared to other mammals. The entire body is basically premature and underdeveloped, not just the head/brain. If anything the brain is relatively over-developed, as it's size is the constraining factor in gestation length.
If I remember correctly, 18-24 months is the estimated gestation time humans would have if not restricted by pelvis size (which ironically is itself a result of walking upright), and this fits better with the idea of being able to walk closer to birth.
There's a chapter in "Born to Run" that ties human physiology and endurance and evolution and brain development together in a way that's so elegant, it almost has to be true. It covers the steps that lead to the need for a bigger brain and walking upright as a means for persistence hunting. Highly recommended reading, especially that chapter.
> Is it training initialization or is it a form of transfer learning?
I doubt it's randomly initialized. The brain has centers for motion which might be honed in at birth for these animals.
Let's not ignore genetics, which pre-populate weights and subnetworks in the form of motion-control centers in the brain. Once you consider the gene-brain interaction over the course of evolution as part of your training, the huge data required for modern ML starts to make more sense.
> Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints
I strongly doubt that human babies are physiologically capable of walking before 6-9 months regardless of their level of brain development. The entire baby is born many months "early" compared with other mammals due to the limitations of the human pelvis.
Hence me giving a range of 6-9 months. My niece was walking not long after 9 months, it's rare but not unheard of. A newborn baby simply wouldn't have the physical strength or rigidity, though.
Earliest I've seen was 9/10 months, that kid looked so strange to me, they could barely hold their head up as they tottered around like a drunk. I wonder if the parents promoted early walking?
People do that successfully, the problem is that if babies skip the crawling phase then it may cause learning disabilities among other things. Which is fascinating in the context of this conversation. The order in which things are learned can be very critical!
Yes, successful crawlers might walk later, kids that learn to bum-shuffle (propel forwards whilst sitting on their bottoms) might not crawl much.
Our eldest wasn't interested in crawling really and went almost straight to walking but not until about 14 months IIRC (it was > 10 years back).
We did baby sign (a sort of simplified BSL; would recommend) and some people suggested it would retard speech - same child modified a sign before 12 months and so could tell us he needed a poo/wee and we'd sit him on the potty to do it all before he was able to talk - saved me changing a lot of stinky nappies! Longitudinally, our kids used baby sign and did/are doing well at school and have all been ahead with language skills (but that's not accounting for confounding variables).
It's curious to see infant developmental stages becoming more and more to be compared to AI/robot development.
I think the problem is they lose a lot of weight early on (can roll over) and then begin to fatten up quickly. The calorie intake relative to body size is immense and for a few months a baby just doesn’t have the muscle strength to support its newfound fat composition (loss of ability to roll) in addition to their heads growing rapidly early on and not having the neck muscles developed yet.
Intel is an Old baby who walks with crutches. Intel delivered the 10nm Processor just 2 months back after a decade of failure. Their 7nm is still broken and so is there 5 and 3nm in research. And they hope to catch up to the competition by 2025 is a funny joke. Intel isnt a reliable delivery partner which is why Apple moved away to its own processors.
Maybe given the length & complexity of a full human life, a bunch of brain factors actually work to slow specialization down to prevent early overfitting.
> that we require a much lower initial learning rate.
The way I understand it is that is not so much required as not selected for. We could certainly be able to walk at one day old if evolutionary pressure made it necessary, but it prioritized other aspects in our cognitive development to survive so we don't. It's not as if we say "oh, horses can walk early and therefore are incapable of rational thought" we evaluate them on their own merits.
Is it? If we ignore compute requirements, how long would a good algorithm take to learn something like balancing a quadrupedal robot, based on real-time feedback to its outputs? It is a simpler problem, but drone flight software based on learning, can re-learn how to fly a quadcopter after something like losing a propeller and a sudden shift in weight distribution, in a few seconds.
A robot could be built that learns how to adapt to something like a limb being lost or added or half its weight shifted to the other end. It could probably learn an approximate optimal to move under those conditions in just seconds, as well. Though, I suspect that foal might actually be nearly as adaptable. Adult horses, or humans... not so much. We have many overlapping models of how to move, probably. I would take only seconds to adapt to half my weight being added on my shoulders, in the same ballpark as our best robotic systems. But if my left leg grew four inches it would take me a lot longer than the robot to learn and internalize the best way to move again. I could barely keep up with that when it was just a few inches a year when I was a kid.
> Is it? If we ignore compute requirements, how long would a good algorithm take to learn something like balancing a quadrupedal robot
Seems easy but why don't we have self driving cars by now?
Walking it not just a mechanical task. Your quadcopter is not learning to interpret raw electrical signals from an IMS and raw electrical signals from a CMOS sensor. The training is also probably done in a simulator, where if it breaks, it is reset and training continues.
A horse is learning to interpret inner ear signals and signals from the optical nerve AND the using those to build a spatial map of the environment. Only then it is actually trying to walk.
A 2 hour old baby horse can not only run on not-flat terrain but is spatially and environmentally aware. It has taken years for Boston Dynamics to develop a platform that is equivalent to a pack horse and even then probably can't respond independently to never-seen-before threats or challenges.
Because the average first time driver is primed with approximately 15 years of training data from observation of not just driving deriving a complex physics model, object detection and identification system, spacial reasoning, natural language comprehensions, theory of mind able to predict behavior of other drivers based on subtle cues, and teen drivers are still awful at it as a rule with much higher accident rate.
Trying to build a neural net or set of neural nets that can do all of those jobs is a daunting task and no one has the time to train the artificial neural network for 15 years. that's why we don't have safe self driving cares yet.
as for the horse, well 50 millions of years of evolution on constantly refined genetic algorithm is hard to beat and that was on top of the evolution of the eohippus as a starting point
We do have self-driving cars we just don't have any we think are good enough, well-tested, and adaptable to mixed-use driving and our various regulatory and safety and cultural needs. You can't get regulatory approval for a car that learns to drive by guided trial and error in the wild -- which is the most direct way of training such a thing. Feedback on what you shouldn't drive over also isn't as immediate and unambiguous, compared to whether gravity will topple it over.
> If we ignore compute requirements, how long would a good algorithm take to learn something like balancing a quadrupedal robot, based on real-time feedback to its outputs? It is a simpler problem, but drone flight software based on learning, can re-learn how to fly a quadcopter after something like losing a propeller and a sudden shift in weight distribution, in a few seconds.
Firstly, the drone software is not re-learning anything; when a baby walks unaided for the first time and you place a rattle in their hands, do you really refer to that as "re-learning walking"?
Secondly, forget about computational requirements and just look at how many iterations have to be run to reach success; mammals (and humans specifically) can "learn" to recognise an entire class of something just from one or two images of that something. Learning something else doesn't "forget" the old thing. A NN trained to recognise faces in images can't, to me knowledge, be trained to also balance a bi-pedal robot, or ride a bicycle.
IOW, what I see currently is that NNs are very one-dimensional - the net that becomes optimised for one task becomes unoptimised for that task if you subsequently add new tasks.
> A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning?
Or maybe having 4 legs (as a foal) gives more natural balance than a human or maybe even a cat or dog.
In the same way technically you only need one leg to walk (hop) around but that's much more difficult than having two.
However, another commenter brought up the fact that human muscles are somewhat underdeveloped after birth. It might make sense that having 4 legs is easier both in regards to balance, as well as because the load per leg decreases to make standing on them easier.
That is probably not the only reason, but probably a contributing factor.
> A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning?
Probably neither. How does the human baby "learn" to breath almost immediately after birth? I suspect the answer here is also the same.
> Edit: As some people below have correctly pointed out, human babies could not walk within an hour for physiological reasons, but nor do they exhibit the basic motor skills, spatial reasoning or image processing required for walking. Many human babies even struggle to feed for the first 24-48 hours or longer.
> Stepping reflex
>
> This reflex is also called the walking or dance reflex because a baby appears to take steps or dance when held upright with his or her feet touching a solid surface. This reflex lasts about 2 months.
Maybe not enough to qualify as "exhibit the basic motor skills"
> Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains...
Isn't it just a matter of priorities? Many birds spend a similar proportion of their life unable to fly, but they're safe in a nest. If you're a foal on the savanna, you need to get up.
Interesting. Could this also partly explain child geniuses? In general they are worlds ahead early in life. But they aren't significantly better then the non child genius peers later in life.
I guess it's firmware of some kind. Newborns already know how to breathe, how to control the heart rate and tons of other low level things. Legs movement is a small feature in comparison.
I guess I can see some parallel the way you describe it, but it is not that tight of an analogy IMO. Correct me if I'm wrong but the Lottery Ticket Hypothesis mainly means that, for many of our tasks, we start with needlessly huge networks, and thus a good portion of the learning process is just the network learning to "cope with the noise" from the extraneous connections
Hence why the lottery ticket network (being a smaller version of the same initial network) can achieve similar accuracy (but not 'better') in much shorter time - because you took away all the confounding stuff. That kind of 'counterfactual' doesn't really translate to a biological domain, right?
Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right? Neither is the fetus brain compared to the baby brain (which is more to the point of the article). So the analogy breaks down there too
I guess what the article describes is a bit like "pre-training" a full network with 'synthetic' experiences that quickly tune it for later. In that sense, I think its like these "dreams" are a 'distilled dataset' [1]. The question then is: how is this dataset being passed on from mother/father to child?
> Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.
> Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right
I think it is the other way around -- adult brain is a physically pruned version of baby brain. We start with lots of connections from everywhere to wherever and then lose a lot of them as we learn, while strengthening useful ones.
Insects that have large migration patterns - all of the individuals die before coming back to the initial spot; only their descendants continue the cycle. How?!
Maybe epigenetics? I mean blank slate theory is off the table anyway...
In addition think about how bees can tell harvest locations to each other by dancing: https://en.wikipedia.org/wiki/Waggle_dance This dance is probably also something those bees know before being born.
In the context of real life we draw a distinction between thus neonatal dreamlike state and post birth "reality". But in the context of a neural net, I don't think such a distinction is necessary. They both sound like they're training the network. Still interesting to consider the parallels though.
So, that sounds like a good reason to iterate over a bunch of random initializations and train them up to find those awesome resulting networks that are smaller but have the same accuracy (if I'm understanding that correctly and it's actually possible to isolate or take advantage of those subnetworks for a final smaller size).
Or... has anyone tries training a network for determining the initialization state of another network? And now I'm wondering if those good starting initialization values only work well for a specific task or if they seem to span all or a subset of tasks, and maybe there's some inherent quality in how the values relate that we can tease out...
In the days when Sussman was a novice Minsky once came to him as he sat
hacking at the PDP-6. "What are you doing?", asked Minsky. "I am
training a randomly wired neural net to play Tic-Tac-Toe." "Why is the
net wired randomly?", asked Minsky. "I do not want it to have any
preconceptions of how to play." Minsky shut his eyes. "Why do you
close your eyes?", Sussman asked his teacher. "So the room will be
empty." At that moment, Sussman was enlightened.
I think it is less relevant to Winning Ticket Hypothesis (in my opinion, the most important paper on deep learning theory in the last 5 years), but about architectures that work to some extend with random weights.
> Inspired by precocial species in biology, we set out to search for neural net architectures that can already (sort of) perform various tasks even when they use random weight values.
Depends on the size of the network I would guess. A Boltzmann brain would apparently take about 10^10^50 years to form from quantum fluctuation in a vacuum. Our hypothetical neural network would be less complicated than that, assuming we're not aiming for general AI, so I think we can safely take that as an upper bound...
With a uniform initialization, presumably you'd never need much more than 2^(network bit complexity) number of samples to come across an ideally-initialized network with decent probability, so that's 2^10^24 for anything that fits in a yottabyte.
To put this into context, retinal waves (essentially self-propagating excitations along the retina) pre-birth have been known to exist and help neural organization for what must be 20-30 years. What's new in this paper is that they additionally confirmed that the direction of the propagating waves coincided with the mouse's typical optical flow pattern, essentially helping priming motion-detection neurons in addition to the well-known edge-detection neurons early.
I wouldn't compare this to dreaming really, it's a very different process where top-down connections hallucinate sensations. This is a completely bottom-up process, it's more or less like a test-bench connected to the optic nerve generating moving edges pre-birth.
> A new Yale study suggests that, in a sense, mammals dream about the world they are about to experience before they are even born.
"In a sense" ... A bit of a click-baity title for what amounts to suggesting that brain wiring is primed for sensing the world, which is not news.
The study itself looks to be more about how the brain primes itself (visual systems specifically). It may or may not have anything to do with dreaming.
Really fascinating. Does this mean the world "view" is encoded into the DNA somehow and that then gets transformed into neural activity?
How could a few billion pairs of acids encode such a world?
What's even more fascinating is that emotions, behaviors, imagination and dreams must all be encoded and not learnt with just a billion-odd pairs / bits.
Meanwhile, it takes a few megabytes for us humans to encode a decent Hello World program in a modern programming environment.
Dreams help the brain in generalizing the world. From the article:
"That is, dreams are a biological mechanism for increasing generalizability via the creation of corrupted sensory inputs from stochastic activity across the hierarchy of neural structures."
I always wondered how our brain is pre-wired to recognize 3D objects right after a birth, but this article is saying that eyes are giving fake image-like input to our brain so that it can perform "pre-training"? Does this imply that our brain is truly a blank slate if these initial seeds aren't given?
I don't think brains are pre-wired to recognize 3D objects.
There was an example case of a blind man with cataracts getting it fixed at around 50 years old. People were curious if he would be able to know if a sphere was round just by looking at it without touching it.
IIRC - not only could he not do that, but he couldn't visually interpret shadows (saw them as black splotches - didn't recognize depth), and was confused why objects got smaller as they were moving away.
I think the human visual system trains on a lot of visual input data, but it typically happens at the baby stage where you can't really interact.
Fascinating anecdote, and I'd love to see the report on this. I wonder what else they learned from such an interesting case study.
But I doubt it's 100% "no pre-wiring". What about gravity and musculature? Surely brains must know or train on these in some way before birth.
And what about fight or flight reflexes and presupposing monsters in the shadows? My understanding is that these had a primitive evolutionary basis and that we came hardwired to respond to certain stimuli.
Yeah - I just meant specifically 3D visual modeling. I think some fears are known to be innate among apes - fear of the dark, snakes, and falling. Also paying attention to faces?
The cataracts thing was an article I saw on HN a few years ago, but don’t remember the title.
> A NEUROLOGIST'S NOTEBOOK about Virgil (pseud.), who lost his eyesight as a child, and regained it at age 50. Tells how he could not adjust to the sighted world, and eventually had to be hospitalized.
Unfortunately, it needs a New Yorker account to view.
Gravity is probably also learned at the same time we learn how to use our muscles from how we must compensate for it when moving, also from seeing how things have a tendency to approach the ground unless something gets in the way, and from the sound things make when they hit the floor.
I don't think there's much opportunity to experience gravity before birth in order to train it, and I can't think of anything the brain would need to know about gravity beyond how we experience it with our senses.
> And what about fight or flight reflexes and presupposing monsters in the shadows? My understanding is that these had a primitive evolutionary basis and that we came hardwired to respond to certain stimuli.
Maybe pre-wired behaviors can depend on non-pre-wired stimuli, like a function that's conditioned on an undefined function and evaluates to false on exception.
sorry but I don't think there is any sense in which the infant brain is a blank slate. Definitely not for vital functions (as seen by the "instinctual" abilities noted elsewhere in this thread, not in language/cognition, and IMO not even culturally/morally.
The researchers call it "dream-like" and it is, in the sense that, like a dream, the neurons are stimulated and activated as though sensory input was occurring despite there being no actual stimuli.
Foetus can have, and will have, direct experience of the world before birth. In the last months of pregnancy they definitely can hear what is happening outside. Will also react to strong noises or gentle pressure.
Seeing differences at daylight or obscurity while in the womb does not seem impossible. Anybody that has closed the fist over a strong flashlight knows that the flesh is not totally opaque.
Is this dreaming? Maybe, maybe not. Other explanations are possible.
The headline is kind of beautiful, even if wrong. In a /r/BrandNewSentence kind of way, but also subtle commentary about cognition and our place in the consciousness landscape.
"Mammals dream about the world they are about to experience before they are born"
Wow, those are some evocative feels in those words.
I immediately thought about Google's "DeepDream" and how reversed back propagation gives us a projection of the network. Growing brains similarly fire before they're trained on real world perceptual inputs. Maybe these have shapes that evolved to be world-like and give babies an advantage in the fitness landscape.
My next thought: what if what we're collectively doing now -- all of our neural connectomes growing, dying, forming new connections, evolving understandings. Society, basically -- is the same shape of an AGI before birth? Maybe alien worlds tend towards this path too. A bad analogy, sure, but maybe it's the shape of the universe's consciousness being bootstrapped, before it awakens. Not getting into metaphysical garbage -- just thinking about what's to come.
I guess the suggestion that fetuses might dream probably seems outlandish if, for some reason, you were already committed to the idea that fetuses are "not alive."
I find it highly implausible that many people believe fetusus are "not alive". You might be referring to the widely held position that an early-term fetus isn't sufficiently developed to be considered a full-fledged human, and that any rights it might have don't outweigh those of the mother who is carrying it?
I've found that "fetuses aren't alive" is a common talking point espoused by people who aren't comfortable sitting in a moral gray area and want an simple excuse to resolve their cognitive dissonance. It's common enough that I can't help but wonder if that is the reason people are so resistant to the idea that fetuses might dream. It seems remarkably similar to "farm animals can't feel pain". A transparent fib people tell themselves to feel better about their positions. Incidentally, I eat meat and I am pro-choice.
Maybe so, though this too seems like a strange belief, and touches on the reason I find surprise at such things odd: it's been known for years that fetuses at late stages have cycles of REM sleep and apparent wakefulness.
It's easy for me to believe that this could have initially evolved to give babies slightly more preparation for the outside world (and therefore a slightly higher chance of survival), and that we just happen to retain the ability throughout adulthood.
Once upon a time, I woke up from a dream and was greeted by friendly beings. They looked at me in wonder, and were surprised I’m awake. The lights in the room were very cozy. Soon I went back to sleep, and there I was, veritably back in this world again. Now I do not know whether I had taken LSD and visited a different world, or I had temporarily woken up from this dream, and now making sense of the waking up as “a LSD experience”.
Alternatively, you were dreaming the whole time. I have _many_ dreams where I "woke up". I'm totally convinced it's real in the dream-- even though I also have telekinesis and can fly.
I've had dreams where I'm lying in my bed, trying to fall asleep. I wouldn't have known I was even dreaming if I didn't stop to check.
For those wondering how to check if you're dreaming: pinch your nose closed with your fingers. If you can inhale through closed nostrils, you're dreaming.
I recall once hearing a talk that discussed, what I recall as, "test patterns" being run across the visual cortex prior to birth. Essentially, waves / patterns of stimulation would go across the optic structure that would be used to, again as I recall it, "calibrate" the visual system.
I dug a bit and was able to find this paper https://pubmed.ncbi.nlm.nih.gov/15289028/ , which seems to track my memory. It has a number of forward and backward citations if you are interested. Perhaps someone with far more than my vague recollection of a talk will step in though!
Are these the same patterns one can see by applying pressure on the inner corners of the eyes? Massaging my tear ducts makes me see strange black/white/gray/color patterns that change over time.
Is it possible that there's some kind of neural transfer from the mother's brain during pregnancy? I've been wondering if there is or if it's all genetic. Evolutionary it would be very interesting if consciousness wasn't genetic, but transferred during pregnancy.
I'd be pretty curious about the mechanism for such transfer. The umbilical cord doesn't contain nerves AFAIK (and even if it did, the whole point of placental reproduction is to isolate the embryo/fetus from the mother, so probably no nervous connection there, either), so that'd preclude a direct connection between their nervous systems. Maybe hormones or some other chemical means?
Parent organisms do influence the way their children develop, at what extent is a different question (I understand it is fractal spaghetti all the way down).
It seems planaria worms can do something like that [0].
I think you missed the point: they're going to be born and have to deal with reality in any case. There would be no possible consequence of a failed unit test.
The woman has no idea if the baby is passing "unit tests" or not, and so no abortion can depend on hypothetical "unit tests." Quit with this made-up nonsense.
To dream about the world that they are about to experience, that's truly amazing. Imagine that the newborns are not aware of the dangers, cope with their surroundings, and become familiar with them.
Maybe telepathy/ESP is real, it’s just that we lose it soon after birth, or in early childhood, like eidetic memory. Or perhaps some people retain it longer - or some animals, hence all the mythical beasts.
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.
We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.
We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
Are these innate patterns of activity priming mammalian brains in a similar way?