"Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it’s a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."
His videos also cover a super wide range of content areas. Just this latest one has a refresher on neural nets, ieee floating point (and explained the accuracy tradeoffs more clearly than I’ve ever seen), fractals, chess engines, and more.
It has been a long standing dream to make videos like this. I have no idea where to start though. I think everyone has these out of the box ideas, but we (or at least I) disregard them as "who will be interested in this".
Dude is way out of the box. Using IEEE floating point rounding error as his neural network transfer / activation function -- never thought to go there.
Tom7 is kind of like the electronic version of the Primitive Technology Youtuber.
It's fascinating watching someone using some of the worst tools ever to make something in about the most labor-intensive way imaginable - and it's just beautiful. It's practically meditative.
I know what you mean by Primitive Technology Youtuber, but I would encourage you to not dismiss the underlying concepts out of hand. Building a chair from green wood with hand tools may not work for Ikea, but it is much more energy-efficient than power tools [1].
There's also something incredibly empowering about working with tools you understand, both in software and woodworking.
Oh absolutely. I'm going to confess that I read this book, "Replay" by Ken Grimwood, which is roughly about a guy who keeps having a heart attack on the same day, and wakes up in his younger body, with all of his memories. Again and again and again. Kind of like Groundhog Day, but for an entire adult life. I kind of became obsessed for a while with the idea that I would like to live my life over again. It was a really unhealthy obsession. It took a while for me to live in the moment, try to not hold on to my regrets, etc.
Back to the topic... I kind of watch things like Primitive Technology out of a hope/fear that I'll either get thrown back in time to when these skills were relevant, or that I'll live in a post-apocalyptic world where they're relevant again. (Which was probably inspired by reading Stephen King's "The Stand," and worrying about how my life would be if there was no electricity.)
But yeah, I love watching people show their enthusiasm... I love watching people demonstrate mastery over tools I don't know how to use... I love watching someone build something... Tom7 and Primitive Technology check all those boxes. Adam Savage's Tested often does, too.
Industrialization and automation lead to higher human populations, so if your point was that the cost of supporting a human makes power tools more efficient than hand tools, I think you have it backwards.
sure, let's all maximise the time we have available to spend on a couch consuming mindless tv.
The "but it takes tiiiiime" argument against making your own things (and achieving a kind of satisfaction, control and learning that's hard to beat) is simply dreadful.
Who are you to decide what people do and don't want to do with their spare time? Some people don't achieve any satisfaction from making their own things; they just see it as a chore. Your disappointment in that is just your subjective opinion, not any reflection of objective truth.
You also seem to think that there's nothing "productive" anyone could do with the time they save from not having to make their own things, which seems obviously false.
Don't project your use of spare time on everyone else. Some of us use spare time towards productive ends. I would personally prefer not to spend all my time living like a subsistence farmer from 1200 years ago.
Tom has a pretty good day job, so I suspect that money isn't the thing.
The sigbovik conference seems to be the motivating goal behind these videos. I think the key thing would be to make sure that sigbovik continues and that people are excited about the papers.
Unfortunately, Tom's a lot more creativity and concept-driven than money driven. So the best strategy is probably to spend the money on some kind of near-C sublight vehicle and take a one-hour jaunt in it, then return to Earth in a year and see what's tickled Tom's fancy since you last saw him.
this solution may have side-effects that are left as an exercise for the Rocketeer.
I lucked out a bit in having only recently learned about the need for a nonlinear transfer function (although it was taught to me as a "That's just how we do it," not with the associated underlying explanation that you can't represent all functions with only linear functions).
I believe that this floating point imprecision was something that David Page used (I think it may have been accidental as he was originally doing something else with it?) to achieve a world record training speed on CIFAR10 by summing a bunch of loss values together instead of taking the average. The sum effectively reduced the precision of the loss value and seemed to have a regularizing impact on network training as best as I personally understand. :) <3
Tom7 back to his usual madness. This time, exploiting floating point rounding, first to allow a neural network activation function to be linear (except for rounding errors), then expanding until ultimately making a 6502 emulator and proving that linear operations + rounding errors are Turing complete.
"Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it’s a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."