Far too much marketing speech, far too little math or theory, and completely misses the mark on the 'next frontier'. Maybe four years ago, spatial reasoning was the problem to solve, but by 2022 it was solved. All that remained was scaling up. The actual three next problems to solve (in order of when they will be solved) are:
If there isn't a path humans know how to take with their current technology, it isn't a solved problem. It's much different than people training an image model for research purposes, and knowing that $100m in compute is probably enough for a basic video model.
Large latent flow models are unbiased. On the other hand, if you purely use policy optimization, RLHF will be biased towards short horizons. If you add in a value network, the value has some bias (e.g. MSE loss on the value --> Gaussian bias). Also, most RL has some adversarial loss (how do you train your preference network?), which makes the loss landscape fractal which SGD smooths incorrectly. So, basically, there's a lot of biases that show up in RL training which can make it both hard to train, and even if successful, not necessarily optimizing what you want.
This is only for review/position papers, though I agree that pretty much all ML papers for the past 20 years have been slop. I also consider the big names like, "Adam", "Attention", or "Diffusion" slop, because even thought they are powerful and useful, the presentation is so horrible (for the first two) or they contain major mistakes in the justication of why they work (the last two) that they should never have gotten past review without major rewrites.
No, they aren't. Locally, the person with the most esoteric knowledge is probably a weird nerd. It's mostly an accident that they chose to invest time in things typically associated with smarts. But globally, the best wizards got there by making it their profession. So maybe at your middling university, the people who could land a job at a frontier lab were nerdy wannabe frats, but at decent universities like MIT or Tsinghua, they're usually just better in every aspect of their lives. E.g. MIT has "math olympiad fraternities" all the cool kids join.
I went to a top 5 ranked school globally (~these lists fluctuate) and have been in elite circles since then. I can promise you that even there the autistic nerd fully outcompetes the renaissaince man.
We don't even try. In the US you demonstrate that you know the rules at one point in time and that's it, as long as you never get a DUI you're good.
For instance, the 2003 California Driver's Handbook[1] first introduced the concept of "bike lanes" to driver education, but contains the advice "You may park in the bike lane unless signs say “NO PARKING.”" which is now illegal. Anyone who took their test in the early 2000s is likely unaware that changed.
It also lacks any instruction whatsoever on common modern roadway features like roundabouts or shark teeth yield lines, but we still consider drivers who only ever studied this book over 20 years ago to be qualified on modern roads.
Some places will dismiss a traffic ticket if you attend a driver's education class to get updates, though you can only do this once every few years. So at least there have been some attempts to get people to update their learning.
> No amount of crashing results in traffic school, just certain kinds of tickets.
Well, sufficient at-fault crashing will suspended your license, and among other requirements for restoring the license may be traffic school, DUI school, or some other program depending on the reason for suspension, so this is not strictly correct. You can't use optional voluntary traffic school to clear points from a collision from your record BEFORE getting a suspension the way you can with minor moving violations without a collision, but that doesn’t mean collisions won’t force you into traffic school.
Which states/counties/cities? IME that rarely happens, tickets are often used for revenue-raising. And some recent laws e.g. 2008, 2009, 2025 CA cellphone use laws cannot be discharged by traffic school, AFAIK.
> Anyone who took their test in the early 2000s is likely unaware that changed.
That's silly. People become aware of new laws all the time without having to attend a training course or read an updated handbook.
I took the CA driver's written test for the first time in 2004 when I moved here from another state. I don't recall whether or not there was anything in the handbook about bike lanes, but I certainly found out independently when it became illegal to park in one.
I don't doubt that many people are aware of many of the new laws. But I strongly suspect that a very significant number of drivers are unaware of many new laws.
The model seems pretty shitty. Does it only look on a frame-by-frame basis? Literally one second of video context and it would never make that mistake.
What dates? 2000 August–2025 August gives 3.18%/year for the consumer index and 7.12%/year for egg prices. Even if you assume egg prices will come down to $3/dozen, it's still 6.24%/year.
Hmm, I think a mixture of beta distributions could work just as well as cateogrical here. I'm going to train it for PixelRNN, but it's going to take hours or days to train (it's a very inefficient and unparallelizable architecture). I'll report back tomorrow.
After another 24 hours of training and around 100 epochs, we get down to 4.4 bits/dim and colors are starting to emerge[1]. However, an issue a friend brought up is that log-likelihood + beta distribution weights values near 0 and 1 much higher:
This means we should see most outputs be pure colors: black, white, red, blue, green, cyan, magenta, or yellow. 3.6% of the channels are 0 or 255, up from 1.4% after 50 epochs[2]. Apparently, an earth-mover loss might be better:
E_{x ~ output distribution}[|correct - x|]
I could retrain this for another day or two, but PixelRNN is really slow, and I want to use my GPU for other things. Instead, I trained a 50x faster PixelCNN for 50 epochs with this new loss and... it just went to the average pixel value (0.5). There's probably a way to train a mixture of betas, but I haven't figured it out yet.
Okay, so my PixelCNN masking was wrong... which is why it went to the mean. The earth-mover did get better results than negative log-likelihood, but I found a better solution!
The issue with negative log-likelihood was the neural network could optimize solely around zero and one because there are poles there. The key insight is that the color value in the image is not zero or one. If we are given #00, all we really know is the image from the real world had a brightness between #00 and #01, so we should be integrating the probability density function from 0 to 1/256 to get the likelihood.
It turns out PyTorch does not have a good implementation of Beta.cdf(), so I had to roll my own. Realistically, I just asked the chatbots to tell me what good algorithms there were and to write me code. I ended up with two:
(1) There's a known continued fraction form for the CDF, so combined with Lentz' algorithm it can be computed.
(2) Apparently there's a pretty good closed-form approximation as well (Temme [1]).
The first one was a little unstable in training, but worked well enough (output: [2], color hist: [3]). The second was a little more stable in training, but had issues with nan's near zero and one, so I had to clamp things there which makes it a little less accurate (output: [4], color hist: [5]).
The bits/dim gets down to ~3.5 for both of these, which isn't terrible, but there's probably something that can be done better to get it below 3.0. I don't have any clean code to upload, but I'll probably do that tomrrow and edit (or reply to) this comment. But, that's it for the experiments!
Anyway, the point of this experiment was because this sentence was really bothering me:
> But categorical distributions are better for modelling.
And when I investigated why you said that, it turns out the PixelRNN authors used a mixture of Gaussians, and even said they're probably losing some bits because Gaussians go out of bounds and need to be clipped! So, I really wanted to say, "seems like a skill issue, just use Beta distributions," but then I had to go check if that really did work. My hypothesis was Betas should work even better than a categorical distribution because the categorical model would have to learn nearby outputs are indeed nearby while this is baked into the Beta model. We see the issue show up in the PixelRNN paper, where their outputs are very noisy compared to mine (histogram for a random pixel: [6]).
And paper money is a Chinese invention. Doesn't mean it's worthwhile to spend two weeks in an anthropology class talking about how much awesomer they are.
- Reinforcement Learning (2026)
- General Intelligence (2027)
- Continual Learning (2028)
EDIT: lol, funny how the idiots downvote
reply