Damn, this is a strikingly simple modification. Basically, modern deep learning optimizers typically calculate the update to the weights each step using some kind of momentum and/or LR scaling based on the running variance of the gradients. This means that, in theory, the actual "instantaneous" gradients from a particular backward pass might point in a different direction than the actual update the optimizer applies. The change the authors propose is to simply ignore any parameter updates proposed by the optimizer that have the opposite sign of the current gradient from the most recent backwards pass. They're essentially saying "only apply the long-term stabilized update where it agrees with the current 'instantaneous' gradient." They show that this simple change significantly speeds up model training.
I'm pretty intrigued by this, but will, as usual, wait for independent replications to come out before I fully believe it. That said, because of how simple this is, I'd expect such replications to happen within 24 hours. Exciting work!
I wonder if there mioght not be an opportunity for a warmup based mask inversion: for the first few epoches, only apply the momentum agreeing with instantaneous - after that, invert it since the momentum would technically have more info?
In any case, good idea - reminds me of the "apply same gradient multiple times" trick from a few years ago. May have weird behaviours at low batch sizes though...
In a hilarious act of accidental satire, it seems that the AI-generated audio version of the post has a weird glitch/mispronunciation within the first three words — it struggles to say "GPT-4.5".
I don't think there is such a legal obligation, because videos like this one exist where devs do share the exact financials of their Steam releases: https://www.youtube.com/watch?v=mPMfcaUulYU
"Redesign Your Logo" was actually directly inspired by this exact brand redesign document, so your instinct is well-founded! Here's a transcript of the official Spirit Phone commentary track where Neil mentions it:
As a bi person who's currently a man, but considering transition, I can't help but wonder if maybe all the bi guys are just... picking up and leaving to become bi gals :P
I think nonbinary people are the most likely to be attracted to many genders.
And women are more likely to be attracted to many genders.
Men are more likely to be attracted to just men (and perhaps nonbinary people) or to just women (and perhaps nonbinary people)
But I think this is due to social conditioning more than something like biological predisposition. Social attitudes on men and masculinity are not very encouraging of same-sex attraction, it's very common for men to get lumped into "gay" or straight". Whereas I think women and non-binary people are more often encouraged to explore queer sexuality or even expected to (well, I suppose gender-queer people by definition have queer sexuality also). Men are also more conditioned to be more competitive in general, and to view other men as sexual/romantic competition.
I'm a man who's a little bit queer, and many of my friends are queer, so my experience is likely influenced by my crowd, but I know many more gender-queer, nonbinary, gender-fluid, and gender-nonconforming people than I know trans-men and trans-women. The AMAB people I know who aren't gay men, and have nevertheless recognized some degree of attraction to men, very often are not male-identified. But I think it's much more common for them to be gender-queer than to be women. But again, this may coincide with me knowing more NB people than binary trans people in general.
For AMAB people who acknowledge their attraction to men (but are not gay men), I do think this awareness of a sexual identity that, in men, is less socially encouraged/understood often leads to questioning the value of identification as a man entirely (especially since the male identity has so much baggage already).
I've been surprised by how differently people experience desire after transitioning. I've seen every possible combination of sexual-orientation switching in trans people. If there's a general direction, I'd say that people are more likely to be bi/pan after transitioning. IMO, transitioning is already taboo, it's not that much scarier to explore being attracted to a wider group of people.
This reminded me of Bishop's Castle in Colorado, USA — an incredible project built almost entirely by one man (who sadly died last year) working on it nonstop for 40 years:
I visited this site once with some friends on a road trip... the guy building it (Bishop) started screaming the N-word at some black bikers and then calmly told them he wasn't racist and made some long rant about the government. We split very quickly as the dude was racist and crazy.
Wikipedia cites a travel book statung that while the place is "major fun", it warns that Bishop is a very unabashedly opinionated and potentially offensive person and will even bring up politics if you visit the castle.
That front stairway is like a bowl of brown M&Ms at a Van Halen concert. Given the code violations on the stairway (no landings [1]), I wouldn't trust the rest of the construction, especially the balconies.
Most fun at a rave I've ever had was at this castle. The guy built it free climbing the whole time and was in conflict with the government over how he got the rocks, which he mostly pulled from riverbeds on public land.
Another out there spot is in Lucas, Kansas. That guy was also fiercely anti-government.
Surely he would also be in conflict about building codes? How was this not stopped? Or is it all actually permitted?
To be clear, I like that this castle exists (although I also like that building and planning codes exist, so I guess I’m conflicted…), I’m just very surprised!
Building codes are extremely local, and not really federally regulated much. They’re consistent most places because jurisdictions will just copy paste them.
It does sound a bit like the Cheval’s Ideal Palace, well worth a visit as well (and also in France like Guédelon, though not in the same area): https://en.m.wikipedia.org/wiki/Ferdinand_Cheval
As a queer person who hasn't used Grindr but has heard about it from others, my understanding is that it has a reputation for being more for hookups than actual dating. Heard a story about a gay male friend-of-a-friend setting up an account on it and getting 4 unsolicited dick pics within a very short period of time.
Obviously this is just secondhand anecdotal evidence—I'm sure someone who has personally used the app could chime in and correct me if I am wrong.
Hell yeah. They do strike me as opening up some exciting possibilities—the fact that we can now do math on semantic information (something that has famously evaded prior attempts to do math on it) is really intriguing.
This reminds me of the antics of streamer DougDoug, who often uses LLM APIs to live-summarize, analyze, or interact with his (often multi-thousand-strong) Twitch chat. Most recently I saw him do a GeoGuessr stream where he had ChatGPT assume the role of a detective who must comb through the thousands of chat messages for clues about where the chat thinks the location is, then synthesizes the clamor into a final guess. Aside from constantly being trolled by people spamming nothing but "Kyoto, Japan" in chat, it occasionaly demonstrated a pretty effective incarnation of "the wisdom of the crowd" and was strikingly accurate at times.
I'm pretty intrigued by this, but will, as usual, wait for independent replications to come out before I fully believe it. That said, because of how simple this is, I'd expect such replications to happen within 24 hours. Exciting work!
reply