I know this is for tokens/text, but can the same concept be applied to images using something like a diffusion model? And then be able to upscale images arbitrarily by infilling?
Yes. See the related work section in the paper: there is a long history of models, recently like MAE and MaskGit, which predict pixels in basically arbitrary orders, and that is useful because it lets you train on subsets of each image, upscale/infill during generation, and so on. (If you know what MAE is, that might be the fastest way to summarize OP: "it's a GPT trained like a MAE".)
People also often forget "orderless autoregression", which was introduced a while back and has been reinvented many times since. See Sec 4 (pg 8) of "Neural Autoregressive Distribution Estimation" [https://arxiv.org/abs/1605.02226]. The main difference from current work is that this 2016 paper used MLPs and convnets on fixed-length observations/sequences, so sequence position is matched one-to-one with position in the network's output, rather than conditioning on a position embedding. Of course, Transformers make this type of orderless autoregression more practical for a variety of reasons -- TFs are great!
Key quote from Sec 4: "In this section we describe an order-agnostic training procedure, DeepNADE (Uria et al., 2014), which will address both of the issues above. This procedure trains a single deep neural network that can assign a conditional distribution to any variable given any subset of the others. This network can then provide the conditionals in Equation 1 for any ordering of the input observations. Therefore, the network defines a factorial number of different models with shared parameters, one for each of the D! orderings of the inputs. At test time, given an inference task, the most convenient ordering of variables can be used."
I have no idea what you are talking about, given that you are linking to a comment a decade old which has nothing to do with Conway's death just now; nor have I commented on Conway in any way today or been on HN today until just now, where I saw your comment. (Nor do I intend to comment now that you have drawn my attention to her passing, as I generally feel that such obituary notices should try to celebrate the good about a person, as posterity will have quite enough time to consider the bad as well.)
If you disagree and wish it 'kept out', my suggestion would be to... not link it? Especially in other pages which have nothing to do with Conway?
I'm sorry, I confused you with a twitter account that used the same name as yours, as I was upset by Lynn's passing. I take back what I said about you being an asshole.
You are, unfortunately, not the first to be misled by my impersonator.
I've done what I can to get that account blocked: I have reported him for impersonation to Twitter years ago, after his previous accounts, and I have asked my followers to report the account to get it banned, and I have asked again just now. If any HNers would like to take a moment to report the false troll https://x.com/gwernbranwen1 account for impersonation of my real Twitter account, https://x.com/gwern , I would be grateful - there must be some number of reports at which he'll finally be banned...
But Twitter takes forever to ban impersonation accounts, so, 'gwernbranwen1' keeps on tweeting that crap.
(Why, you might wonder? I don't know. I have no idea who or why whoever is behind that account does it. They have never explained it anywhere I've seen, nor do they seem to particularly hate me or are trying to ruin my reputation; their main fixation seems to be misogynist hatred of a few women like Julia Wise. I've met Julia Wise all of once, who is a nice woman who I liked, but I otherwise have no particular connection to her, and I have no idea why anyone would want to harass her online like that or why they would do so under my name. It's just the Internet, I guess: https://gwern.net/littlewood )
