You assume that for small steps (I.e taking some noisy code and slightly denoising) you can make an independence assumption. (All tokens conditionally independent, given the current state).
Once you chain many steps you get a very flexible distribution that can model all the interdependencies.
A stats person could probably provide more nuance, although two interesting connection I’ve seen: There is some sense in which diffusion generalises autoregression, because you don’t have to pick an ordering when you factor the dependency graph.
(Or put otherwise, for some definitions of diffusion you can show autoregression to be a special case).
There’s a reason we have formal verification as the highest guarantee for software. To ensure that we have a complete assurance of what the program can and can not do, the semantic of each of its components needs to be known. Recursively.
A minor change in one token can change the meaning of the whole software. Programming is just trying to enforce semantics on instructions (how well is that done is software engineering’s realm)
An algorithm like merge sort is just semantic constraints. Which is why most books go with their own notations as code does not really matter.
At most, LLMs and diffusion can be regarded as fancy searches. But, what you actually want is semantics and that’s why you can design lots of stuff on paper. But we do it with the code editor because feedbacks are nice and libraries’ documentations (if they exist) lie about their semantics. And we read code because there’s nothing more complete about semantics than that.
Hm suppose for argument sake that feeding a batch of data through some moderately large FF architectures takes on the order of 100ms (I realise this depends on a lot parameters - but this seems reasonable for many tasks / networks).
Now suppose instead you have an CTM that allocates 10ms on the standard FF axes, and then multiplies it out by 10 internal “ticks” / recurrent steps?
The exact numbers are contrived, but my point is : couldn’t we conceivably search over that second arch just as easily?
It just boils down to whether the inductive bias of building in some explicit time axis is actually worthwhile, right ?
I think this is article is too generous about the use of stdio - I have found this extremely buggy so far, especially in the python sdk.
Also if you want to wrap any existing code that logs or prints to stdout then it causes heaps of ugly messages and warnings as it interferes with the comms between client and server.
I just want a way to integrate tools with Claude Desktop that doesn’t make a tonne of convoluted and weird design choices.
The thing is though, once you regulate crypto then what’s the point? You are left with a highly inefficient/expensive and immutable database.
Bitcoin solves (or attempts to solve) for exchange in absence of trust and regulation. But this is a stupid thing to solve for, because without trust and regulation you can’t even have a functioning society.
I'm 99% against crypto and think it's mostly a tool for wasting limited human innovation capital, enabling fraud, terrorist financing, ponzi scheme gambling, and pump and dump rug pulling.
There are a few areas where I find crypto interesting or useful:
- The existing banking and payments industry is too Christian / Mormon. It extra-judicially regulates anything it views as "vices". This industry is supposed to be dumb payment rails with hooks for FinCEN to stop crime. It's not supposed to be your pastor. If anything, business integration with regular payment rails would help them be better regulated.
- "Stablecoins" or whatever the hell they're called seem like a wickedly efficient way to move money between businesses and countries without paying large fees or having to wait for clearance. Ideally they can even cut Visa and the fintech monopolies out of the equation. This is more of a B2B rather than consumer / individual application, and it seems genuinely useful. It also seems compatible with the existing FinCEN / FINRA / AML regulations. If you look at it long enough, it doesn't even feel like crypto. Just a new type of efficiency.
The latter may make the former a non-issue, especially if the existing fintech industry receives more competition from upstarts that don't have to pay the legacy gateways and their frictionful fees.
> "Stablecoins" or whatever the hell they're called seem like a wickedly efficient way to move money between businesses and countries without paying large fees or having to wait for clearance.
This seems to be 'regulatory arbitrage' rather than anything concrete to do with the tech though. i.e. there's nothing inherent in a stablecoin that can't be done more simply, it's just currently a way to skirt bureaucracy and fees, which are already diminishing in many places.
> "Stablecoins" or whatever the hell they're called seem like a wickedly efficient way to move money between businesses and countries without paying large fees or having to wait for clearance. Ideally they can even cut Visa and the fintech monopolies out of the equation. This is more of a B2B rather than consumer / individual application, and it seems genuinely useful. It also seems compatible with the existing FinCEN / FINRA / AML regulations.
Yeah, these seem like a really good (potentially the only one) use case for crypto. Note that these are really just replicating dollar (and potentially euro) dominance for a new age, and whoever ends up winning here will probably be the new Visa/Mastercard (with all of the problems that entails).
>seems compatible with the existing FinCEN / FINRA / AML regulations
Barely. If $1m of crypto moves from one private wallet to another how do you know who's doing it or what the purpose is? Could be something legit, could be paying for drugs/terrorism - no definite way of knowing.
I was under the impression Circle with USDC was trying to be very FinCEN / AML compliant. That you had to agree to their terms to be able to use the token, that you couldn't transact off of a regulated exchange, and that they do plenty of KYC to know what's going on.
I'm not sure of the details to be honest. Certainly if I want to change normal US$ in a bank into USDC then I have to go through KYC etc. If I then transfer it to a private wallet and then send it to someone else's wallet for an iffy transaction they'd see on the blockchain it was me but not necessarily who the other wallet was.
A way to make it less traceable is to transfer it to an iffy crypto exchange, change it and then withdraw it somewhere else. Some of those are not very good at keeping records and the like. Like I use mexc who can change usdc and when I wanted to do my tax return they said they couldn't retrieve transactions over 18 months ago - they delete the data they say. And mexc isn't the worst.
> scams and financial crimes seem to be the only concrete use case of the technology
the technology is showing transactions on a public ledger literally (unlike tradfi), nobody is hiding anything (until we go to Monero and similar, but that's different). What's however happening, and we come back to the first point, is the lack of "rules" so that in effect financial "crime" is legal.
The thing this really hits home for me is how Apple is totally asleep at the wheel.
Today I asked Siri “call the last person that texted me”, to try and respond to someone while driving.
Am I surprised it couldn’t do it? Not really at this point, but it is disappointing that there’s such a wide gulf between Siri and even the least capable LLMs.
Siri poped up and suggested me to set a 7 minute timer yesterday evening. I think I did it a few times in the week for cooking or something. This is a pretty stupid suggestion, if I need it I would do it myself.
I have been developing with MCP for a few weeks now, making some small python and javascript servers to integrate with Claude Desktop.
I am yet to see a use case that wouldn't be better served with an HTTP API. I understand the need to standardize some conventions around this, but at the heart of it, all "tool" use boils down to:
1. an API endpoint to expose capabilities / report the API schema
2. other endpoints ("tools") to expose functionality
Want state? ("resources") - put a database or some random in-memory data structure behind an API endpoint. Want "prompts"? This is just a special case of a tool.
Fundamentally (like most everyone else experimenting with this tech), I need an API that returns some text and maybe images. So why did I just lose two days trying to debug the Python MCP SDK, and the fact that its stdio transport can't send more than a few KB without crashing the server?
If only there was a stateless way to communicate data between a client and a server, that could easily recover from and handle errors...
APIs aren't self describing by default, but it feels like a missed opportunity for REST and HATEOAS proponents to show the value in those approaches. I've always said REST is for human clients, and LLMs seem like they should be human enough for this kind of thing.
I personally think we should resist the temptation to separate out values from the discussion. Like, even if it was the case that some extremely high-level of inequality turned out to be "optimal" in some GDP-maximizing way, the question still ought to hinge on some notion of fairness or justice (in my view).
(Although, I guess if your metric is "citizen satisfaction" maybe that's not as terrible).
In any case, there are lots of case studies showing what happens at the extreme inequality end of the spectrum. As I mentioned in another comment, my favorites come from Piketty "Capital in the 21st Century" and Acemoglu & Robinson "Why Nations Fail".
In the latter, the case study on the rise and fall of Venice is particularly fascinating - huge economic growth due to inclusive economic institutions that promoted social mobility, followed by a downturn once the aristocracy moved to entrench their own interests at society's expense. This seems to be the central thesis of the book, although I'm only a few chapters in.
The parallels with modern US politics are pretty hard to ignore though.
I haven't read the examples you mentioned, do they point toward the source of the instability being extreme inequality, or the lower (economic) classes being unable to survive in dignified conditions? (read: can we tolerate great inequality if it comes with increased standards of living for everyone?)
I mean, it seems possible that with greater absolute wealth people might tolerate greater relative inequality but like … why run the experiment?
Anyway, that’s the path we are on - Piketty doesn’t hazard a guess about if/when society collapse, IIRC the point is just that things will only continue to get more unequal unless there’s a major crisis or some political intervention (e.g wealth or inheritance taxes - imagine trying to get the US to agree to that lol).
Also you can’t ignore relative inequality because wealth is strongly coupled to political power/influence.
Look at the head of DOGE - he spent $300m to get himself a top gov role, in charge of regulating his own business and restructuring US gov spending to suit his own whims.
In terms of percentages / orders of magnitude it’d be like your avg citizen spending $100 for a top gov role.
When I was at uni I read Thomas Piketty's "Capital in the 21st Century", and parts of his newer "Capital and Ideology" (although I never quite got through that one).
The big takeaway for me was that wealth inequality never improves without some major catastrophe (war, revolution, plague etc). The proposed model is really intuitive and compelling (tldr; return on capital has historically always been higher than real growth, which guarantees indefinite concentration of wealth until there's a crisis).
Last year's Nobel Economics Prize winners, Acemoglu and Robinson, tell a similar story in "Why Nations Fail", which I am working through at the moment. Although in their case, they seem be suggesting a more causal link between erosion of political and economic institutions and the collapse of empires.
I wish these ideas were more broadly accessible and understood. The real risk of total societal collapse should transcend any partisan fighting about ideal tax rates, government inflation/unemployment targets etc. Everyone has a common interest in there not being a violent upheaval (arguably the rich most of all).
Last I checked the numbers, current wealth inequality seems about as bad as it was before the great depression. And its not enough to just say "well, absolute wealth is more important". As others have pointed out, its not stable to have such huge relative wealth disparities. And that's before you even consider corruption.
The book "The Great Leveler: Violence and the History of Inequality from the Stone Age to the Twenty-First Century" by Walter Scheidel makes a similar argument:
> Are mass violence and catastrophes the only forces that can seriously decrease economic inequality? To judge by thousands of years of history, the answer is yes. Tracing the global history of inequality from the Stone Age to today, Walter Scheidel shows that inequality never dies peacefully. Inequality declines when carnage and disaster strike and increases when peace and stability return. The Great Leveler is the first book to chart the crucial role of violent shocks in reducing inequality over the full sweep of human history around the world.
I would also recommend "The Dawn of Everything" by David Graeber and David Wengrow. It explores "the roots" of inequality. A lot of interesting insights there.
It’s just that for N tokens, autoregressive model has to make N sequential steps.
Where diffusion does K x N, with the N being done in parallel. And for K << N.
This makes me wonder how well they will scale to many users, since batching requests would presumably saturate the accelerators much faster?
Although I guess it depends on the exact usage patterns.
Anyway, very cool demo nonetheless.
reply