On: The avalanche that hit math is coming for AI development

Dwarkesh has been skeptical about AI claims for a long time. In his interview with Dario earlier this year, he pressed him hard. So this is a fairly recent change of opinion.

What he says still doesn't map to my daily practice. I use Astra and Fable extensively and they need an incredible amount of guidance and harnessing to produce fairly simple systems. Any decent code is produced with tons of static tooling, agent files, harnessing, and even then, there are myriad mistakes. My main point of skepticism is whether software engineering is verifiable at all, once you factor in architecture, code design, let alone the human element in usability, requirements, etc.

As far as I'm concerned Dwarkesh could be prey of AI psychosis... I don't think a proxy metric like the average daily spend of AI researchers going up proves anything.

The technology has undoubtedly advanced but for me it's still a coin toss on whether we're in an exponential or an s-function.

Taking a step back, it's basically getting very good in one arena, with who knows how much money is being spent refining things. But nobody appears to be working on the other aspects of intelligence that I (as a lifelong sf reader) would expect from an AI, let alone a superintelligent one. There is no permanence of anything except at the "build the next model" level and some notes it keeps and can scan quickly. Instead of an intelligence that focuses on you for a moment, you're getting a new copy every time you interact with it after a certain amount of time. There is no "game loop". It's just there to respond to you and be ordered about. It will never reply in a text chat hours later wondering how your latest meeting went. It doesn't have a concept of concepts. It regurgitates words and code in helpful ways, but there is no understanding there. It doesn't know what it means when it says things, it's all just a (very impressive!) act.

I've been building neural nets in C as a past-time hobby, and I get the feeling that I've been peeking under the curtain too much. The magic is going away. I know they have been getting creative with training, different types of neural nets, and so on. I don't know that I want these organizations working on building an actual sf-like AI, because that might actually human-ending disastrous, but this side road they've gone down isn't going to get there from where I'm standing.

I'm also starting to worry that all I do is regurgitate words and have no real understanding either :)

Thank you for saying this. Sometimes I'm like, wow, this thing is way smarter than me, and others I'm wondering how it can be so dumb.

In my experience, s-curves are much more prevalent in nature than exponentials.

Counterpoint: A human lifespan on a thousand-year S-curve would see only exponential.

This is pretty much what we're discussing: inside it it's very hard to tell. You don't even need to expand the timeframes that much for the point to hold.

I understand. My point is that the timeframe of this S curve may in fact be that large. And the curve that steep.

IOW, if an S-curve looks exponential for your entire life, and the life of your children and their children and theirs, does it matter that it's technically an S-curve and not truly exponential? I would argue that no, it doesn't practically matter for the people in that scenario, both options are indistinguishable and change everything completely.

That also matches my observation, particularly in economics once you factor in a scarce resource. But it's not like I have a convincing metamodel to say whether that or the other distribution is more common. But neither do people who fervently believe in the exponential...