> Spent over $400,000 in API priced tokens with GPT-5.6 Sol and GPT 6 Astra
Tangent here, but I think this bit is super interesting!
You could viably hire someone to do this work for that kind of money - I think the interesting thing is that substantially less interested/experimenting engineers would consider paying for a human to do this work, than would happily chuck a big amount of money into an LLM.
I don't have any suggestion about why that exists, but it's a strange and interesting contract.
I found the whole section super interesting... I'll copy it here:
I used a lot of OpenAI models to try and complete this port. In total I did over $400,000 in API priced tokens with GPT-5.6 Sol and GPT 6 Astra. They wrote over 1.3m lines of Rust over multiple months of /goal loops and never got past like 84% compat.
When I saw how little my Claude Code limits were burning, I figured it'd be fun to throw Opus 5.5 at this. It had a working v0 in 10 hours.
I assumed it kept using the code the Codex models wrote. I was wrong. Opus 5.5 started from scratch. It got further than Astra in 1/10th the time.
I let it keep going, and it definitely did. Total token spend was ~$24,047 of API spend over 2 weeks. I was using my Claude accounts, and it worked out to somewhere between 925% and 983% of my $200 plan weekly limits.
Expensive, for sure, but not that bad considering how much work has went into typescript-go.
> I assumed it kept using the code the Codex models wrote. I was wrong. Opus 5.5 started from scratch. It got further than Astra in 1/10th the time.
Ouch, that is brutal and honestly, quite embarrassing but confirms what I have been seeing for a while. Personally, I find output from current OpenAI models still very hard to parse (though it has gotten better vs the pre-trains from both labs in mid/late 2025), thus hard to truly understand, verify and get comfortable maintaining vs current Anthropic models. I do occasionally see a higher ceiling in well scoped tasks with OpenAI models at the cost of (frequently) deviating from the original prompt in (sometimes) very destructive ways.
Could be that this hard-to-read output doesn't just go over my limited capacity/skills but with current models can become simply impossible to untangle beyond a certain size even when one has (essentially) infinite resources via multiple subs and different models.
Would also work with my suspicions for why OpenClaw (mainly build with Opus 4.5 and its post-trains) has been this hard to truly "fix", requiring highly paid Nvidia engineers, multiple months, (literally) infinite resources from OpenAI including access to internal models and yet still holds records for CVEs. Heck, another one was found just 7 days ago after what I'd argue was one of the most extensive hardening sessions any piece of software has ever undergone.
Makes my (multiple) decisions to start from scratch more than once on a major reworking of the existing tabbing interface in Firefox a bit less painful. Learned with each, found gaps in my knowledge, thanked the amazing docs the Firefox devs have been maintaining for decades and while starting from 0 was painful, getting back to MVP is easier than ever. When I hit a point were I was starting to struggle to truly parse additions a model was making to the patches applied to Firefox source code (even if they worked), I always found that pushing even slightly beyond that would incur painful, but hard to notice regressions, introduce major DB maintenance burdens as some models struggle to understand that in development regressions and incompatibility are acceptable and schema transitions aren't needed pre-release (still a case with GPT-6.1 Sol, less so post Fable for Anthropic), make me uncomfortable concerning privacy/security/data loss prevention (especially as I have seen Fable 5 cut some privacy/proper data removal corners in simple CRUD) and simply take away my control about the implementation. Wouldn't feel right to release something in that state, what I have now is fully understandable and thus could be maintained even without models.
Still expecting bugs of course, massively dreading security findings or even worse, possible data loss given browsers handle some of our most important personal+professional data and will surely have taken some embarrassing approaches that might have a much more performant solutions when implementing an infinite canvas of webpages, but still, rather that then also knowing I wouldn't even know where to start understanding a feature.
More so if, like with ts-rust, even (nearly) infinite tokens couldn't get me unstuck.
> Ouch, that is brutal and honestly, quite embarrassing but confirms what I have been seeing for a while.
What's most interesting is that Opus wasn't told to toss out the Codex crap. It decided to do that on its own. Which might mean that LLMs are actually getting a reasonable taste.
I've seen on youtube that some guy implemnted game engine with Astra and Claude. Astra wasn't really given fair chance because it didn't decide to work as long as Claude and the guy didn't force it. But still, scripting code within the engine that came from Astra was chaotic, messy, piling up things, but the code that Claude made looked downright pleasant.
I think it's hard to know what to make of that. Observations with little insight.
Maybe it just needed to be prompted differently? Maybe Astra starting from scratch could have done it? Maybe Opus was somehow trained more on the Golang implementation.
Could be. Feel at 6k commits there is a lot to be learned here, might be useful to compare code quality, approach taken by each model and assess whether what Astra created was truly unsalvageable.
Prompting wise, would be interesting to know how much steering truly happened across the project, if one wanted the model to truly work without detailed input, there isn't much that could be done to improve. How much was lined out and set I'd love to know, don't see it anywhere in the commits though, just the AGENTS.md and a few other docs files. "Port TS compiler, checker and LSP to Rust, never ask any questions" would be pretty funny though.
This is Theo's project, and he has been very transparent about how he is only doing these things because api prices are heavily subsidized by subscription pricing. The reason people aren't willing to pay 400k to an engineer to do this work is because they aren't paying the AI this much.
I don't think I've seen an enterprise paying API prices doing these sorts of rewrites, and honestly, I think until that happens I will remain skeptical about LLM AI's viability as a profitable business.
Right now, LLM rewrites like this work if and only if there’s a very robust suite of tests to code against. Otherwise the port is uncanny. Enterprises are willing there’s just often too much complexity and not enough verifiability to make it happen.
He's not paying that much in practice. But even if he did, you couldn't hire 3 engineers (I'm assuming you'd get 3x 133k each) that could do this work in anywhere close to this amount of time. It would be a multi-month long project. So "to do this work" is not really comparable.
It was an experiment. Memes are literally in the motivation. The approach and timescales would look very different if he approached it as the main project in the company. So I wouldn't take the current timescales as a reliable description of anything.
To be fair, NSF grants for pure mathematicians were never as substantial as for applied sciences, but they would routinely reach 500k–1M territory. That's the amount going to a single proposal, and grant panels in, say, analysis would definitely spend millions per application season. NSF funding data is public and can be looked up.
I'm using past tense because iirc NSF has recently reduced their funding volumes, although I'm not following the situation very closely.
From what I've read they had PM's and a team, a small Manhattan style project, I've never heard of a maths project being tackled this way, it would be interesting to duplicate with people.
NSF by itself contributes $200-300M a year towards math grants. Universities spend a billion+. Add corporate funding and other sources (DoD, other government agencies, private foundations) and you’re easily looking at a few billion dollars a year just in the US going to mathematicians.
Tangent here, but I think this bit is super interesting!
You could viably hire someone to do this work for that kind of money - I think the interesting thing is that substantially less interested/experimenting engineers would consider paying for a human to do this work, than would happily chuck a big amount of money into an LLM.
I don't have any suggestion about why that exists, but it's a strange and interesting contract.