I'll agree but from the other direction. AI continues to absorb my job as a senior systems software engineer (c/c++) and after a couple months I've only spent a few hundred dollars using gpt-5.5/5.6 and codex. I have no idea what people are doing to burn so many tokens but for me this is laughably cheap and every day I discover new capabilities. I don't care if costs go up or down, it's so cheap for what I get that I don't care.
> I have no idea what people are doing to burn so many tokens
Agentic workflows is what consumes a lot. When you have an automated agentic loop working towards a given goal. If you use an LLM as a support for your own work you don’t end up consuming that much tokens, if you have multiple agents working on things independently, reviewing the work of other agents, etc you very, very quickly burn all your budget
I learned that running goal for hours produces exponentially more slop than running targeted prompts over and over again manually.
Personally, I use gpt 5.5 high with planning every time and plan various smaller features/changes in parallel, then approve them one after another. This allows me to steer it (which I need more often than not) before approving the plan, thus reducing the otherwise accumulating slop.
Using goal doesn't work for everyone, unless you have an unreasonably strong test suite or harness that the agent can verify against.
Like a junior engineer, I find the models to be too ambitious and unable to steer themselves at a high level yet. What I’ve done to address this is prompting the model to break down its plans into more atomic steps. For whatever reason, they’re still lazy at planning.
Agentic loops are being promoted by the same people selling tokens, abstracting away the cost per token, and doing everything in their power to obfuscate costs.
I think a senior dev/architect + some good models is still the goated combination.
Generating code and building features, even before AI, was never the issue. Stability, knowing what to build when, and boring business problems (licensing, distribution, sales, etc) were the limits.
It is the difference between giving an LLM an epic and say "You figure it out" and giving the single tasks' breakdown you envisioned and build incrementally on top of it.
With the latter you can, for example, say "Wait, this should be an interface because later on we need different concrete implementations". With the former, the agent doesn't do that, gets to the point where you actually need the flexibility interfaces give you and refactors everything to handle that. That is at least 2x the work/tokens. Multiply this for all the decision points you have to do to deliver a big piece of work and you have your bagillion tokens consumed.
Work on a project where you can verify the functionality instead of reviewing the code in any detail.
Use worktrees to parallelize development on multiple tasks.
That's all there is to it.
In many cases, this means a new solo project rather than a project at work with a team.
In my iOS app with around 100k LOC, Claude Code typically uses 150k context for small tasks.
For tasks that take longer and run the tests to instrument and investigate outcomes, the context grows to 250k-600k. With a few of those in parallel, busy days can consume a lot of tokens.
My numbers come from the spend my company makes using our corporate plan. Not sure what the pricing is relative to any other plan. It's definitely not the consumer rate as we're a pretty big shop burning a lot of tokens as a whole.
Agentic workloads are the most batch friendly, latency insensitive, geography insensitive, migration insensitive tokens that a big lab ever sells. In the ads business such inventory is called "remnant". The sausage is made of whatever is left over when the choice cuts have been removed.
This talking point from Anthropic that Claude Code sitting in a Ralph Loop is burning top sirloin interactive session tokens is bad faith hogwash and it only flies because most everyone who has run this shit at scale either already works there, sells them hardware, or hopes to be an acquisition target.
I'm none of those things, so I'm happy to tell you they're lying. I know, it's hard to swallow, but it turns out Altman and Amodei are occasionally full of shit.
Interactive use cases: the web interface, the mobile interface, the design tool. The fast variants.
In an HBM bandwidth constrained setting you're dealing with something called "roofline analysis" (comes originally from NUMA work circa ~2009 but it's applicable to modern GPUs). Great diagram from the JAX people:
In order to get your money's worth from a modern GPU (or disagg rack like an NVL72) you need to decode (the one token at a time thing) across big batches of context windows. To the left of that point where it hits "the roof" you're idling tensor units. TensorRT-LLM likes batches of 4096, so BS=4096.
In the case of one person chat prompting their local LLM, BS=1, totally bandwidth limited.
So the game is to set some latency target with some control theory primitive (PID or something) and then delay the next token until a batch is big enough to not waste tensor units. This is a real trick when a human is waiting (you've probably seen the thing in Claude.ai where it's all bursty and then they reflow the whole block with JavaScript).
Agentic workloads are huge piles of context windows where you've always got enough who want the same experts on the next token, you're always to the right of that intersection. And it doesn't really matter if it's on the other side of the world, or lags by a second, it's fine.
Claude Code soaks up all the tensor units that would be idle until they're full, and only then does it leak into the capacity reserved for highly interactive use. It's the bottom of the barrel until it's rinsed the fuck out.
They want more margin on agentic tokens. That's it. The COGS on them is the absolute lowest of anything they do.
Not correct. I'm assuming it's an enterprise plan. I have no idea what deal my company has worked out with OpenAI but I do not yet have a personal account (but soon will when I have the time to spin up proxmox on my compute brick that's been gathering dust since last year).
Why is everyone still operating under the assumption the current token costs will remain so heavily subsidized? We could see $200-400/hr in token costs once these companies need to turn a profit
These are probably mostly the enterprise customers - they may use the same amount of tokens as you do, but they have to pay the API price. From my experience the API is significantly more costly. We had one user ask for and receive usage credits on Claude, the bill the next day was to the tune of $400.
That's around 100k/year if used at the same rate for every workday. So the question becomes: does it make your engineer X% more productive, where X is some multiplier based on their salary? There are some software engineers out for sure who are expensive enough that this is worth it.
To be clear, some of the token expense is because it's encouraged, and it's encouraged at some companies so that people will break out of their existing workflows to hopefully find useful new ways of working or building
There is more to it than this, but much of the cost structure around subscriptions etc is specifically designed to allow for that experimentation.
There are good cynical takes, here, too. At the current model costs I don't need to optimize my expenses, but that could change if it climbs eg above 30% of my salary^
Note: this is an easy thing to prove ROI on. If I'm writing 5-6x more code and reviewing commensurately more code, and those PRs are better-tested and get us to shipping quality features faster, this is easy to justify and we are not that price sensitive
Ignorance. Bad code hygiene and poor prompting. As someone who barely codes, I had a few old vibe coded projects from pre agent days when gemini ide had basically no usage limits that ballooned to multi 1000+ line files with backlog of bugs. I only stopped because dumb models starts breaking down at the point and project was serviceable for my needs. Come agentic coding and models smart enough to fix issues, but codebase is so filthy it does it wildly inefficiently. Like a few prompts would consume my 5 hour quota. Took a few days to get a decend agent.md up and refactor codebase etc and now I'm sipping tokens. I'm sure many people are still in that boat. Many of us literally don't know any best practices and can't tell agents how to behave.
In retrospect, I should have just spend a few days learning the basics, but you don't know what you don't know. And part of me can't help but feel companies aren't exactly prompting agents to be courteous when onboarding newbies because they want people like me to get hooked, and token maxxing on their end helps. I spent few $100 more than I should getting subs/tiers I didn't need, but at the time it was small $$$ for productivity gains from going from 0-1.
