Indeed you can't be sure. But on the other hand a bunch of the commentariat has been claiming (with no evidence) that we're at the midpoint of the sigmoid for the last three years. They were wrong. And then you had the AI frontier lab insiders who predicted an accelerating pace of progress for the last three years. They were right. Now, the frontier labs rarely (never?) provide evidence either, but they do have about a year of visibility into the pipeline, unlike anyone outside.
So at least my heuristic is to wait until a frontier lab starts warning about diminishing returns and slowdowns before calling the midpoint or multiple labs start winding down capex. The first component might have misaligned incentives, but if we're in a realistic danger of hitting a wall in the next year, the capex spending would not be accelerating the way it is.
Capex requirements might be on a different curve than model improvements.
E.g. you might need to accelerate spending to get sub-linear growth in model output.
If valuations depend on hitting the curves described in the article, you might see accelerating capex at precisely the time improvements are dropping off.
I don’t think frontier labs are going to be a trustworthy canary. If Anthropic says they’re reaching the limit and OpenAI holds the line that AGI is imminent, talent and funding will flee Anthropic for OpenAI. There’s a strong incentive to keep your mouth shut if things aren’t going well.
I think you nailed it. The capex is desperation in the hopes of maintaining the curve. I have heard actual AI researchers say progress is slowing, just not from the big companies directly.
> Indeed you can't be sure. But on the other hand a bunch of the commentariat has been claiming (with no evidence) that we're at the midpoint of the sigmoid for the last three years.
I haven’t followed things closely, but I’ve seen more statements that we may be near the midpoint of a sigmoid than that we are at it.
> Thy were wrong. And then you had the AI frontier lab insiders who predicted an accelerating pace of progress for the last three years. They were right.
I know it’s an unfair question because we don’t have an objective way to measure speed of progress in this regard, but do you have evidence for models not only getting better, but getting better faster? (Remember: even at the midpoint of a sigmoid, there still is significant growth)
I thought the original article included the strongest objective data point on this: recent progress on the METR long task benchmark isn't just on the historical "task length doubling every 7 months" best fit, but is trending above it.
A year ago, would you have thought that a pure LLM with no tools could get a gold medal level score in the 2025 IMO finals? I would have thought that was crazy talk. Given the rates of progress over the previous few years, maybe 2027 would have been a realistic target.
> I thought the original article included the strongest objective data point on this: recent progress on the METR long task benchmark isn't just on the historical "task length doubling every 7 months" best fit, but is trending above it.
There is selection bias in that paper. For example, they chose to measure “AI performance in terms of the length of tasks the system can complete (as measured by how long the tasks take humans)”, but didn’t include calculation tasks in the set of tasks, and that’s a field in which machines have been able to reliably do tasks for years that humans would take centuries or more to perform, but at which modern LLM-based AIs are worse than, say, Python.
I think leaving out such taks is at least somewhat defensible, but have to wonder whether there are other tasks at which LLMs do not become better as rapidly they also leave out.
Maybe it is a matter of posing different questions, with the article being discussed being more interested in “(When) can we (ever) expect LLMs to do jobs that now require humans to do?” than in “(How fast) do LLMs get smarter over time?”
Or are the model author’s, i.e the blog author with a vested interest, getting better at optimizing for the test while real world performance aren’t increasing as fast?
There are a few other limitations, in particular how much energy, hardware and funding we (as a society) can afford to throw at the problem, as well as the societal impact.
AI development is currently given a free pass on these points, but it's very unclear how long that will last. Regardless of scientific and technological potential, I believe that we'll hit some form of limit soon.
So at least my heuristic is to wait until a frontier lab starts warning about diminishing returns and slowdowns before calling the midpoint or multiple labs start winding down capex. The first component might have misaligned incentives, but if we're in a realistic danger of hitting a wall in the next year, the capex spending would not be accelerating the way it is.