I think part of the problem is that it's easy to draw a trend-line across a batch of data you actually know about, like oil reserves or food supplies, but incredibly difficult to make educated estimates about how we will resolve constraint problems in the future.
Naturally, the most rigorous analyses are going to be heavily weighted toward historical data, but will carefully ignore predicting future breakthroughs and paradigm shifts. Any analysis that tries to highlight humanity's adaptive ability would be laughably ignored, or at least minimized as being overly speculative.
So here we sit, with our broken pessimistic models, because they're the only ones the data supports.
It's not just the pessimistic scarcity predictions that suffer from long term extrapolation. Optimistic ones of abundance have missed the mark just as badly. Alvin Weinberg, writing in the June 1970 issue of the Bulletin of the Atomic Scientists, made an optimistic case for the very large scale use of breeder reactors -- 160 terawatts of breeder reactor capacity installed by 2050. His physics are perfectly sound. But the demand growth which this scenario assumes has not materialized.
He imagined that the world might come to demand 20 kilowatts per capita (more than twice the present per capita use in the US) and have 20 billion people living in it. Neither population growth nor energy consumption per capita growth have kept up with his estimates. Accordingly, nuclear fuel reprocessing and the breeder reactor alike have remained niche technologies without broad industrial impact. There was never enough uranium consumption growth to economically justify complicated reactors or fuel cycles.
It is difficult to make predictions, especially about the future.
Naturally, the most rigorous analyses are going to be heavily weighted toward historical data, but will carefully ignore predicting future breakthroughs and paradigm shifts. Any analysis that tries to highlight humanity's adaptive ability would be laughably ignored, or at least minimized as being overly speculative.
So here we sit, with our broken pessimistic models, because they're the only ones the data supports.