This is still on retrospective data. The machine learning graveyard is filled with models that worked well on retrospective data, but did not hold up in a live inference setting. Just ask Zillow. The real test is whether they can predict the weather 14 days out in 2025.
I am guessing they did not want to set up the data pipeline to run inference in a live setting. But that is what I would need to see to be a true believer.
ECMWF runs many such models at their site, a run two or four times per day, and they have verification statistics too, no need to doubt the accuracy.
The Google model is probably the best so far but ECMWF's own diffusion model was already on par with ENS and many point-forecast models (graph transformers, not diffusion) outperform state-of-the-art physical models.
What is missing is initialization directly from observations. All the best-performing models initialize from ERA5 or other reconstruction.
> One caveat is that GenCast tested itself against an older version of ENS, which now operates at a higher resolution. The peer-reviewed research compares GenCast predictions to ENS forecasts for 2019, seeing how close each model got to real-world conditions that year.
And GenCast was tested against and older model which performs worse.
> The ENS system has improved significantly since 2019, according to ECMWF machine learning coordinator Matt Chantry. That makes it difficult to say how well GenCast might perform against ENS today.
And the testing makes it "difficult to say." The obvious conclusion is "run a new set of tests" but they'd rather pay of the verge to publish half truths instead.
It's like the super trading algorithms who achieve perfect scores during backtest.
The question is, how does it perform on unknown events.