Very cool to see! I was active in the UO emulator communities back in the day but mostly with SphereServer. It is interesting to see here in the comments how many people were inspired to programming because of UO!
All the recent LLM advances would make for very interesting and very fun NPC interactions in a MMORPG today too. Even small player community servers could be viable long term because of the ability to seed complex interactions with NPCs into on-going story lines.
Pretty sure I still have the source to SphereServer sitting somewhere on my NAS. It was my first exposure -- in early high school -- to coding in a group and operating a Linux server.
Very few companies run the vertically-integrated stack because it is prohibitively expensive to do so with current NWP versus what you can sell it for with only marginal forecast improvements. I know several companies have tried this with integrating their own observation sources and ended up with worse performing forecasts. Oops.
I'm very interested to see how the ML modeling revolution changes this. The ability to perform global forecasts on a single GPU should make it cost competitive for more companies. I know several companies are already deriving their own weights for the forecasting component so that they can sell them. Google appears to be working on the next piece of the puzzle too with using ML for the data assimilation step, or skipping that altogether and using observations to go directly to forecasts.
There are a few groups working on leveraging observations more directly in the ML forecast models and skipping over the assimilation/analysis step. However, unlike the original ML forecasting problem (which, let's be honest - was grossly over-simplified by the existence of ERA-5, which has been treated as "ground truth" for the atmosphere and used to teach models how to simply go from state at t=1 to state at t=1+\delta t), there's reason to believe that incorporating the observations will be substantially more difficult, given the complexity and bounty of the observations themselves and the challenge of framing a tractable, useful ML problem on top of them.
The entire slate of commercial acquisitions planned or in progress can be found at [1]. It's pretty anemic; NOAA has spent far less than what folks were hoping they would. I think a major part of this is that the private sector really didn't have very many high-TRL observation systems that could readily be integrated into NOAA's assimilation and forecasting systems. Lots of planned constellations and ideas about things to do in the future, but just not that much stuff that was ready to package-up and deliver to NOAA. The most successful acquisitions for GPS/GNSS-RO and buoy/drone data seems strongly bolstered by the fact that these data were already readily assimilated by existing infrastructure.
The private sector has really embellished its capabilities to the detriment of the CDP and other programs. I think too many industry players saw NOAA's expansion here as a potential slush fund to fully subsidize their R&D, but again the TRL of planned observation systems was too low and so the system didn't really work efficiently. Classic policy failure - would make a fantastic case study or Master's thesis for someone studying weather in an STS program!
You buried a lot of great analysis in your white paper that is hard to find! The comparison to NOAA Atlas 14 really shows what to expect with this dataset and how to better use it.
My guess is that potential customers who know how to use this data with their flood model also know how to derive this data from the sources. You may need to compute inundation maps for X year return periods in order to reach customers who need this information but don't know how to use flood models.
Really nice website and backend though! It's so fast even given the volume of data. Very impressive
Thanks. I'll have a think about how I can make those comparison plots easier to find.
Yep, inundation mapping would certainly be useful to a much wider number of people. I'll have to look into the existing competition and work out whether there's space in the market for another player.
I can confirm that this dataset is indeed part of the AWS Public Dataset Program. We were finalizing some details, but this dataset is now listed in the Registry of Open Data:
Enjoy! I’m personally very excited about this dataset, and couldn’t be more impressed with the people, mission, and, well, everything at the Smithsonian.
Source: I work on the AWS Open Data team / had a small role in this (normal lawyerspeak caveats apply that my views and opinions are my own)
This is weird. The Smithsonian page [1] says "Data hosting provided by AWS Public Dataset Program", and there's the Amazon blog post you linked, but the data set seems to be missing from the registry Amazon publishes. [2] I guess no one submitted a pull request yet? [3]
All the recent LLM advances would make for very interesting and very fun NPC interactions in a MMORPG today too. Even small player community servers could be viable long term because of the ability to seed complex interactions with NPCs into on-going story lines.