It's puzzling that there's so much documentation, both about how to run it and the internals, while failing to explain what it does. What are the inputs and outputs? It looks like it does something with satellite photos?
After a bit of digging I found this in their FAQ on https://madewithclay.org/, it was the best answer I've seen to your question:
> Clay harnesses AI, satellite images, and other spatial data to organize information about what’s happening in precise locations around the world. We give Clay millions of satellite data and use the latest AI tools so it can supervise itself learning about Earth through those images. As it learns, we benchmark how those skills improve its capacity to do important tasks like creating land cover maps, detecting crops or burn scars, or predicting carbon stock.
Right now it seems that they've trained a true foundational model. It can recreate its training inputs, but that only matters as far as downstream performance goes. Based on a brief scanning of the repo & website:
- Their training process is a self-supervised MAE to recover the original satellite images and some additional metadata.
- They're trying to then use this for plastic pollution detection, deforestation, mining, etc.
I'd liken this to a llama-base. It's exposed to a lot of satellite data, but to do something useful really has to be fine-tuned (or clustered in the case of unsupervised embeddings). No word yet on the performance against these specific downstream tasks.
Thanks. It sounds like the input is raw satellite photos and output might be cleaned-up maps based on the data.
The page on "location embeddings" [1] seems interesting:
> In CLAY, the encoder receives unmasked patches, latitude-longitude data, and timestep information. Notably, the last 2 embeddings from the encoder specifically represent the latitude-longitude and timestep embeddings.
Perhaps it could figure out the latitude, longitude, and orientation from a satellite photo (geolocation).
> Clay v0 is a self-supervised modified vision transfer model trained on stacks of Sentinel-2, Sentinel-1 & DEM data. It is trained as a Masked Autoencoder (MAE) to reconstruct the original image from a masked image.
> Each data entry is a stack of 10 bands of Sentinel-2, 2 bands of Sentinel-1 & 1 band of DEM data. The model is trained with 3 timesteps of data for each location, with a total of 1203 MGRS tiles globally distributed, each of size 10km x 10km. The data was collected from the Microsoft Planetary Computer.
> The model was trained on AWS on 4 NVIDIA A10G GPUs for 25 epochs (~14h per epoch) in December 2023.
MGRS is Military Grid Reference System. I believe 5 characters corresponds to about 60 miles by 60 miles.
So presumably the result is an embedding vector representing e.g. a specific spot on earth around a specific 3-year period, such that you can compare that embedding vector with other points to get an idea for what's similar and maybe categorize them against known vectors for things like had-a-lot-of-deforestation?
This seems very cool but perhaps additional explanations vs non-ai are useful?
I'm having trouble visualizing benefits of using ai as opposed to traditional data analysis for these purposes:
- Track changes in forest cover
- Allocate funding for environmental
justice
I still don't understand what they do. I have some ideas, but their websites are surprisingly lacking in application examples. It reminds me a lot of crypto pitches on startup accelerator events of the past.
The FAQ claims: "Clay provides key infrastructure like pre-computed raw data analysis, apps, an API, cutting-edge models, and benchmark tools."
... but I cannot find most of these. EDIT: Found something - the Product section does, after enormously patience-taxing scroll jacking, show off a few model capabilities.
(Getting that text required using the inspector, their FAQ page - you know, the page with the information you want to be commonly known - breaks copy and paste.)
They have funding from the Radiant Earth incubator, so they've been able to convince someone of this, but it must have been with better material than this website. Maybe they should share that one :-)
