
Nearly two decades of satellite images available in Nasa’s Worldview - eaguyhn
http://blogs.discovermagazine.com/imageo/2018/06/10/nearly-two-decades-revealing-satellite-images-now-available-fingertips/
======
waffle_ss
A project I wish existed is an open source competitor to Google Earth's
automatic generation of 3D models from satellite images. Their generator is
pretty smart and even generates pretty good approximations of trees, bushes,
streetlights, and so on. This is very useful when you need to figure out 360°
line-of-sight from a point in space (i.e. a viewshed) for things like planning
where to put RF antennas as topographic maps don't take vegetation or other
obstructions into account.

Google Earth even has built-in viewshed generation which uses their 3D model
data (works great), but unfortunately the 3D model layer only seems to exist
for fairly populated areas (probably because it's computationally expensive) -
so doing RF planning in rural areas requires more traditional legwork.

So it would be cool for a project to provide similar functionality for
OpenStreetMap using this public satellite data. OpenStreetMap does have 3D
modeling support but last time I looked it seemed pretty focused on buildings
and people hand-editing the data.

~~~
backprojection
Google's 3D models are based on aerial imagery, which is higher resolution,
and suffers from much less atmospheric interference.

~~~
brandon272
The fact that Google has so many cities mapped for aerial imagery is very
interesting to me. Do we know how they get this aerial imagery data? (i.e. Is
it one or more subcontracted companies that specialize in aerial photography?)

~~~
oliveshell
According to [1]:

"The US National Agriculture Imagery Program (NAIP) offers aerial image data
of the US at one-meter resolution, including nearly complete coverage every
several years since 2003. Earth Engine includes this data as well as sample
imagery from several commercial providers."

[1]:
[https://earthengine.google.com/datasets/](https://earthengine.google.com/datasets/)

------
pul
Here's the link to what this article is referring to:
[https://worldview.earthdata.nasa.gov/](https://worldview.earthdata.nasa.gov/)

~~~
stinos
Nice. Is there a simple explanation somewhere of what the different bands
mean? E.g. I'm looking at my place now and we have this spot on our property
where the vegetation is quite different and this exact spot shows up clearly
on the NDVI layer. Houses etc do as well so could be temperature?

~~~
agcopenhaver
NDVI is a well established vegetation index that takes advantage of the
relatively unique spectral qualities of vegetation, in that it absorbs large
amounts of red light and reflects large amounts of infrared. It is the
normalized difference of a sensor's near infrared band and its red band. There
are numerous band combinations available to highlight certain materials within
a scene.

Here is a link describing some additional indices:

[http://www.harrisgeospatial.com/docs/AlphabeticalListSpectra...](http://www.harrisgeospatial.com/docs/AlphabeticalListSpectralIndices.html)

Additionally, here is a brief description of different individual bands and
how they can be used:

[https://gisgeography.com/spectral-
signature/](https://gisgeography.com/spectral-signature/)

~~~
stinos
Thanks for this, very helpful!

------
noiv
ESA's Sentinel fleet offers similar images with much higher spatial but lower
temporal resolution, here's a two days old nearly cloud free view of San
Francisco:

[https://apps.sentinel-hub.com/eo-
browser/#lat=37.7504&lng=-1...](https://apps.sentinel-hub.com/eo-
browser/#lat=37.7504&lng=-122.3142&zoom=12&time=2018-06-09&preset=1_TRUE_COLOR&datasource=Sentinel-2%20L1C)

------
gizmodo59
Is it me or the level of detail is not good in some regions even compared to
Google Earth?
[https://worldview.earthdata.nasa.gov/](https://worldview.earthdata.nasa.gov/)

~~~
noiv
Where available GE has images taken from planes updated after several years.
WorldView's images are updated every day and taken from satellites.

------
throwawaymath
This is great news for a project I'm currently working on:
[https://news.ycombinator.com/item?id=17237445](https://news.ycombinator.com/item?id=17237445).
I'll be sure to look into this and see what's new compared to what's already
out there. There is _a lot_ of public satellite imagery.

That said, I wouldn't be surprised if this isn't actually the first surfacing
of this particular dataset. My experience is that most data of this kind
(satellite, land station, oceanic/marine, balloon, radar, etc...) is already
available online. It's unfortunately mired in a sea of complexity, filled with
shifting departments, confusing acronyms, funding depletion and musical
chairs.

Here is my favorite example. The NOAA works directly with NASA for much of
this kind of data, and stewards it online here:
[https://www.ncdc.noaa.gov/data-access](https://www.ncdc.noaa.gov/data-
access). Observe that the URL indicates the department is the NCDC (National
Climatic Data Center). As it turns out, the website informs us that the NCDC
no longer exists, and is now known as the NCEI (National Centers for
Environmental Information).

But let's put that detail aside. Now we go to the page listed for satellite
datasets: [https://www.ncdc.noaa.gov/data-access/satellite-
data/satelli...](https://www.ncdc.noaa.gov/data-access/satellite-
data/satellite-data-access-datasets). Lots of acronyms - acronyms for
everything! Let's try the OISST (Optimum Interpolation Sea Surface
Temperature) dataset:
[http://www.ncdc.noaa.gov/sst/](http://www.ncdc.noaa.gov/sst/). Whoops, that
404s. It also doesn't tell us what the acronym stands for. Fortunately we can
find the canonical page by Googling for it.

Okay, let's try another dataset back on that first page. How about the
SSM/I-SSMIS (Special Sensor Microwave/Imager and Special Sensor Microwave
Imager Sounder):
[https://www.ncdc.noaa.gov/ssmi](https://www.ncdc.noaa.gov/ssmi)? Well, we're
told we have to go to [http://www.class.noaa.gov/](http://www.class.noaa.gov/)
to get that data. Now the first thing we see on _that_ page is an "urgent
notice" that URLs are breaking, and anyone using the current ones (for e.g.
FTP access) has to migrate to a slightly different URL with a choice of
subdomains appended. Moving past that initial notice, if we actually try to
find the SSMI dataset, we find out that it's actually listed as the DMSP
(Defense Meteorological Satellite Program):
[https://www.bou.class.noaa.gov/saa/products/search?datatype_...](https://www.bou.class.noaa.gov/saa/products/search?datatype_family=DMSP).
Now to download this data, we're presented with this wacky interface:
[https://www.bou.class.noaa.gov/saa/products/psearchDMSP](https://www.bou.class.noaa.gov/saa/products/psearchDMSP).

This is basically why I decided to work on my current project to consolidate,
document and organize all of this data in a public search engine. It feels
like so much climate data is only public in the absolute minimum sense of the
word, because trying to access it (or even make sense of all of it) transports
me back to 2006. There's a very real sense that the right hand doesn't know
what the left is doing here. Clearly these organizations are aware that
accessing the data is difficult, and they must have good intentions by going
through the effort to document it and host it in FTP servers. But there are so
many conflicting names and acronyms for the "stewards" for overseeing this or
that data, which redirect you to various "portals" for searching it, which
tell you to add data to your "cart" to "order" in some sort of contrived
interface. It's maddening.

Well, there ends my rant for the day. If anyone has any good ideas for working
on this, please feel free to get in touch.

~~~
craftsman
Have you tried NASA's Earthdata Search?
[https://search.earthdata.nasa.gov/](https://search.earthdata.nasa.gov/)

~~~
throwawaymath
I have. I think it's definitely a step in the right direction. But I have a
few observations specific to EarthData Search:

1\. It places a huge emphasis on visualization. The interface is
extraordinarily opinionated. The first thing you're presented with a map which
is, in my opinion, completely superfluous. The intention seems to be an
intuitive interface, but I would not call this intuitive whatsoever.

2\. Search results are not organized in a transparent way, and metadata is not
presented consistently. There is no single format for the principal name,
stewarding agency, associated department(s), citation information, dataset
size, category, temporal resolution(s), spatial resolution(s), date range,
geographic range, update frequency, access information, etc for datasets in
each result. Contrast this with Wolfram Alpha or Dark Sky, which both neatly
inform you of the sources used in various queries and requests.

3\. The search experience is very slow. This doesn't seem to be intrinsic to
how much data is being searched - rather, it seems to be an artifact of how
results are being presented. It's not actually reading over the datasets
themselves, it's just returning dataset metadata. But search queries for
generic climate terminology can take tens of seconds to load, with no progress
indicator.

4\. This site still suffers from what appears to be a systemic problem of
maintainability: it's easy to trigger "Error retrieving site preferences There
was a problem completing the request" errors (on latest Chrome, latest macOS).
An alert in the upper right hand corner indicates that ASDC_DAAC LARC data is
currently experiencing download errors and there's explicitly no estimate for
when this will be resolved.

EarthData Search is one of the websites I was thinking of when I said
"portals" \- this is clearly a well-intentioned effort, and it's a lot better
than browsing undocumented FTP servers. But for climate scientists (or cross-
pollinated researchers, such as statisticians and "data scientists"), a much
more static interface with a common format for searching and documenting all
of this data would be much more useful.

~~~
earthscienceman
I'm a scientist working on satellite data products at one of the two four
letter agencies you named. I have published papers on these data products and
myself work on processing of level 2 and 3 data products. A key point that
needs to follow your comments is that various data products that are often
purpose made for the goals of the people that use them... and that
interpreting said data requires understanding the choices made, although this
can be found in the papers that document the products. What this means though,
in the long term, is that there's often more than one data-set for each
parameter and they're all horribly documented.

 _[most data of this kind] .. is unfortunately mired in a sea of complexity,
filled with shifting departments

NOAA works directly with NASA for much of this kind of data, and stewards it
online_

You seem to have a good grasp of the current state of "climate data" in the US
(and Europe). It's hard for scientists to simply get enough money to launch a
satellite and that money usually doesn't include a budget for "packaging and
organizing data to be published". For the most part, NASA provides "official"
data products for its satellites and then enters into partnerships with other
agencies like NOAA (and other scientists) to use and host these data products.
Often this means refining the NASA products into something more useful. Every
grant I've seen that's creating/taking its own have had one line that says
"will put data up on ftp" and nothing more.

Also, as you're likely quickly finding out... the storage and processing
requirements are often "enterprise class". For quick reference, the level 3
product I work on (EASE-gridded daily means) is roughly 30TB. This is one
product from one satellite for one type of observation. The level 1 data
that's at native time/spacial resolution is something like 3TB per day. There
are several agencies who work on a similar product with different goals, from
the same level 1 data.

I can say in no uncertain terms that you're unlikely to see anyone organize
these data sets any time soon. I admire you for attempting to tackle it. I'm
not sure what your mission is but if I were to advise you myself (having seen
30 different agency sponsored hosting sites come and go) your time would be
most wisely used in two ways:

1) If you're really determined to host the data yourself... find and choose
what you think are the most important products from each satellite, archive
those products and deeply document the sources (i.e. papers published and up-
to-date links to related products that you aren't hosting).

2)Or what I think needs to be done: Create a web portal... like... a really
good web portal. This doesn't exist, which is obviously the motivation for
what you're doing. Something that cleverly organizes and documents the
different types of climate data and the places that you can find them and the
places/papers they're documented in. In my humble opinion, this is what needs
done most. NASA/NOAA/everyone else will gladly host the data but they're never
going to create a "climate data portal" that documents and catalogs data from
the other agencies and sources.

Well, not never... they've tried. It's just really difficult, these are the
ones I know:

[https://climatedataguide.ucar.edu/](https://climatedataguide.ucar.edu/) (NCAR
Boulder, CO, USA)

[https://rda.ucar.edu/](https://rda.ucar.edu/) (NCAR "research data archive")

[http://nsidc.org/data/search/](http://nsidc.org/data/search/) (NSIDC Boulder,
CO, USA)

[https://data.nasa.gov/](https://data.nasa.gov/) (NASA, USA)

[http://cci.esa.int/data](http://cci.esa.int/data) (European Space Agency)

[http://www.geoportal.org/](http://www.geoportal.org/) (ESA portal)

[http://www.ipcc-data.org/](http://www.ipcc-data.org/) (IPCC focused data
sources)

[http://www.ciesin.columbia.edu/](http://www.ciesin.columbia.edu/) (CIESIN
Columbia University)

[https://iridl.ldeo.columbia.edu/index.html](https://iridl.ldeo.columbia.edu/index.html)
(IRIDL Columbia)

[https://help.ceda.ac.uk/article/99-download-data-from-
ceda-a...](https://help.ceda.ac.uk/article/99-download-data-from-ceda-
archives) (CEDA, London UK)

 _the right hand doesn 't know what the left is doing here_

It's more so that the right hand has nothing to do with the left hand. Each
agency, each satellite, each group of scientists, is totally independent of
the other. While sometimes there's noble leaps taken to work together...
there's not much incentive financially or otherwise. Not to mention that each
team is so specialized that they often know nothing about the other 98% of the
data. Take me for example, I've helped work on a very popular data set for a
specific part of the earth system... I don't think I've _ever_ once looked at
ocean data... I don't even know what's out there or how good it is or where
you find it.

~~~
mturmon
Thank you for writing the above well-informed comment.

One important observation at the top of the comment is: "data products ... are
often purpose made for the goals of the people that use them" \-- and this
explains a lot of the atomization and overlap among these data.

You can see this effect across NASA science data (i.e., not just Earth). A
group of investigators sees the potential to make an innovative new
measurement, and develops the instrument or retrieval system to get it. This
can be a decade-long process.

The best people to talk about the features and limitations of that data are,
clearly, that team. And they are generally quite approachable for other in-
discipline specialists at the usual conferences and workshops. There will be
published documents as well, but they are best if you _already_ know what you
want.

But these methods do not help outsiders! It can be hard for out-of-discipline
people to discover what data (among what's available) would be helpful or
applicable for their purposes. It can be very tricky - "you can't use data X
for purpose Y because we filter effect Z out of X, and Z is critical to doing
Y."

One other general point - related to the data maturity pipeline. NASA is
chartered with developing, exploiting, and archiving novel measurements. NOAA
is chartered with operational analysis and use. So there is a handoff needed
there, and a difference in emphasis.

------
hanselot
Someone has to keep the UFO hunters busy.

