

Making Clouds Go Away on MapBox Satellite - incanus77
http://mapbox.com/blog/improving-mapbox-satellite-by-making-clouds-disappear

======
ghshephard
The maps look absolutely beautiful. Can I suggest you drop Apple a call and
offer your assistance? I just came back from Singapore, and good portions of
their hybrid maps were basically unusable because of cloud obstruction. (To be
fair, Google also has a bit of cloud coverage of Singapore - but their hybrid
maps were completely usable).

Cloud obstruction of satellite imagery six months after Apple released their
mapping data, seems to indicate a fundamental lack of knowledge of how to deal
with those type of obstructions, so hopefully your call will be well received.

~~~
huhtenberg
Got a number?

------
brunosan
Wouldn´t the sun-synchronous orbit allow you to use a similar a pipeline with
local-time filtering to create a night-time map? That would be quite amazing,
and also useful for socio-economic folks (or even correlate that with OSM
coverage)

I could only find a related talk about this here:
[http://modis.gsfc.nasa.gov/sci_team/meetings/199905/presenta...](http://modis.gsfc.nasa.gov/sci_team/meetings/199905/presentations/Muller_Night-
Time.pdf)

------
jcr
The images are amazing. When looking for clear skies, I was curious if you
saved any metrics regarding how often a particular location (pixel) is clear
or cloudy?

(NOTE: I have no intention of visiting the cloudiest place on earth with my
vampire hunting kit. ;-)

You might find the weather simulation work done on the old (2005) "Earth
Simulator" or newer (2011) "K Computer" supercomputers in Japan really
interesting.

[http://www.hpcwire.com/hpcwire/2011-06-20/japanese_supercomp...](http://www.hpcwire.com/hpcwire/2011-06-20/japanese_supercomputer_is_new_top500_champ.html)

~~~
celoyd
_I was curious if you saved any metrics regarding how often a particular
location (pixel) is clear or cloudy?_

We didn’t, but it would be pretty easy to re-derive. The cloudiest places at
this scale are the sides of mountains in the Intertropical Convergence
Zone[0], where wet air rises several thousand meters, thus cooling and thus
condensing. For example, tomorrow I’ll be spending a lot of time looking at
Andes in Colombia, Ecuador, and Peru.

0\.
[http://www.youtube.com/watch?feature=player_embedded&v=Y...](http://www.youtube.com/watch?feature=player_embedded&v=YtJzn8A725w)

------
jre
This is pretty cool !

I worked on a similar problem during my bachelor thesis[1] : I wrote an
algorithm to remove clouds from NDVI images (used for deforestation
detection[2]). We used fourrier, moving windows and some ugly hacks to detect
and interpolate cloudy data. There are some details in my report.

[1] <http://ape.iict.ch/teaching/DiplomaReports/2009_Rebetez.pdf>

[2] <http://terra-i.org/terra-i.html>

------
noiv
shamelessplug: Actually MODIS scans Earth using different wave lengths. The
near infrared channel can be used to detect clouds and even to distinguish
them from ice. Using a little recursive AviSynth Script I was able to produce
an animation of daily changes in the European part of the Arctic with last
year's data. It starts with all the clouds and then adds each day's cloud free
pixels. The goal was to visualize how Kara Sea melted out within a few days.
Enjoy: <http://www.youtube.com/watch?v=j-EDFs8f_78> /shamelessplug

------
micro_cam
Cool. I hope they do it separately for winter and summer so as to give an idea
of typical snow/ice cover.

~~~
celoyd
Author here.

We’re doing almost the opposite – as little non-permanent snow and ice as we
can manage. The reason is simply that we want to make this as useful as
possible as a general-purpose base layer, and people usually want to see
spring/summer growth. It looks subjectively right even in regions where it’s
only around for a minority of the year.

If you’re interested in seasonal dynamics, I recommend Blue Marble[0]. It’s
really good work – we’re in touch with some of the people who made it, and
they’re sharp folks. But it’s half the resolution that we’re aiming for, and
has some areas of distracting interpolation artifacts. Our main goal is
accuracy, but we’re also going for aesthetics in a way that Blue Marble
wasn’t.

0\. <http://earthobservatory.nasa.gov/Features/BlueMarble/>

~~~
micro_cam
Thanks for response.

I think what you are doing makes sense for most purposes and I look forward to
seeing the final layers.

I'm especially interested in snow cover because of a side project of mine
(hillmap.com) that is targeted at backcountry travelers (ski tourers, hikers
etc...especially winter travelers since it does things like avalanche terrain
analysis in a canvas overlay).

Blue marble is cool and there are some overlays from NOAA etc but winter
satellite photos good enough to answer questions like "is this typically an
open snow field or heinous bushwhack" or "does this lake freeze over" would be
really useful for planing trips in winter or colder climates.

If you guys (or someone) hosted the daily datasets it would be possible to do
the processing client side in a canvas for a user defined time periods which
would be an awesome tool.

~~~
celoyd
Hmm. I don’t know of any specific product that does what you need (high
resolution and near-realtime is usually an expensive combination), but I would
definitely look through the National Snow and Ice Data Center’s projects:
<http://nsidc.org>

------
Demiurge
This is interesting, but very low on detail. For example, was another modis
product used for clouds or was it based on a simple EVI/water cloud mask? Is
this MOD09A1, 500m or 250m? Lastly, why or how is this MODIS layer going to
compliment existing Landsat or DigitalGlobe?

------
vladtaltos
looks pretty cool. great work. I especially loved the fact that your mehod
does not blur the edges significantly and the color profiles are pretty good.

I have a few questions you haven't mentioned in your blog and would appreciate
it if you can answer them here.

is there some sort of public ftp server you can log in and get say the last 2
days' maps of any given coordinate ? and if so can you share the address ?

additionally, are the image coordinates that come with the images really that
accurate that you can do pixel comparisons over time without doing any sort of
registration ?

------
Houshalter
Wouldn't the clouds still interfere with the averages though? That is areas
that have lots of clouds would seem whiter than they really are? It seems to
look alright though.

~~~
dsl
They detect the images with clouds, then average the rest.

~~~
Houshalter
Ok that makes sense. I thought they were using this as a replacement for
filtering out pictures with clouds.

~~~
celoyd
Well, we’re actually filtering out _pixels_ with clouds. That’s what makes it
seamless. Or, really, it’s extremely seamy, but all the seams are very subtle
and between individual pixels instead of larger regions.

~~~
whatshisface
How do you know the difference between a cloud and a snowy/white sand area of
ground?

~~~
Demiurge
There is a lot of work done for this in the scientific community, there are
many algorithms. (is <http://modis-atmos.gsfc.nasa.gov/MOD06_L2/>) The most
basic way is to look at the reflectivity, which must be very high, and the
temperature, which must be very low. In this case, however, they probably do
simply throw out all the snowy/white areas.

