
Show HN: Sensor data and sky photos for ML: new take on a weather app (Android) - jacobsheehy
http://play.google.com/store/apps/details?id=com.allclearweather.android&120hn
======
jacobsheehy
I make All Clear Weather (Android, US-only for now, see below for why) because
I demand better of weather apps. Many of the top weather apps [1] do nothing
but display sometimes-outdated weather data in poorly formatted tables.
Frankly the top 20 look exactly the same. I'm making a new weather app because
it's 2018 and the category is stagnant. I love the weather and I'm sick of
weather apps that don't do anything interesting. Here is what I am
passionately dedicated to bringing to a weather app to make it _interesting_
and useful at the same time:

1) Your phone probably has a barometer in it! Most Android phones do because
Google likes to use it as an altimeter to give you better indoor directions.
So with iPhones also having barometers (for the iPhone health app), there are
now _billions_ of phones around the world are carrying barometers that next to
no weather apps are using. Yeah WU and Dark Sky have a toggle, but what does
it do? Do they talk about how they are using the sensor data? I don't think
so. With All Clear you can at least _see_ the sensor data from the weather
sensors in your phone and _know_ that it is being analyzed and processed into
'virtual weather stations' so it can be eventually included in the main
weather models. You'll know this because you'll be able to see it happen in
the app as I make progress. Here is a quick graph I made:
[https://imgur.com/gallery/IPj6w7N](https://imgur.com/gallery/IPj6w7N) of
Hurricane Florence's pressure recorded by phones in North Carolina a few weeks
ago. It may take some imagination to see how the data could be useful (it
needs cleaning, privacy protection, quality control, etc) but you can _see_
the data in there and I know it can be extracted into useful numerical weather
model inputs.

2) All the phones in the world have cameras. These cameras are taking photos
of the sky and the weather all day long but nobody is using this data in
weather forecasting. That's obviously because it is difficult to extract
meaningful numeric weather data from a photo, but I believe it can be done.
All Clear has a feature to let users send in photos of the sky with tags, and
I am going to use this data as a training data set to a machine learning
classifier. If it works, it should be able to automatically tag weather
information in _any_ outdoor photo - assuming the training dataset is good and
my ML is good.

3) Current conditions in a lot of weather apps are out of date, since the
source data is out-of-date. A lot of the US MADIS stations update only hourly,
and they can miss updating in intervals when significant weather happens.
Sometimes they update and sometimes they don't. But I think it's so
frustrating to look outside and see it is raining while your weather app just
happily denies the existence of that rain and tells you it's just partly
cloudy. So I let you fix it, for yourself and for others nearby. If enough
people all submit the same weather condition, contrary to the station reports,
the app will start showing that condition as the Truth for current conditions
in that area. (safeguards built in to protect against cheating).

4) I'm sick of the same UI in all the weather apps. I get that the UI in All
Clear isn't perfect yet (it's a new project! and it's a hobby! and I'm not a
UI designer!) but I am absolutely going to try new things in terms of _how_
the optimal weather data can be displayed to a user to maximize quick-check
understanding. It's too easy to be misled by the text, icons, and gridded
numbers in most weather apps.

5) I am dedicated to open source. While the whole app isn't open source yet
(it will be!) the main sensor code that does the barometric pressure data
collection along with other sensors, is open source on GitHub:
[https://github.com/JacobSheehy/AllClearSensorLibrary](https://github.com/JacobSheehy/AllClearSensorLibrary).
I would encourage other weather app developers to enable features in their
apps to aide in the crowdsourcing of potentially very useful weather data that
is otherwise idle in phones.

6) It's in the details. The US NOAA Forecast API that the app uses only
returns its text forecasts with F and mi built-in as units. I wrote an
algorithm (regexes mostly) to convert the text forecast units on-the-fly so
you can be reading it and see it as C and km if you like, it's seamless. Well
there is one error I know about, sometimes it will say "quarter of an inch of
rain" which is a bit of a pain to parse, but I'm working on that too!

Another detail worth mentioning is the smart retry. During development I
discovered that the API returned errors (or didn't return at all) and so
rather than just showing the user a failure screen, the API does a nice
backoff-retry to try really really hard to get the data for you. And if it
can't do that? It knows that probably that forecast office or station is just
_down_. So it changes the location parameters a bit and tries again, hoping to
catch a different station forecast that would still be relevant to you. I
don't have the numbers for how often it fails to load vs. other weather apps
(I hope it's not worse!). But I know this: I put in some serious effort to try
my best to show you the weather no matter what. On that topic...

7) Another feature of the app is that it does a lot of work to attempt to pick
accurate icons. I find that the icons shown in a lot of weather apps are
misleading at best when it comes to describing the weather. It turns out that
it's really hard to show the user the right weather information in just an
icon. So this is the main reason for the animations in the app: to make the
visuals more correct and more engaging. There is logic in the app to not show
the default icon that comes from the weather API: if it did do that, then a
30% chance of thunderstorms shows you just the thunderstorm icon. Most apps do
this. But chances are? The weather will be just partly cloudy, no
thunderstorms. So the app shows a partly cloudy animation with a faded,
flashing lightning bolt to indicate the probability. Further work is being
done in future updates to the app to make this even more accurate.

\-----

Alright, there's what All Clear can do that other weather apps can't. Maybe it
should be your next weather app but maybe it is missing a critical feature for
you. __What 's that feature? __What do you need or want in a weather app that
nobody is delivering today?

All of that above featureset is free by the way. Part of the problem with not
enough original and quality weather apps is that they each cost money to run
and users are often unwilling to pay. So for All Clear it is free to use all
the features of the app; You can pay $1/month if you like to extend the
forecasts from 3 days to 6, and extend other features in the app like the
number of visible weather pictures (from 10 to 100). The subscription will
help pay for the servers that run all the custom weather data code as well as
future development of neat weather experiments using smartphone sensors.

And a note about it being US-only. I tried to make this international from the
get-go, and wasted months using IBM's API from WU that barely worked at all.
Then they yanked the payment form off the web the week I was going to publish
the app and buy their $400/month package. No longer an option. Now you have to
wait for a phone call, might be weeks, and the prices are like $3,000 for the
equivalent data (or something, hard to say). If you want to read some of the
drama about this, check out this WeatherUnderground thread:
[https://apicommunity.wunderground.com/weatherapi/topics/weat...](https://apicommunity.wunderground.com/weatherapi/topics/weather-
underground-api-changes) where people are describing the confusion as WU cuts
off their keys and makes the service dramatically more expensive.

So the app is using the US NOAA data for free and I am investigating
alternatives for international weather data.

[1]
[https://play.google.com/store/apps/category/WEATHER/collecti...](https://play.google.com/store/apps/category/WEATHER/collection/topselling_free)

~~~
matt_the_bass
Cool idea! Would you mind clarifiying some questions?

1\. Are you looking to do your own weather prediction or just better package
the prediction results from existing weather models? I ask because weather
modeling is hard and there are lots of PhDs working on continual improvement.
They also use an huge amount of processing power to do this.

2\. Are you planning on open sourcing the collected data too or do you hope to
monetize the data?

~~~
jacobsheehy
Thanks!

1\. I'm definitely just starting simple. My first goal that makes sense to
target is: a) quality control algorithms for the data that make b) it possible
to do data assimilation into WRF. I would not expect this project or myself to
do the actual WRF data assimilation, that would probably require outside help.

For example, I know that this process can be done with the barometer data, but
it is difficult to do this for a couple reasons. First is user privacy, most
methods of using this data require a user id attached to the lat/lng and
barometric pressure. This isn't great, it's a log history of user's location.
So that's why I'd like to make 'virtual weather stations', which would be my
first target.

Perhaps in the distant future this project could span its own
predictions/models but I don't have a particular plan for that.

2\. Well as I alluded to above the raw data is sensitive to user privacy, so
the ideal system deletes that data after a few days of processing/QC control,
hashing, etc to preserve as much as we can of the useful information while
deleting the user's private location history. This makes the dataset a lot
less monetizable, since fewer weather-industry folks who'd like to integrate
with WRF would be interested in the data, given how far removed my final
product would be from the source barometric pressures (virtual weather
stations are a brand-new abstraction and they have not been tested).

I'd say that at first the data that I make in terms of 'virtual weather
stations' may be an open API, but it depends on how much that costs to run.
The project is supposed to make money from users buying it: it has a $1/month
subscription optional to show users how to directly financially support the
project. Any future sales of weather data would be a long way away and would
require me to prove out its usefulness, something that hasn't been done with
'virtual weather stations' yet (but has with the raw barometer data, darn)

Great questions, thanks!

~~~
pstch
> Any future sales of weather data would be a long way away and would require
> me to prove out its usefulness, something that hasn't been done with
> 'virtual weather stations' yet (but has with the raw barometer data, darn)

Honest question, as I don't know much on the subject : why should future sales
of weather data be a long way away ? Aren't there already end-users interested
in low-volume API access ?

Also, just to clarify, by 'virtual weather station', you only mean devices
that run the application, excluding other non-official weather monitoring
systems ?

