
Why you should not pirate Google’s geo APIs - spenrose
http://petewarden.com/2013/09/09/why-you-should-stop-pirating-googles-geo-apis/
======
dewitt
FWIW, Google doesn't particularly like it when people use APIs outside the
terms, either. Having to take down an app is never fun or enjoyable for anyone
involved. And the cases in which the terms aren't programmatically enforced
right away isn't necessarily a choice; often it is difficult to differentiate
at scale between legitimate usage and disallowed usage.

And while I don't know how every other company writes their developer Terms of
Services, I do know that in Google's case we measure each word carefully and
try hard to strike the right balance between offering as much as feasibly
possible, while trying to create something long-term sustainable (so even if
the usage grows far beyond our initial goals, we're still in a position to
keep them available). We're hardly perfect at it yet, but we're certainly
getting a lot better over time.

Also worth noting, often times these terms are designed to honor upstream data
contracts, so if for no other reason, it's worth trying to respect them
because it's the right thing to do for the other relatively small parties,
too.

~~~
Mithaldu
I would have some sympathy with this if you made it easy for single people to
pay you to be able to use your data in a manner outside of the ToS of the free
APIs.

However, for the Elevation API there are two choices:

    
    
      * free and heavily restricted
      * business api, which looks extremely sketchy †
        and is blocked to most people
    

The gaming world has learned this with Steam and friends already. People will
pirate the hell out of digital offerings that are not available in a
reasonable manner. Steam recognized this and made games available in the most
convenient manner humanly possible and are making money hand-over-fist. Sure,
there are still pirates in gaming, but those are the people who are either
paranoid about Steam or those who truly couldn't afford it anyhow.

You're still in this pre-realization phase and have only one single entity to
blame if people "misuse" your APIs: Yourself. Make the APIs available for
normal people in a reasonable manner and you will find adherence to your ToS
becoming the norm, not the exception.

† By sketchy i mean the facts that there is zero pricing information, it's a
buzzword-slaughterhouse, and the only way to get at it is to fill out a form
with a lot of information, an indication of what one might want to do and
which offers no further information or even actual flesh-and-blood human
contact.

~~~
joshma
I don't understand this feeling of self-entitlement - who are we to judge
whether or not an API has been offered "in a reasonable manner?" I'm guessing
it's very resource-consuming to collect, process, and maintain the data
necessary for geolocation APIs (especially given the upstream providers).

It'd be one thing for Google to be deceptive about it, offer a free service,
and try to upsell you every point along the way. It's quite another thing for
Google to dedicate an entire section in the API intro page [1] about Usage
Limits.

Saying that "Google has nobody to blame but itself" is interesting when the
blame is usually from developers to Google, not the other way around.

[1]
[https://developers.google.com/maps/documentation/elevation/](https://developers.google.com/maps/documentation/elevation/)

~~~
Doctor_Fegg
Building your own elevation API is absolutely trivial; I don't know why anyone
would even want to pirate Google's API for large-scale usage.

Firstly, the data is freely available. It's NASA's SRTM [1], downloadable from
a zillion mirror sites.

Then just take some code to calculate a lat/long offset and find the right
position in the right tile. Bob Osola has some easily portable PHP for this if
you need it [2].

Free hint: Putting both your tiles and your (Lua) lookup code in Redis, then
calling the latter with EVALSHA, is a really neat and fast way to do this.

If you really need ocean depths, higher resolutions than SRTM offers, or stuff
above 60 degrees latitude, then it gets a little more complex. But for 90% of
cases doing it yourself is eminently practicable.

[1] [http://www2.jpl.nasa.gov/srtm/](http://www2.jpl.nasa.gov/srtm/) [2]
[http://www.osola.org.uk/elevations/](http://www.osola.org.uk/elevations/)

~~~
hackula1
Building a robust elevation api is definitely not trivial for the average dev
using the google api. If someone thought the only way to get elevations was
through google then they definitely will not know how to find and compile the
data, then write the TIN (triangulated irregular network) algos necessary to
appropriately estimate the elevation of an arbitrary point. No, it is not that
hard, since tools like Grass can do a lot of the TINing stuff for you, but
just running Grass, much less scripting it, is not trivial for someone
unfamiliar with GIS. Making this stuff stable and performant on a server (as
opposed to a one off calculation on a dataset) is also not trivial in the
least.

~~~
Doctor_Fegg
Worrying about TIN is the 10% case. Simple interpolation within SRTM raster
tiles is all that's needed for 90% of cases (IMHO, obviously).

~~~
hackula1
I would not think interpolation would be that much easier than a creating a
TIN. I have not looked at the dataset though, so it might be "pre-
interpolated" (probably using a TIN if I had to guess), in which case this
would be pretty easy with something like pyqgis on a flask server or whatever.
The entire api could basically just be:

    
    
        @app.route("/elevation")
        def elevation():
          fileName = "/path/to/raster/srtmRaster.tif"
          fileInfo = QFileInfo(fileName)
          baseName = fileInfo.baseName()
          srtmRaster = QgsRasterLayer(fileName, baseName)
          res, ident = srtmRaster.identify(QgsPoint(request.form['lat'], request.form['lon']))
          return ident[0]
    

I can test this out and throw it on github with some polish if anyone is
interested.

------
TallGuyShort
Here's another reason that I wish was more prevalently discussed: it's not
okay to violate Terms of Service anyway. Don't accept the terms? Don't use the
service. We'd all have a field day if "big brother" violated the terms of an
open-source license, but the freedom we enjoy in the community comes with a
responsibility to honor our agreements.

~~~
revelation
Violating the terms of a _license_ is very different to violating the _Terms
of service_ you "sign" with a button click.

~~~
TallGuyShort
Actually I'd argue that you're more obligated to abide by something you
explicitly checked "I have read and accepted..." than something that was
listed somewhere in a file you downloaded with some other stuff. You probably
just happen to like the terms of the GPL or ASL more than you like the terms
of Google's API. I do too - but at what point are we just losing all integrity
because we're too lazy to read a long document we have no intention of
actually agreeing with?

~~~
_delirium
If there is an actual click-through _and the license is understandable by a
human_ , perhaps. Usually there is neither, though.

I rarely even know how to look up terms of an API, or that there is an API. If
I know the name of the API, I can look for "Google Foo API", but that name
isn't always obvious. If I just see a URL with an obvious schema, and I change
the HTTP GET parameters to retrieve different stuff, am I allowed to use that
in, say, a small-scale art project, or not? The usual internet default is that
public URLs are implicitly giving blanket permission to retrieve them, except
that retrieval by spiders should abide by robots.txt. Google itself makes
heavy use of this ok-to-retrieve default assumption.

If I go to [http://maps.google.com](http://maps.google.com) there is no
license presented to me as a condition of using the service. I can however
look for fine print and seek out the link "Terms of Service" of my own accord,
at which point I get this:
[http://www.google.com/intl/en_ALL/help/terms_maps.html](http://www.google.com/intl/en_ALL/help/terms_maps.html)

Those terms specifically forbid several things I'm definitely _not_ doing
(mass-downloading data, using it for turn-by-turn navigation, etc.), so my
first read of it makes it sound like grabbing Maps content in my art project
is okay. But then I'm not a lawyer, so maybe it isn't okay.

~~~
res0nat0r
Ridiculous.

Playing dumb isn't an excuse. Can I stop paying my morgtage on my house
because the 40 pages of documents I signed I don't 100% understand? No.

If you want to enter a legally binding contract that you don't understand you
hire someone to help you do so, or face the consequences if you signed
something you were not smart enough to be signing.

~~~
_delirium
> if you signed something you were not smart enough to be signing

That's good advice, but I didn't sign anything here.

I have in the past signed contracts with companies agreeing to use services or
software in certain ways (generally pre-release stuff under NDA), and in those
cases I do indeed read the contract carefully before signing it, sometimes
redlining parts of it before they countersign, and I abide by our mutually
agreed result. If I don't feel I understand it, I refuse to sign, or ask for
clarification.

However, to my knowledge, I have never signed a contract with Google. If they
had asked me to do so, I would certainly have refused if I didn't understand
the terms. But I do not believe they have even asked me to. All I have done is
visit their website, just like they have visited my website.

I have a ToS on my own site, but as far as I can tell, Google just accesses my
site however it wants and does not read it (they definitely don't pay me for
the premium tier, despite exceeding the stated free access limits).

~~~
res0nat0r
>However, to my knowledge, I have never signed a contract with Google.

To use their API you will need to agree to their TOS

[https://developers.google.com/maps/licensing](https://developers.google.com/maps/licensing)

~~~
_delirium
I have never visited that page. I have visited this one:
[http://maps.google.com](http://maps.google.com)

That webpage serves map tiles over HTTP, without asking me to agree to a
contract first. I take it that I may therefore access the map tiles over HTTP?
Obviously I assume I cannot redistribute them, as they're copyrighted. And it
is good netiquette to observe robots.txt for bulk/crawl access. But beyond
that I don't believe me visiting their page constitutes me "signing" a
contract with Google, any more than Google visiting my page constitutes them
"signing" a contract with me.

Perhaps there is some other API with nice JSON stuff and OAuth or whatever,
but that's not what I'm talking about. I'm just doing HTTP GET requests for
map tile URLs.

~~~
Killswitch
I think you're confusing accessing a public version of their data with
accessing their API, which this thread is about. When you use maps.google.com
you are NOT agreeing to any terms, because they don't really require any to
use their data on their site, but when you use the API, you're required to
agree to their terms before being able to use it.

~~~
drdaeman
Aren't the same exact APIs are serving as backend to that "public version"?

If so, looking at whatever data's on your wire (browsers are subclass of user-
agents for a reason), and then doing similar requests by example does not make
you sign some agreement with Google, right?

~~~
enneff
> Aren't the same exact APIs are serving as backend to that "public version"?

Nope.

------
blackmad
I've been meaning to put together a more technical blog post about the various
geo pieces that we've released at foursquare. But for now, here's one that's
an overview of the things we've been working on:
[http://engineering.foursquare.com/2013/06/11/quattroshapes-a...](http://engineering.foursquare.com/2013/06/11/quattroshapes-
a-global-polygon-gazetteer-from-foursquare/)

The biggest piece is [http://twofishes.net/](http://twofishes.net/), our
coarse (city/neighborhood level) geocoder based on geonames data. It also
serves as our coarse reverse geocoder when we combine it with polygon data,
which we've released at
[http://quattroshapes.com/](http://quattroshapes.com/), as a combination of
open government data and synthesized data.

[http://zetashapes.com/](http://zetashapes.com/) is a side project of ours,
where we're seeing if people want to draw their own crowdsourced neighborhood
boundaries in US by coloring in a map of basically city blocks.

I'm always eager to chat about open geodata as well as complimentary
commercial solutions.

------
jacalata
His site
[http://www.datasciencetoolkit.org/](http://www.datasciencetoolkit.org/)
certainly looks interesting! One of the most impressive kinds of arguments -
"I think you should stop doing x and here are the resources y I've
created/curated to make it easier for you to do so."

~~~
jack-r-abbit
I was thinking the same thing. Lots of people get the first part. But not so
much with the second part.

------
ubasu
From the article: "I’m not RMS, I love open-source for very pragmatic reasons.
One of the biggest is that I hate hitting black boxes when I’m debugging!"

RMS started the free software movement because he wasn't able to adapt a black
box printer.

~~~
mkopinsky
Even if RMS started from a pragmatic standpoint, his position has clearly
moved far beyond pragmatism.

~~~
reginaldjcooper
Has it? I think it's very pragmatic to want control over your hardware and the
ability to repair problems or add functionality as desired. If you consider it
from what is best for society as well, it's very restrictive to have hardware
where the software cannot be modified.

------
ChuckMcM
I wonder how much of this is an outgrowth of 'hackathon culture' which is to
say "Hey we can do this thing ..." and get to some demo without having the
foundation bits worked out. (or more importantly the future foundation bits).

There are a lot of people who used to make 'kit cars' which was they took a VW
bug chassis, removed the bodywork, and installed some fiberglass body on it.
Really "cool" looking sports cars. But you couldn't really say "We're going to
be the next Porsche selling our new "Wangdingo" sports car which, as it turns
out, is a fiberglass body on top of a VW bug chassis. The difference between
kids making a prototype which looks cool (professional even, some of these
cars were really really nice) but as a 'new car' it was always a cheat since
the other bits weren't engineered, they were 'acquired' and outside the plans
of the original engineers for the chassis. Compared to the 'motorhome'
business where the company buys a Chassis from Ford or Chevy _as a chassis to
build a new vehicle on._

Makes for an interesting conundrum. You can fail fast and iterate prototyping
like this, and you can arrive at a saleable product for which there is no
route to production because the underlying tech you rely on cannot be used
'legally' the way you want to use it, and the amount of money the product
might conceivably generate is less than the cost of re-creating the crucial
bit of technology from scratch.

~~~
cinquemb
It even gets more complicated as `dewitt mentioned: _" And the cases in which
the terms aren't programmatically enforced right away isn't necessarily a
choice; often it is difficult to differentiate at scale between legitimate
usage and disallowed usage."_

This sets up a situation where people/companies can effectively 'get away'
with it. So what is an aspiring entrepreneur to do when they are able to take
advantage of a situation like this in a way that the other company isn't in a
position block it (without throwing the baby out with the bath water) nor are
they even trying to do what you are doing when you combine such usage with
other "legal" means or data one collects themselves that is providing some
value to people they tested their service/'hack' with?

------
jimmyrocks
If you're looking for elevation data, the USGS provides this for free for the
US.

Example:
[http://gisdata.usgs.gov/xmlwebservices2/elevation_service.as...](http://gisdata.usgs.gov/xmlwebservices2/elevation_service.asmx/getElevation?X_Value=-104.88938886666669&Y_Value=39.59397308786357&Elevation_Units=feet&Elevation_Only=&Source_Layer=)

~~~
picea
Also worth noting, Adam Schneider's GPS Visualiser website is excellent for
quick elevation lookups globally.

Handy for me as Geoscience Australia (another good resource) unfortunately
charge for their DEM data.

[http://www.gpsvisualizer.com/elevation](http://www.gpsvisualizer.com/elevation)
[http://www.ga.gov.au/](http://www.ga.gov.au/)

------
nucleardog
Here's my problem:

1\. The TOS aren't heavily enforced; thus 2\. No one I work for cares about
complying with the TOS; thus 3\. No one is particularly supportive of my
spending additional time to work within the TOS.

Using and contributing work time or resources to an open source project is
going to be a hard sell until there is some pressure on the business end to
change. Having our data sources cut off, or having to reimplement on different
providers would help.

------
Sukotto
[Edit to add] Yes, you're right. When I compare their map side-by-side the
same area in maps.google.com they are clearly different. I should have looked
more carefully before asking my question.

So I guess my question _should_ be "If they had the exact same page except
using a Google map, would they be in compliance?"

-

It's not clear to me in what way the "Crime doesn't climb" page violates the
quoted terms of use. Would someone please explain it to me?

    
    
      The Elevation API may only be used in conjunction with
      displaying results on a Google map; using elevation data
      without displaying a map for which elevation data was
      requested is prohibited.
    

They used the API to display results on a Google map.

Their site used the elevation data to display a map for which elevation data
was requested.

-

I'm not being facetious. It seems to me that their use matches the
requirements. So what exactly is the problem?

~~~
bdon
The map in question looks like it's actually from Yahoo! Maps:
[http://maps.yahoo.com/](http://maps.yahoo.com/)

~~~
X-Istence
I don't think so, it says Nokia in the bottom right, along with CartoDB:
[http://cartodb.com](http://cartodb.com)

~~~
bdon
Indeed, it looks like Yahoo! uses Nokia "here" maps as a data source.

CartoDB is not a map provider, it's a web-based GIS suite that lets you import
different base layers.

------
iamleppert
It's rate limited for a reason and every other so called "better alternative"
will have some sort of limitations in place, if they want to stay in business.
Nothing is for free. By saying people will just magically make a free,
unlimited API service as long as people stop using Google's is dumb.

------
chatman
Everyone should try OpenStreetMap based APIs, e.g. MapQuest Open.

~~~
potatolicious
I tried one part of the API recently - I'm building an app that's partially
intended to give local cycling directions.

Now I'm between a rock and a hard place. One the one hand some parts of the
MapQuest Open API are _extremely_ bad, on the other hand Google has an all-or-
nothing license for their geo data, and their native iOS maps component sucks.

MapQuest's cycling directions in Manhattan is worse than having no directions
at all. They have a specific cycling flag you can set, but results will
blatantly ignore bike lanes and send you down _extremely_ wide avenues with no
bike lanes and heavy traffic. IMO using MapQuest's cycling directions, at
least in NYC, is an extreme hazard, and I can't in good conscience integrate
it into my app.

~~~
graue
How do the directions compare to YourNavigation?

[http://www.yournavigation.org/](http://www.yournavigation.org/)

Despite the clunky interface, I was pleasantly surprised by its biking
directions: not as good as Google, but not terrible either. (I was in NYC when
I tried using it for awhile.) It's fully open-source.

~~~
potatolicious
Just plugged in some of the test endpoints I used with MapQuest.
YourNavigation seems to have two bicycling modes, "bicycle" and "bicycle
(routes").

The former seems to disregard bike lanes entirely and generates some pretty
hazardous recommendations (including cycling down 5th Ave, or busy sections of
Park Ave in Midtown).

The "bicycle (routes)" mode does better and seems to respect bike lanes, but
not consistently. There are some endpoints where it still maneuvers you into
dense traffic with no bike infrastructure, when a bike-laned street is nearby
and takes only marginally longer. I just checked the OSM's dataset and the
data is good - so it isn't a matter of someone inserting a bike lane where one
doesn't actually exist.

The app I'm working on is for NYC's new bikeshare system... Still not an
acceptable alternative unfortunately. I may just give up and use Google Maps
in the iOS app, but the performance of Google's native map component is
extremely, extremely bad.

~~~
Doctor_Fegg
CycleStreets (UK cycle-based journey planner) are on the verge of open-
sourcing their code - as in, in the next day or two. It's a highly fine-tuned
cycle planner based on OSM data; here in the UK, at least, the results are
streets ahead of Google, which will merrily send you down busy, fast trunk
roads with no cycle lane.

You could either take their code and run an instance for NYC, or smile sweetly
at them and ask if they might host one themselves. Github link:
[https://github.com/cyclestreets](https://github.com/cyclestreets)

------
nraynaud
just to clear up a misconception, the SRTM is just unusable in it's normal
form, you need a lot of time to clean it up and cross it with other sources.
This guy does a lot of curation:
[http://www.viewfinderpanoramas.org/dem3.html](http://www.viewfinderpanoramas.org/dem3.html)

(and I also suck google's elevation data for a test because that's the
simplest way to do it:
[http://nraynaud.github.io/webgcode/test_threejs.html](http://nraynaud.github.io/webgcode/test_threejs.html)
)

------
javajosh
Okay, so I have a question: how can open source geocoding services sustain
themselves? Doesn't it cost money to gather all of that data? Where does that
money come from if the service is free? I would expect an OSS geocoding
service to be a) seeded with data that is out of copyright (basic geography
and placenames, b) spruced up with any well-regarded modern free datasources
(probably government funded), c) spruced up even more from volunteers donating
GPS data.

Would a geocoding service be a good candidate for a state-run utility?

------
tegansnyder
If you are looking for a good mapping alternate checkout Leaflet.js. You can
load from a variety of tile servers. I think a while back someone had a
tutorial an setting up your own tile server on AWS. Actually
[http://tiledrawer.com/](http://tiledrawer.com/) provides a base AMI. Another
good commercial solution for geocoding is
[http://smartystreets.com/](http://smartystreets.com/) if you need to do batch
processing.

------
timClicks
Does anyone know of a good guide to DEM (Digital Elevation Model) formats? New
Zealand has some great open data[0], and I expect that it would be a fun
exercise to create your own elevation API from it but I don't know how to get
started

[0] [http://koordinates.com/#/layer/1418-nz-80m-digital-
elevation...](http://koordinates.com/#/layer/1418-nz-80m-digital-elevation-
model/)

------
jetd
I have one comment and one question.

Q: Are OSM/Nominatim good for most countries now? When I tried it was very
good to good enough for developed countries like US or UK but for others
clearly not.

C: I really dont get Google mindset. They sell geoservices. OK. How much money
that would bring 1mil, 10mil or 100mil in over 20bil company. What is the
logic? Creating barriers? I dont know.

------
lhnz
You would have thought San Francisco's crime rate would have been higher with
all of these unauthorised API usages... (Yep, the data was pirated by a guy in
San Francisco. Ironic.)

