
How We reduced our Google Maps API cost - haybanusa
https://blog.cityflo.com/how-we-reduced-our-google-maps-api-cost/
======
disillusioned
Well, that's more clever than a company I did due diligence for.

Their strategy was to have a pool of API keys attached to new accounts that
would take advantage of the Google Maps API free tier, and monitor its usage.
As the free tier usage would run out, the system would roll over to a new API
key automatically.

Wrote that one up in big red marker in my report...

~~~
DeathArrow
It's not clever, it's just much more simpler. Using linear interpolation for
the time between 2 stops will have low accuracy because for the particular
situation time might not be linear with position and distance. Also traffic
incidents might happen.

Going with a pool of free keys will be much more dependable, even if somehow
more complicated to manage and easier to break.

~~~
bad_user
It's probably against their ToS and Google could detect it and ban all of
those accounts, or the servers' IPs.

You can then keep fooling them by creating new accounts or changing IPs
(assuming your usage doesn't have clear patterns they could look at).

But such events would be clearly disruptive for the business. Works for a POC,
but if your business has actual customers, it's a terrible solution.

~~~
disillusioned
This company was shockingly deep into their lifecycle to still be using this
approach. And yeah, they'd cycle IPs as needed too. I think the thought was
that Google isn't doing a ton of fraud analysis for this particular modality
of fraud. Still though...

~~~
DeathArrow
You don't have many options if you need high accuracy: you have to pay a lot
or try to trick Google which might be both immoral and against the law and for
sure is tricky, hard to maintain and you can't count on it in the long run.

Let's hope there will be alternatives to Google provided traffic data. For now
they seemed to monopolized it by offering it for free while losing money to
discourage competition.

~~~
raverbashing
> You don't have many options if you need high accuracy: you have to pay a lot

What happened to actually trying to solve problems with programming?

Interpolation is one solution. Caching is another. Temporal analysis. Put
everything together.

You don't need to query the magic Google box for every small update you make
(and they might get that info from the transit providers, which given my
experience are not that great sometimes).

------
AnotherGoodName
I did the same years ago. We were providing realtime suburb data for a fleet
of trains. Each train received a GPS coordinate once per minute, we took this
and displayed the suburb. So 1440 updates per day per train. For the fleet it
was going to be over $100 a day in API costs.

We were going to not display suburb data because of cost. In the end I found a
creative commons placename database (geonames.org). For placenames with >500
people it's ~10MB of data and that covers the entire planet (surprisingly
small). I then wrote a KD-Tree based library to look it up the nearest point
in this table extremely efficiently (log(N) time).

I'll admit i haven't updated or maintained it. The server running it has been
chugging along well though >5years later.
[https://github.com/AReallyGoodName/OfflineReverseGeocode](https://github.com/AReallyGoodName/OfflineReverseGeocode)

~~~
saila
Have you used PostGIS at all? It seems like a good choice for this kind of
lookup.

[https://postgis.net/](https://postgis.net/)

~~~
AnotherGoodName
PostGIS would be worth it if there was a lot more to do on that project and
full respect to that project. I remember looking at it and it felt like it was
a sledgehammer for that particular job. You still had to write up a lot of
code to import the dataset you had and then extract it again. Not to mention
maintenance since you now have a whole dependency chain installed on your
server. This was a single engineer project.

The above solution is a copy paste-able set of classes with zero dependencies
that will output the nearest place. Sometimes a single purpose solution is
perfect and i'm really not kidding when i say i haven't maintained or even
looked at it in over 5 years yet it's still running fine as part of a larger
application.

------
superpermutat0r
If you track the vehicles every day and collect the location data, you can
easily augment the Open Source Routing Machine to give you traffic accounted
estimates.[0] Combined with some Kalman filters you'd get almost perfect
estimates when live.

Of course, this is for a use case where you have similar routes every day,
this allows you to really tune the Kalman filters.

0: [https://github.com/Project-OSRM/osrm-
backend/wiki/Traffic](https://github.com/Project-OSRM/osrm-
backend/wiki/Traffic)

~~~
cookie_monsta
My understanding is that Google's does real time traffic reporting so well
because it's constantly pinging the location of android devices. There have
been several write ups on how to spoof it, but historical data is never going
to be a match for a feed like that.

~~~
thejosh
If you carry lots of android devices it will alter the traffic on Google Maps
-
[https://www.youtube.com/watch?v=k5eL_al_m7Q](https://www.youtube.com/watch?v=k5eL_al_m7Q)

~~~
throwaway2048
I find it insane that the fact that Google is constantly harvesting device
locations is blatantly obvious from things like this, but yet if you make that
claim in other contexts people will be extremely skeptical and demand
extensive proof.

~~~
Danieru
Those phones all had Google Maps app open. That Google is harvesting device
locations while you have a Google app open showing your GPS location should
not be a surprise.

What would be a surprise is if Google forced on your GPS to gather your
location in other contexts.

~~~
gog
It is not a surprise, it is already happening. You can turn it off, but AFAIK
if you don't it collects this information even if the Google maps application
is not open.

~~~
apacheCamel
From my personal experience, I have an Iphone and it asks me about once a week
about Google Maps using my location in the background and if that is okay. I
always tell it no. I didn't know I could turn it off directly in the app,
thank you for the information.

------
karussell
If they have their own fleet they can generate their own (historic) traffic
data e.g. via Map Matching and use an open source routing engine like
GraphHopper with OpenStreetMap data. (disclaimer: I'm one of the developers of
GraphHopper.)

It is unclear to me whether their current practice is in harmony with the
Google Maps TOS. For some places there are also open traffic data sources:
[https://github.com/graphhopper/open-traffic-
collection](https://github.com/graphhopper/open-traffic-collection)

~~~
pedro_hab
Yeah, I think you can't cache gmaps data, each user has to request at the time
of use, "to ensure the freshness of the data".

But I'm not sure if it applies to this use case.

------
samcheng
For what it's worth, I found OpenStreetMaps and the Open Source Routing
Machine to be sufficient for our purposes.

If you don't want features like real-time traffic awareness, it's worth
investigating the open source tooling. It can save a LOT of money.

~~~
sudhirj
I think real time traffic awareness is exactly what the poster wanted, but
yeah it seems like people who don’t need that would be better off using OSS
where possible.

~~~
samcheng
Yeah, unfortunately I'm not aware of any open/free sources of real-time
traffic data.

~~~
karussell
Here is a collection I started some time ago:
[https://github.com/graphhopper/open-traffic-
collection](https://github.com/graphhopper/open-traffic-collection)

~~~
samcheng
These seem to be average / historical data sets, not "is there an accident /
traffic jam on my route right now."

Thanks for the excellent work, though!

------
kartayyar
This seems to be violating the Maps API terms of service?

[https://cloud.google.com/maps-platform/terms](https://cloud.google.com/maps-
platform/terms)

(a) No Scraping. Customer will not export, extract, or otherwise scrape Google
Maps Content for use outside the Services. For example, Customer will not: (i)
pre-fetch, index, store, reshare, or rehost Google Maps Content outside the
services; (ii) bulk download Google Maps tiles, Street View images, geocodes,
directions, distance matrix results, roads information, places information,
elevation values, and time zone details; (iii) copy and save business names,
addresses, or user reviews; or (iv) use Google Maps Content with text-to-
speech services.

(b) No Caching. Customer will not cache Google Maps Content except as
expressly permitted under the Maps Service Specific Terms.

~~~
xtony
Caching is allowed (for up to 30 days). Here are the specific terms for the
API in the article: [https://cloud.google.com/maps-platform/terms/maps-
service-te...](https://cloud.google.com/maps-platform/terms/maps-service-
terms#1.-directions-api)

"Customer can temporarily cache latitude (lat) and longitude (lng) values from
the Directions API for up to 30 consecutive calendar days, after which
Customer must delete the cached latitude and longitude values."

~~~
DeathArrow
That might be a mistake on their side which they might repair now that they
started to ask for lots of money for the data.

~~~
wstuartcl
rofl. funny.

------
KingOfCoders
We moved to Stadia Maps with significant cost reduction, but Stadia matched
our use case quite well, I can see others need to stick with Google Maps.

[https://www.eventsofa.de/campus/migrating-away-from-
google-m...](https://www.eventsofa.de/campus/migrating-away-from-google-maps-
and-cutting-costs-by-99/)

Shoutout to Stadia Maps CEO who was always very very helpful.

~~~
lukeqsee
Thanks for the kind words, KingOfCoders!

I'm cofounder of Stadia Maps, and I'm happy to field any questions folks may
have.

------
DeathArrow
The conclusion I get from this is not plan anything on "free" features
promoted by SaaS providers because that can hit you hard in the future. Either
plan based on something payd and covered by a contract or try to come up with
another solution, maybe an in-house solution, maybe a free solution with a
failover based on another provider.

~~~
sm2i
At a cloud storage company i worked for we had the most cost-intensive part of
our infrastructure, the object storage ready to be switched over from Azure to
AWS, just in case.

------
mkchoi212
Random thought but I think it’s funny how about 40 years ago, people would
write about their tricks to save program memory usage. Now, people are talking
about how to shave costs of doing API calls.

~~~
CamperBob2
Meanwhile, the old maxim "All programming is ultimately an exercise in
caching" seems to have been forgotten entirely. This whole thread is just
surreal.

~~~
dev_tty01
Yeah, it is amusing. Given the small set of fixed routes, this problem is just
crying out for "local" caching of the expensive data. I guess what is old is
new again...

------
einpoklum
As others suggest, switching to OpenStreetMap is something to consider.

However - it is also important to _contribute_ funds to OpenStreetMaps - for
cityflo and for us.

Donations:
[https://wiki.osmfoundation.org/wiki/Donate](https://wiki.osmfoundation.org/wiki/Donate)

Individual membership:
[https://wiki.osmfoundation.org/wiki/Membership](https://wiki.osmfoundation.org/wiki/Membership)

For organizations: [https://welcome.openstreetmap.org/how-to-give-
back/](https://welcome.openstreetmap.org/how-to-give-back/)

The OSM Foundation in general:
[https://wiki.osmfoundation.org/wiki/Main_Page](https://wiki.osmfoundation.org/wiki/Main_Page)

~~~
chowes
They mention this in the article

------
rkda
Author mentioned OSM but they could also use Mapbox's navigation APIs if they
don't want to host things themselves.

[https://www.mapbox.com/navigation/](https://www.mapbox.com/navigation/)

~~~
Reason077
It would certainly be interesting if they had compared the quality of Mapbox’s
traffic data to Google’s. Perhaps the authors tried this and the data wasn’t
as good? But it’s not mentioned.

OpenStreetMap itself is just the base map, not a provider of traffic data or
routing.

~~~
rkda
Yeah. Mapbox is getting there but I don't think they have the same volume of
users contributing traffic data as Google does.

~~~
lucb1e
Indeed, and it will be very hard getting anywhere near as good as Google's
without owning Android which nobody else could possibly do.

I still think location data collection for the public good should be run by
someone who is not Google / Facebook / some other company that doesn't already
track everything about you. I'd be happy to send my real-time location to a
company like Mozilla or some mapping company (just in the tiny netherlands I
know of AND, Geodan, OsmAnd... I'd be fine if any of them organize this).

We also have some open data via the government, based on detection loops in
the road, but if I remember correctly it's only (or mainly) on highways.

------
kijin
Since they have actual vehicles on the ground, they could collect data on
actual travel times under various traffic conditions and apply a bit of ML to
predict how long it's going to take under current conditions. No need for any
Google API there.

But now that their Google bill is only ~$50/day, it might not be worth
building their own prediction system.

~~~
DeathArrow
If ML could predict everything with accuracy, they would be better by
predicting lottery, stock prices or Forex pairs.

I used an app which predicts public transportation arrival time based on
historic data and it mistakes most often than not, sometimes by a lot.

~~~
srean
To me your comment sounds unnecessarily rude.

I guess you are reacting to private smug conversations with people who
eulogize ML but are clueless about what they are talking about. Usually the
amount of conviction they add to their comments and advice is inversely
proportional to their knowledge. I am tired of those too, to the point of
being allergic in spite of being an ML practitioner myself.

The parent comment could have been worded better but I think you will agree
that there is scope for using ML or statistical modeling of some kind to
optimize the use case both in API cost and accuracy. Making some minimal
number of API calls would very likely be part of such a system.

------
barrkel
Taking this approach to its logical conclusion, one could sample Google to
infer a congestion heat map for all the main routes in an area, and calculate
the transit time estimates directly from the inferred heat map.

IME congestion is also often modal, there's probably reductions in sampling
effort you could make by noticing the patterns in how commuters route, whether
school is on break, and what the latest roadworks are.

------
blumomo
So could increasing pricing under circumstances help to make your customers
more creative, more efficient and more ecologically sustainable by making your
costumers consume less of your computing power and bandwidth?

~~~
ant6n
Does it matter whether google or the consumer of the google api caches data?

~~~
lrem
Placement of the cache definitely matters, even more so if you can take
advantage of the more specific data freshness requirements.

Disclaimer: I work in Google. But I also worked a bit on effects of cache
placement in my PhD.

~~~
boojing
Why is the fact that you work at Google and have researched caching for your
PhD a disclaimer? I fail to see how it denies your previous statement

~~~
lrem
Actually mehrdadn is right, I should have written "disclosure". Which I am
contractually obliged to in all discussions relating to Google.

Then, I might be perceived as biased due to that employment. I might even be
actually biased. Better disclose that up front.

------
melbourne_mat
They mention a 94% budget saving but there's no mention of the drop in
accuracy (which I'm sure has occurred)?

~~~
cookie_monsta
That's the one line that kept me reading 'til the end (and never appeared,
even though other approaches were dismissed on accuracy grounds)

------
punnerud
Why not save the route, decode the polygon into its respective coordinates and
use that to find overlapping roadparts? This would work all of Google Maps,
also for bikes or walking.

Example here on how to decode: [https://github.com/geodav-tech/decode-google-
maps-polyline](https://github.com/geodav-tech/decode-google-maps-polyline)

------
austurist
It is a proof of that Google API is overpriced. Google apparently sells
something, that can be computed as well.

~~~
prepend
I wonder how Google prices this api? The actual cost must be difficult to
determine because if the source data.

Perhaps they have some margin formula for “buying” data from their internal
sources.

It seems to me that they are value pricing and since they are close to a
monopoly on this service they are testing the price elasticity.

~~~
filoleg
The data for traffic is usually needed in real time, not just dumping
historical data in one call and sitting on it. With that in mind, the most
obvious pricing policy seems to be to either charge per API call or to have
subscription tiers where you have a cap on maximum number of API calls per
hour or per day or whatever other time interval they choose.

~~~
prepend
Sorry, I meant the price it sets for the api calls. The article calls out
$10/1000 so the current price is a penny per call. I’m interested in how they
set that rate.

------
bartkappenburg
We also had an increased Maps bill after the new pricing model for each map
load on a detail page for a specific real estate object. Our map is now
disabled by default and can be loaded with a “show map” button on a static
cached map image.

~~~
cma
Seems like the more frugally people use it, the more they can up the price.

~~~
asdfasgasdgasdg
Only if the goal is to stop getting people to use it. If the pricing of the
API was meant to be reflective of the cost of running and to encourage
customers to use it more efficiently, then increasing the price beyond what
they have done is maybe not a good idea.

------
Enginerrrd
I really don't mean to be snarky, but if this counts as an innovation worth
writing about, I am severely undervaluing my worth.

------
darkerside
I wonder if they simply tried calling Google and negotiating a volume based
discount. I realize Google is notoriously impersonal, but I always prefer if
you can solve a problem without code.

~~~
mdorazio
I worked with a major automotive company that was upgrading its head units to
be more user friendly. Their #1 customer request at the time was to have it
use Google Maps, so they tried to negotiate a discount with Google. I can't
share the discount amount, but it was laughably small and the company would
have been shelling out millions of dollars a month to Google for the
integration. Needless to say, they stuck with a different mapping provider.

~~~
lucb1e
I was actually kind of surprised that my spouse's new company car did not come
with either Google Maps or OpenStreetMap but instead credits the map data to
some mapping company from eastern Europe if I remember correctly -- the data
looks like whitelabeled TomTom or HERE data.

I just wish they would either augment their data with OSM or augment OSM with
their data: now, with proprietary data, the data was already old when we got
the car, and the fixes I contributed to OSM are obviously not in there.

Instead of contracting some middle man for inferior map data, they could have
used completely free (as in beer) data for the regions where OSM is more up to
date and complete than any commercial provider (empirically, this is most of
the land mass on earth plus countries with a lot of mapping enthusiasts and/or
free government data, like Germany and the Netherlands).

~~~
mdorazio
For what it's worth, traditional (i.e. not Tesla) OEMs generally had a "no
open source anything" policy for a long time, for a variety of reasons I don't
really agree with. That's changing a little bit these days, but not a whole
lot.

------
aimad41
I've build my own tile server map with mapbox gl spec, thx to them, with
simple osm features, data about transportation is fair enough to get some
basic calculation features, i mean it's a good start, Google maps pricing is
insane, if u plan to support thousands of users like us this gonna kill your
profitability, we have no money to invest for that part and imo u should be
vendor less if u want to develop ur activity til u buisness model works at
least

~~~
lucb1e
Hey, just a heads up: the "thx", "u", and spelling mistakes make your comment
less attractive to read. I'm not saying your English has to be flawless, but
things like using "u" instead of "you" is a choice.

------
devit
Does the API not give timing info for parts of the route like the website
does? If it does, then they could just query for first stop to last stop and
infer all subtrips.

Also given they seem to own the buses, they could give the drivers a rooted
Android phone and get the data by reading the memory (or network traffic after
reading the TLS key from memory) of their running instance of Google Maps that
they are using to drive, which should be undetectable by Google.

------
fedorareis
While that seems like a good start it seems like you would want to supplement
that approach with the GTFS and GTFS-realtime feeds from transit agencies
where available.

Having worked for a company that provided transit software I am well aware
that GTFS-realtime data isn’t always accurate (especially the ETAs). But it
seems like combining their ETA estimate with yours and the scheduled arrival
time would be a reasonable approach. Or if you just started building a model
of the average speeds of the bus on each link using the GTFS-r speed and
location data you could probably build up a pretty good historical model of
the traffic flow between 2 stops at any given point during the day.

Yes, I am aware that scheduled times in transit can be meaningless. There are
a lot of factors that contribute to that from traffic to poor schedule
planning. They can however be a useful baseline estimate.

------
ankit219
I like the assumption made while the bus is between two stops but that could
be very inaccurate in a geography like Mumbai. I have seen your posters at one
of the metro stations and assume that you operate in the city. I feel the
biggest choke point in the city is the number of signals - around the metro
stations - and hence the uncertainties around timing before and after crossing
the signal. Traveling in a cab, this is where Google maps timing fails too by
about 5-10 minutes. Not a big deal as a customer, just a mildly interesting
titbit in my daily travel (before Corona)

One thing to look at given you would have everyday data is to see if there is
a pattern (if not already did it). Then you can suggest commuters to take the
bus now and save commute time v taking after 15 mins.

------
bluesign
This takes accuracy hit for sure, total time of route can be different than
the total time of segments. Especially if the route is long.

~~~
iforgotpassword
Why? That would entirely depend on how Google implements this. Ideally the
result would match up exactly. If not you can certainly approximate the error
introduced by this method well enough. They already needed to have something
in place anyways to account for the bus actually stopping at all the stops.
The API doesn't know about this in their old approach.

------
asjw
Or switch to HERE that is much cheaper than Google

------
amelius
Only works if you have a fleet. Perhaps we should share the data so even
people without a fleet (or a small fleet) could benefit.

------
thelastbender12
This is a pretty nifty solution! To nitpick, I'm curious how much the ETA
between successive stops fluctuates.

For example, if you have 2 buses on the route `A-B-C-D-E-F-G-H`, using the
same value of T(F-H) for the bus starting off at A and another one already at
D, might not be quite right?

~~~
DeathArrow
They said in the article they sample the time as frequently they deem
necessary.

~~~
thelastbender12
Hmm, could you please expand? What I'm trying to get at is that the Bus at D
and the bus at A expect to reach stop F at different times. So, the ETA
between F and H is never quite the same for them, as the traffic conditions
could change.

When google maps gives me an ETA, I assume they account for it (or their ML
model does from the vast troves of past data).

~~~
DeathArrow
I get they compare the prediction from maps with actual time for a few stops /
busses and if the difference is big they make a new API request, making Google
wealthier.

------
tomjuggler
I used to use Google's geocoding API when it was free, really hate it when
they make stuff and then when it gets popular start charging $$$

I still haven't found a viable free replacement, unfortunately. I'm just doing
this for personal projects, so no big budget.

~~~
NetToolKit
FYI, we at NetToolKit offer 1,000 free queries per day via our development
key, and the cost beyond that is very affordable at $10 for 100,000 queries.

------
raz32dust
How does this take traffic into account? It is clever and I enjoyed the read,
but it is essentially doing some of the work GMaps is doing for you yourself,
losing some accuracy in the process. That's probably a great trade-off in this
case but not always.

~~~
giovannibonetti
As mentioned in the article, they still get estimates from the Directions API,
which takes traffic into account. However, instead of getting the ETA for each
bus they get it for the path, apply some clever interpolation and refresh the
base data from the API once a while

------
tech_dreamer
Point to note : "It was our route design that essentially helped us decouple
the bus location updates and travel time computation for each bus, thereby
reducing any redundant calculations"

------
coopsmgoops
I'd like to have known the error compared to the GM estimates. Some sort of
Monte Carlo approach where you check your estimate against Google's.

------
AtomicGalaxy
I wonder why none are talking about an automated GUI script instead of
multiple keys? (I know both are against Google’s ToS)

------
AtomicGalaxy
Is instead of using multiple keys, writing an AutoGUI script better? Of course
I know both are against their ToS.

------
yaktubi
Switch to open street maps and reduce by 100%!

------
chiefalchemist
I believe there's another way to look at it. More from the consumer's POV and
less as pure technology per se.

First, is the fuild ETA absolutely necessary? Surely the bus has a schedule.
Is it on time, or not. If not how late might it be? That's not the same as
ETA. Certainly, they've collected and keep collecting enough data to infer
such things. That is, 2 mins late to Stop B translates to what at Stop F and
stop N.

The consumer doesn't need ETA per se. They need to know if they're going to be
on time to their destination or not.

Like putting mirror next to an elevator to shorten the perceived wait; there
might be other opportunities to solve this problem.

If you're interested in such things, this book from a year or so ago was
intriguing.

Rory Sutherland

Alchemy: The Dark Art and Curious Science of Creating Magic in Brands,
Business, and Life

4.6 out of 5 stars

[https://www.amazon.com/Alchemy-Curious-Science-Creating-
Busi...](https://www.amazon.com/Alchemy-Curious-Science-Creating-
Business/dp/006238841X)

------
map08
We at nb.ai help companies with similar get off of Google solutions, and are
able to achieve similar reductions in API calls, along with typically, an
IMPROVEMENT in accuracy. We find building good machine learning, traffic and
routing models, along with OSRM and GraphHopper goes really far

