
Ask HN: Who here would be interested in a free daily updated GeoIP database? - dutchbrit
I recently was playing around with Maxmind&#x27;s free GeoIP database but was disappointed at how inaccurate the free version was.<p>I started hacking away yesterday at making an alternative database that seems to be more accurate, but it&#x27;s still in the works. Who here would be interested in a free alternative, including and excluding cities, updated on a daily basis, and if so, which format&#x27;s would you like to see?
======
oalders
(Disclaimer: I work for MaxMind)

As far as formats go, if anyone is thinking of putting together a custom
database with information keyed on IP address or IP ranges, I'd encourage you
to look into our MaxMind DB format:

[http://maxmind.github.io/MaxMind-DB/](http://maxmind.github.io/MaxMind-DB/)

There are open source file readers available in many languages:
[https://github.com/maxmind/?utf8=%E2%9C%93&query=MaxMind-
DB](https://github.com/maxmind/?utf8=%E2%9C%93&query=MaxMind-DB)

There's also an open source writer
[https://metacpan.org/pod/MaxMind::DB::Writer](https://metacpan.org/pod/MaxMind::DB::Writer)

Our officially supported libraries are actively maintained and pull requests
are always welcome.

I'm actually about to begin writing a blog post on how you can create your own
custom databases. I'll share it on HN once it's ready.

~~~
Firegarden
Can you please confirm that the free versions are not 'good enough'?
[http://dev.maxmind.com/geoip/geoip2/geolite2/](http://dev.maxmind.com/geoip/geoip2/geolite2/)

~~~
oalders
If you want to compare the accuracy of different products, you can do so on
this page:

[https://www.maxmind.com/en/geoip2-city-database-
accuracy](https://www.maxmind.com/en/geoip2-city-database-accuracy)

To get the actual product comparisons, you'll need to enter a country in the
input box. For instance:

[https://www.maxmind.com/en/geoip2-city-database-
accuracy?cou...](https://www.maxmind.com/en/geoip2-city-database-
accuracy?country=United+States&resolution=50)

You can try refining the distances to see how the different products compare.
If you fiddle with it a bit, you'll see that the results do vary from country
to country. So, "good enough" will depend on some combination of your
tolerance for error + which countries you care about.

~~~
pczzy
It's accuracy but not free

~~~
devin
I'm not sure I understand. You can use the links provided and compare those
results to the results you get from the free version.

------
mgkimsal
I'm not sure I want a free alternative. I want an inexpensive option that I'm
pretty sure won't get acquired and shutdown, but also has reasonable enough
revenue to remain in business, occasionally blessing me with new features.

A free service like locationkit (seen here a few days ago IIRC) looks OK, and
I'm reasonably sure it's supported by revenues for other services from the
parent company. However, even that feels like it might get shut down if it
becomes too big a burden.

Something with a good understandable license that I can pay a reasonable
amount ($5-$20/year, for most low-mid volume use cases) would be great, as
long as I can really use it for commercial purposes. I had some high hopes for
Yahoo services a few years back, but they seemed determined to plague
everything with confusing or commercial-hostile licensing.

~~~
pvg
That sounds a lot like some of MaxMind's products.

~~~
mgkimsal
I think it is. I've not tested their quality, but saw their pricing - buy
credits then they get used up per API call, similar to Twilio's per minute
deductions from a credit. If the OP has a way to best MaxMind's quality,and
can charge, that'd be good. But the OP was talking about a 'free' model, which
I'm not really interested in.

------
dangrossman
How/where do you get more accurate data than MaxMind does? And under what
license do you have access to this data, and under what license would you be
able to offer it to others? If all the legal stuff looks right, and the data
is indeed more accurate, then of course there's going to be interest in it. I
think it would take a while for word to spread and links to propagate, but
eventually you'd see quite a bit of sustained traffic on that project. IP-to-
location data is a very common need.

~~~
clinta
My only experience with MaxMind was the trouble we had when we got a new IPv4
assignment from ARIN. I filled out the form on their website the first week,
our address was in the WhoIS records for our IPs, and our BGP announcements
made our location obvious, yet they had our IP's (obviously from ARIN) showing
up as originating from South Africa for months.

Seems like anyone who can look at BGP tables and whois information can get
more current and accurate data than MaxMind.

------
hannob
honest opinion: I hate geoip. I think judging your visitors from their
geolocation is more and more becoming a problem. It goes against the
fundamental value of the Internet being a global network. I don't want Netflix
to sell a sub-standard product in some countries because some licensing foo. I
want everyone on this planet to have access to the same Internet.

If I could I would redesign internet routing in a way that the webpage doesn't
know where a request comes from.

~~~
geofft
> I don't want Netflix to sell a sub-standard product in some countries
> because some licensing foo.

"some licensing foo" is a great way of eliding that in some countries, the
service would be _illegal_. The alternative is that the product could not
exist at all, if there is no way to negotiate a contract for it.

You can certainly argue that products should not exist if that would involve
contradicting some abstract ideal for society, or that it's good for things on
the internet to blatantly violate mundane real-world laws until real-world
governments become irrelevant. Respectable people hold both of those
positions. But they're not very widely-accepted positions.

~~~
derefr
Presume a hypothetical world where everybody's traffic is onion-routed, or at
least VPNed. Streaming-content licensing would still happen in such a
world—the location-enforcement would just have to happen by other means, like
account phone-number verification.

~~~
geofft
That still involves Netflix selling a sub-standard product in some countries,
no?

I'm not being pedantic about the argument, I just don't see how there's a
benefit to any fundamental values if we still allow this delta, but we're
picky about what technical means are used to make it happen. I could see an
argument that geo-IP databases are often _wrong_ and that's a problem, but
that seems like a very different argument.

------
lucb1e
I use GeoIP fairly regularly. How I would set this up:

\- Free version: only updated once [every 6 months / a year] and non-
commercial use only

\- Paid version: $9.99 for a year of free updates

\- Updates come both in diff and in full version (for the commercial version)

\- Use a common format, or use multiple formats. People love csv, but more
serious applications probably use an existing format (I see maxmind a lot in
this thread).

\- Perhaps an API with heavy ratelimiting for people who need only a very
occasional lookup (for very occasional use, I use ip2location.com/demo
currently; for more serious stuff I download a database).

Having a free version with a nice license might also get you included in a lot
of open source software, which increases name visibility and makes people
write parsers for your format of choice.

A minor note on payment: accept Paypal and/or Bitcoin, or I (and many other
Europeans) can never pay you. These credit card-only services are impossible
to order from.

Idea for the diff: if you have a binary format, perhaps use this:
[https://www.chromium.org/developers/design-
documents/softwar...](https://www.chromium.org/developers/design-
documents/software-updates-courgette)

~~~
raquo
Why is paying with credit cards hard for Europeans?

~~~
dutchbrit
Because the majority don't have/want credit cards.

~~~
slapresta
I'm European and I'm yet to find a credit card field where I can't put my
debit card number. I'm not sure I understand what you're saying.

~~~
lucb1e
Debit card number is not your IBAN, right? Because that's the only number on
my bank card and it never works (because all forms want your "credit card"
number, I thought).

~~~
slapresta
Oh, no, it's not your IBAN. It's a number that looks like a credit card
number, but the card it points to is actually a debit card.

------
SteveCoast
I've been building exactly this:

[http://browserlocation.co/](http://browserlocation.co/)

I have a JS library coming also for use when the w3c geolocation fails. The
data is open and downloadable with cities etc (CSV right now). There's a
mailing list for updates, and anyone can contribute just by opening the
website in a browser!

------
PythonicAlpha
In the EU, the sales tax now (I think, since this year) must be paid to the
country, the buyer comes from (before, the sellers country was decisive). Also
the height of the tax depends on this. So, at least in Europe there are plenty
of small companies that could need such a thing, at least as long they don't
have a payment-provider that handles the sales-tax for them.

I myself, could also be interested, but I think, I will leave that to my
payment-provider ... (ugly, EU!)

 _Edit: Ok, I should not, but why the downvote, please? When I wrote something
wrong, than please explain!_

~~~
_delirium
You almost certainly don't want to be using IP geolocation to compute sales
tax, rather some kind of billing address explicitly provided by the customer.

~~~
PythonicAlpha
In that case, the customer could cheat. I am not totally sure about the
current discussion on that, but the problem is, that you definitively can not
rely on the address given of the customer, because in this case the customer
will always give a country with the lowest sales tax. Also you must provide
proofs for the location. The IP address is IMHO (and not only my opinion)
better and will be more likely be recognized as proof instead of relying
solely on some customer input.

But, you (because you are so sure about the best method) by sure know, that
the laws are explicitly for non-physical goods, where no delivery-address must
be provided, right?

Here is a link, how one company is doing it (forget about government
information, they seldom go so much in details!):
[https://rachelandrew.co.uk/archives/2014/12/08/consensus-
and...](https://rachelandrew.co.uk/archives/2014/12/08/consensus-and-vatmoss/)

They use the IP address to validate the user given address and for later
proof.

 _BTW: Downvoting all the time does not help the discussion. I try to discuss
this topic seriously and I was bringing some very relevant topic here, and am
constantly downvoted. (It is not my fault, that the EU came up with such laws,
and provide no useful guide how to implement it!)_

~~~
_delirium
In Denmark, at least, billing address is standard for non-physical goods as
well. It's validated via the payment method, typically credit card or bank
transfer. It's _possible_ to cheat this, too, but harder than just making up a
fake address. Afaik none of the main online sellers in DK are relying on IP
address currently, in part because accuracy is not good enough, and in part
because people traveling across borders is common (many residents of Malmö,
for example, have Danish IP addresses during the day and Swedish IP addresses
at night).

~~~
PythonicAlpha
A parallel poster posted an official link from the British Government:

[https://www.gov.uk/government/publications/vat-supplying-
dig...](https://www.gov.uk/government/publications/vat-supplying-digital-
services-to-private-consumers/vat-businesses-supplying-digital-services-to-
private-consumers)

There is clearly stated, that IP information is one of the proofs accepted for
the country determination.

So, maybe next time, please first ask before judge.

------
heyalexej
It shouldn't be free, it should be reasonably priced so you don't run out of
money and can keep this thing running for years to come.

There's huge demand for it - just search for geoip on Github. Some more
pointers are here [0],[1]. So many projects, but most of them work with
Maxmind data, which is the problem. Inconsistencies in the data they provide
makes it painful to make performant fallback solutions. Performance across
geolocations and lack thereof in "exotic" geos is another one. It seems none
of these services is deployed across the globe. There's a need for a blazing
fast, reliable service with granular, comprehensive data, that is performing
well for millions of requests across the globe, be it for mobile or web
traffic.

I found a great base[2] and I think Go is a perfect language for this kind of
service.

Here are the free checkers I tested the other day, the IP is a good example
for incomplete data:

    
    
        checkers = [
            'http://freegeoip.net/json/103.1.29.57',
            'http://geoip.nekudo.com/api/103.1.29.57',
            'http://ip-api.com/json/103.1.29.57',
            'http://ipinfo.io/103.1.29.57/json',
            'http://www.telize.com/geoip/103.1.29.57'
        ]
     
    

[0] [http://ip-api.com/docs/statistics](http://ip-api.com/docs/statistics) [1]
[http://geuis.com/2012/09/21/jsonip-com-close-to-
serving-10-m...](http://geuis.com/2012/09/21/jsonip-com-close-to-
serving-10-million-requests-a-month/) [2]
[https://github.com/fiorix/freegeoip](https://github.com/fiorix/freegeoip)

~~~
vld
ip-api guy here.

> It seems none of these services is deployed across the globe

Our free service has servers in USA, EU and Asia

> There's a need for a blazing fast, reliable service with granular,
> comprehensive data, that is performing well for millions of requests across
> the globe, be it for mobile or web traffic.

We have more than 2 billion requests per month!

~~~
blackice
I also recommend this service, been using it for awhile now and It's pretty
accurate.

------
tokenizerrr
That would be amazing. I'd love a datadump that could be accessed on my own
servers instead of having to hit an API. The format of the datadump doesn't
really matter to me, as long as there's documentation on how to read it.

~~~
dsl
Most geo data providers (like MaxMind) provide datafile downloads so you don't
have to use an API.

~~~
tokenizerrr
Right, but those aren't usually free ;) On the other hand there are plenty of
free (limited) APIs.

~~~
wise_young_man
They do have a free version which is just a bit less accurate than the
commercial version.

[http://dev.maxmind.com/geoip/geoip2/downloadable/](http://dev.maxmind.com/geoip/geoip2/downloadable/)

~~~
tokenizerrr
Oh, I didn't know that. How much less accurate? Will it accurately determine
country of origin?

~~~
egze
Yes, it will

------
rebelde
I have long thought that we could do better than maxmind. I have a website
with tens of millions of users all over the world that could help you build
this database is your want to lead the effort. I imagine showing users what
country we think they are in and they correct it if it is wrong.

~~~
mike-cardwell
I imagine browser language settings would be a useful indicator too. Not
perfect obviously, but highly useful given enough data.

~~~
rebelde
That, too. We could use a number of different indicators to help figure it
out.

------
Gys
So it would be a 'simple' table dump ? Updated daily by adding only the
changes ?

I would be interested. Although a service like ip-api.com is not expensive ($8
per month or so if commercial use).

The structure of the data seems very simple: ip address range plus a location,
I guess. So maybe just dump as a simple csv text file ? Always easy to
process.

You have to make sure the legal side is covered: how do you get your changes ?

------
clouserw
Mozilla runs an open/free geolocation service:
[https://location.services.mozilla.com/](https://location.services.mozilla.com/)

~~~
vardars
[https://mozilla-
ichnaea.readthedocs.org/en/latest/config.htm...](https://mozilla-
ichnaea.readthedocs.org/en/latest/config.html#geoip) "The geoip section
contains settings related to the maxmind GeoIP database." If I don't get it
wrong, this service relies on MaxMind's db for GeoIP.

~~~
cpeterso
The Mozilla Location Service does use MaxMind for GeoIP, but the service's
primary use is to map Wi-Fi SSIDs to longitude/latitude position. Mozilla has
its own Wi-Fi database continually fed new data from the Mozilla Stumbler app
(on the Google Play Store and F-Droid markets) and Firefox for Android. The
Mozilla database also has a partnership with commercial Wi-Fi company Combain
and the OpenCellID project for fallback results.

[https://blog.mozilla.org/services/2015/03/06/combain-deal-
he...](https://blog.mozilla.org/services/2015/03/06/combain-deal-helps-expand-
mozilla-location-service/)

(I work at Mozilla and I used to work on the Location project. :)

~~~
cpeterso
I pitched the idea of Mozilla creating a free GeoIP database by collecting
(opt-in) traceroutes from Firefox users to some Mozilla servers. The
traceroutes could then be overlaid to create a physical map of the Internet. I
know other projects have mapped the internet, but this approach would work
from the endpoints in, not from some centralized crawler.

------
Hengjie
A slight tangent but good feedback to OP regardless:

I use MaxMind at work but because we're on the RoR stack, we have to download
and commit the massive 30MB databases into our repository in order for it to
be used on Heroku. We use
[https://github.com/yhirose/maxminddb](https://github.com/yhirose/maxminddb)
which seems to be the only good option available that reads the new "mmdb"
format. It's a complete pain to have to commit such massive DBs for it to work
and would prefer for Rails to fetch the database upon initialization, cache it
for at most 2 days and redownload if it hasn't already been restarted by
Heroku.

If you can provide a solution to that, it would give us enough value for me to
pay a few dollars per month for it.

~~~
ericclemmons
I'm in a similar boat, but just have a license key on the machine (committed)
and use the maxmind updater to fetch our licensed db.

It happens once per provision, and a cron job to keep it up to date.

------
dutchbrit
On a related note, what's the best way to host big databases for free? As a
release on Github?

~~~
onethumb
Very interested, but worried that if it's free, it won't last or be
maintained, etc.

But in specific answer to this question, if it's really public and free, why
not see about adding it to Amazon's public datasets [1]? S3 is fast, durable,
available, and I've enjoyed these other data sets.

[1] [http://aws.amazon.com/datasets/](http://aws.amazon.com/datasets/)

------
nicolewhite
I would be interested. We _pay_ for Maxmind and still get a lot of inaccurate
data.

------
sajal83
Id be interested, especially if it used same file format as Maxmind's binary
dat file ... so it can be a drop in replacement.

~~~
dutchbrit
Added to the list! ;)

~~~
autarch
Another MaxMind employee here ...

Please don't try to use the dat file format. The binary files ending in ".dat"
are the Legacy format, which has never been documented. Use the MMDB format
linked by oalders in the top comment.

------
mike-cardwell
I would be very interested in such a thing. Where are you getting your data
from? Does it work for IPv6 too?

~~~
JosephRedfern
Hmm. Can you determine such things using BGP? I guess that wouldn't be very
accurate, though...

------
bytesandbots
You seem to be solving a very sought after problem which also makes it so much
more difficult. Where do you plan to source this much info?

Can somebody make me understand why ipv4 assignments along with their requests
are not made public?

~~~
devicenull
> Can somebody make me understand why ipv4 assignments along with their
> requests are not made public?

What do you mean? With any of the RIR's you can get it down to a specific
company. With ARIN, you're supposed to be able to get it down to a particular
customer. What more information do you want?

------
bisdata
On a related note, would anyone in the community be interested in an
inexpensive service that differentiates dynamic from static IPs? Dynamic IPs
would be accompanied by an estimate of lease duration.

~~~
lucb1e
I wouldn't want this to exist because it discriminates my dynamic-looking IP
address from server IP addresses (even though mine is statically assigned to
my Internet subscription) and I have enough trouble hosting things at home
already.

But I understand that's not really an argument: you might say you don't want
nukes to exist because they hurt people. I just wanted to mention it.

That said, maybe. Usually an rdns lookup does the trick already though
(a4-8e-ef-78.dynamic.ziggo.nl or cpe-12-34-56-78.rr.com or
80-100-131-150.adsl.xs4all.nl is clear enough).

~~~
bisdata
Yeah rDNS is helpful, but as you pointed out with your home IP, many dynamic
IPs (those that are rotated by an ISP) are actually granted very long leases.
The service I'm proposing would make an additional effort to estimate the
duration of dynamic leases.

------
dg81
Two considerations that help avoid the IP issue, but require user permission
and so have a big obvious disadvantage:

1) We can prompt mobile users to enable location/gps and get their location
that way.

2) We can prompt modern browsers to allow location services and get their
location that way.

This is if a user wants to have their location 'auto-set' instead of via
pulldowns.

3) A hard problem to solve is - mapping a unified set of city names to
timezones, and updating the timezones as per Daylight Savings Time and general
timezone updates.

Anyone have references for 3) ?

------
adminprof
I'd be interested if it included cities. If there was a javascript plugin to
automatically help contribute (is that possible), I'd add it to some of my web
pages. CSV is fine.

------
vivekv
I love a good free service where the data quality is improved. I have played
around with maxmind's service a bit but the data for India is far from
accurate. I have been hunting for a local database (no service) since I run a
IP lookup service [http://ipof.in](http://ipof.in) where I want the service
subsumed in. This is a free service for anybody who uses it. It supports
XML/JSON and plain text.

------
wiml
Have you considered simply contributing to the service at
[http://www.hostip.info/](http://www.hostip.info/) ? They seem to be doing
what you describe, and have been around long enough to have reasonably good
information (in my experience)— I'm sure they would make use of any additional
data, or hacking on their backend, which you might provide.

------
openservices
There seems to be heavy interest in a low-cost service, even free if it was
sustainable. Just curious if anyone has seen innovative pricing around
services, similar to how open-source software made expensive software much
cheaper or free, where the vendor made the main "service/data" much less
expensive but monetized in a different way.

~~~
openservices
So what I'm envisioning are services designed for zero profit. Prices are
indexed to AWS/Azure/Google prices; as they fall, the service's prices falls.
Customers who need predictable pricing can buy long term contracts, as they do
on AWS. Customers who use the service more, pay more, because the service
offers utility-based pricing. The goal is to make the service as cheap as
possible and maintained by the community. The source code and all data, except
private data (e.g., API keys), is open and editable by the community.

Think it would be an interesting way to deliver services and data, just not
sure how it turns into a business (since the API/service portion is designed
to be zero profit).

------
fale
I would be interested, personally and I believe many other developers would
also be interested :-)

------
linshibo
I prefer csv. I care more about the content it provided, such as location ,
city, longitude , latitude， isp. In my company, we need a accurate database
very much.And now we use IP2LOCATION-LITE-DB11.CSV and ipip.net .

------
Firegarden
I have a domain name you could use localipy.com. Just tell me your
nameservers.

~~~
Axsuul
I like the generosity here but no one in their right mind should consider this
kind of offer unless the domain was transferred to them. There's too much risk
involved and things that can go wrong unless a domain is 100% owned by you.

~~~
Firegarden
Right. I have so many names and I never put them to good use :S

------
wineisfine
Tons of devs are interested

------
rajacombinator
I'd be interested. I'm not sure daily updates add much value, how often do
these things change? If you have a way to provide more accuracy though, that
is very interesting.

------
j45
This would be very welcome. Also having SQL load files ready to go would be
something that could be very easy to become dependant on and start paying for
more frequent updates.

------
karmajunkie
I'll echo the "low cost, not free" thought. Stuff takes money to maintain. I
don't want to build a service that depends on an unsustainable business model.

~~~
notatoad
Maxmind _is_ low cost, not free, and has a pretty good track record. What
would you want from an alternative?

~~~
rebelde
I would want: \- Easier updates. Now it is a minor hassle to update my MySQL
database table from CVS. I need to do it again soon. \- Availability in
different formats. For my personal needs, I would love to download a MySQL
MyISAM table. \- More accurate data, including "ambiguous country" for
employees of multinational firms. \- Related open source tools through github.

------
anjanb
Yes, one initial download and then subsequent deltas on a regular basis. Will
be willing to pay. I know of plenty of Fortune 500 companies will be willing
to pay for it.

------
dg81
Location to Timezone would be of value to me. No IP involved, want to map a
Google Places / Location City value ~ to a timezone. Would pay monthly fee.

~~~
thecodemonkey
We offer this at [http://geocod.io](http://geocod.io) :)

~~~
dg81
I took a look thanks. Pricing seems good, however we have a very international
audience. We need to get timezones for geocodes (preferably Google Place IDs)
from all over Asia and the Middle East.

How good is your global timezone data? Is it possible to map Google Places
(from their places autocomplete form) to a timezone?

------
cryptodev
Can you provide some contact information? I would like to contribute as well,
I find Maxmind extremely unreliable when it comes to Home ISPs.

~~~
dutchbrit
Shoot me an email at sam.granger@gmail.com and I'll provide you with updates

------
ars
I'd definitely be interested, but I'd also be worried that you could really
sustain interest in doing this for a decade.

------
social_quotient
I have a domain I've not done much with that might be perfect for this
geoip.io

Let me know if interested or if we could maybe partner on this

------
cryptodev
Can you provide some contact info? Would like to help adding changes, I
personally find Maxmind inaccurate as well.

~~~
dutchbrit
Shoot me an email at sam.granger@gmail.com and I'll provide you with updates

------
dutchbrit
Thank you all for your feedback, I will continue working on this and make a
new post once this project goes live.

------
blablabla123
Me! Once or twice a year I need that stuff on different occasions and each
time I find it really annoying.

------
david_ar
Also see [http://hostip.info](http://hostip.info)

------
matiasb
Would it be possible to run some kind of GeoIP DB over a P2P/DHT network?

------
kenetik
Would LOVE this, possibly in json? Id even consider hosting it for you.

------
TheSwordsman
I think a lot of people would be interested. I'm also curious of the
implementation details. Open source? :p

To echo others, aim for low-priced and not free. That'll give you some
operating revenue to avoid it going the way of the dodo.

------
jack9
We do millions of lookups per day. We do not use Maxmind.

~~~
heyalexej
Mind to elaborate a little bit on this?

------
sintaxi
hell yes!

------
RandyRanderson
Note that you could build a simple API with Google App Engine:

[https://cloud.google.com/appengine/docs/python/#Request_Head...](https://cloud.google.com/appengine/docs/python/#Request_Headers)

and it would likely be free if the number of requests was below a certain
threshold per month.

