Hacker News new | past | comments | ask | show | jobs | submit login
Unofficial APIs (github.com/rolstenhouse)
297 points by tornupinside on May 20, 2020 | hide | past | favorite | 72 comments



Haha the pokemonGo Api! I was in the final stages of a bootcamp right when Pokemon Go came out, and a bunch of us were desperately trying to put ourselves on the map by doing something lit with Pokemon Go. We discovered that the spawns were cycled hourly, so if we could gather enough data we could quite accurately map out every pokemon in the city. Then we came up against not knowing anything about parsing the data we were just yanking back from requests to the API... I think some point in the authentication phase? So we saw that python library, saw that they had solved the problem, but it was using something we had absolutely no idea how to transcribe to Node, and this is like, a bunch of 3 month old freshfaced Javascript devs. Figuring out Python was kinda out of the question (better to focus on pitching ourselves as hardcore Node/Javascript devs in the upcoming job search).

And then, like, three days after we started the journey, someone had not only beat us to the punch, they had done so at a professional level. I don't remember which site it was, but not only did it work perfectly, it didn't require the "crowd sourced" solution to mapping out pokemon that we were counting on. The developer/s had somehow figured out how to just yank all pokemon locations. And on top of that the app was functional, gorgeous, even had its own URL (we were still in the "all our apps are laughinggiraffe.herokuapp.com domains" phase).

All in all, great experience. A nice fresh slap in the face to how much work we had ahead of us, and good fun had anyway.


There were amazing city maps for active pokemon spawns. It would show all currently spawned pokemon, as well as how much longer they would be there for. It was fun to see what is around you at any given point, but only marginally useful without notifications.

I noticed that one of the more popular SF pokmeon go mapping websites was using a simple REST API. You gave it a coordinate, and it would spit out locations for all pokemon within a mile radius or so. I created a background python task on my computer that would hit this API every 2 minutes with my apartment's location, plus like a mile range around it.

If it found any rare pokemon (Gyrados, Dragonite, etc.) the python program sent a message to a specific slack channel that I set up. It was an easy way for me to get push notifications on my phone when something spawned around me. From there, I would high tail it on my bike and catch the pokemon. I still have this Pokemon channel at my work Slack instance haha.

It was amazing. My pokedex filled up so quick. This approach only worked for about 3 weeks until Niantic majorly cracked down on their API, which rendered all the API scrappers obsolete. But that period was by far the most fun I had with Pokemon GO. I whipped that program up in like 3 hours on a Saturday and it was the only time in my life where I've felt like a l33t hacker, even though I was doing some very basic REST API operations. The actually impressive thing was the mapping website that was scrapping the actual Niantic API.


I had a friend that ran some of that software. The trick was to just make a bunch of accounts and just fake them walking all across the map to gather the locations. Basically crowd sourcing it, but with fake people.

The number of fake accounts was a function of how large of an area you wanted to cover and how often you wanted it to update.


I remember some really good public sites, but the quality cycled up and down and the Pokemon devs fought back a bit. I used some tool that basically was what OP described, but let me easily self-host a server. At the time I think there was some hack where the server component could fake location, so you'd just send it off to walk a geofenced area and could browse on mobile remotely while playing the game. Ran it off my laptop, then a DO droplet. Good times.


I got another fun story from the other side. I worked at one of the few small companies that provided free Openstreetmap-powered basemaps at the time: you could just grab our map tile URL, plug it into Leaflet or any other mapping lib, and get a instant basemap with OSM data, provided you gave credit to us and under some not really restrictive ToS. In fact you probably used us at the time.

The technology behind those tiles wasn't really sophisticated back then -- a few oversized and replicated PostgreSQL servers with PostGIS and the OSM data loaded and synchronized frequently with a few materialized views on top, and a bunch of servers painting tiles with Mapnik using them as a source (there are a few options that started to prove to be more efficient at the time, but that's what we decided to go on then for other reasons), with a few layers of caching on the front. Tile rendering times for cache misses weren't really good depending on the complexity of the requested tile, given most stuff wasn't prerendered, but once things got cached and since most popular maps were localized to specific regions that got quickly cached, this worked pretty well after the first visitor had came.

But then... Pokemon Go and your bazillion maps came with the worst scale test for our design you could ever think of: a volume of tile requests a few levels of magnitude higher than usual, of locations from the half of the world playing Pokemon Go back then, zooming to _their streets_ (which were randomly distributed all over the world and therefore most probably uncached at that zoom level) to try and find their closest Charmanders.

Needless to say, those were some few nice days of firefighting and playing whack-a-mole replicating databases, adjusting caches, banning requests from the worst offenders and, at some of the worst points, everything that included the word "poke" on their domain.


Mapzen? :)


Stories like this is why I love the HN community


Was the site you remember instapokego.com per chance? :-)

It's down now but here's the archive: https://web.archive.org/web/20170308154111/www.instapokego.c...


There were a lot of devs working on that at the time, don’t feel bad.


> Designed to inspire your next Friday night hack.

I've been hit with some burnout recently, and this is the kind of inspiration I need. Small, fun, short term project to get some juices flowing, as opposed to forcing myself to work on personal projects I currently don't have passion for, but feel I need to work on out of some weird sense of obligation.

Thanks, this is actually a really great help.


Upvoted, it's also what eventually got me out of a burnout. Didn't program side projects for a long time (apart from work) and then just started small hackish things for personal use.


To find more of these unofficial APIs, Google:

[site:github.com "unofficial" AND "API"]

Or click here:

https://www.google.com/search?q=site:github.com+%22unofficia...



Shameless plug, but here's a website I created for April Fool's Day a while back that uses node-dominos-pizza-api: https://ryankoval.pizza/ (source at https://github.com/rkoval/order-me-pizza)

The package works pretty well in my experience. Something that I found mildly interesting though is that Dominos has not changed their API since I implemented it. Since this is unsupported (and honestly, probably frowned upon), I was expecting this to break early/often; thankfully, that has not been the case.


That's awesome. I've always wanted to set up my own hardware button to order pizza for fun haha.



Awesome! I was just looking at integrating Omnifocus and Notion to keep better notes and history of tasks, letting Omnifocus focus on just the task management side of the world.

It'd be cool to link up a script that takes a new task in Omnifocus under the right tag, and push a template into Notion for the note taking, and sync a link back into the notes field of Omnifocus for quick access.


Notion plan to release an API soon. They included in their pricing plan update yesterday.


No offense to the people who built this, but sharing your username and password with random packages on the internet seems like a bad idea, especially for finance apps.


Working at a cybersecurity *aaS company has both made me paranoid about 3rd party code that I pull and also made me realize that at some point paranoia is paralyzing and some amount of risk is necessary.

Bad idea? Likely.

Worth the squeeze? YMMV.


The bad idea in this scheme is to not take a look at what the code does, considering these are open-source packages.


I agree, but that's not sufficient in an open source software supply chain. You also need to inspect the dependencies and you need to do this every time you pull any new versions.


Sharing your username and password is considered "high security" nowadays whereas merely "standard security" would be installing a root kit or running shell code each time you want to install something.


Tangentially related, but I've been working on something for merchant websites (Amazon, Target, Walmart, etc) that you can feed a product url, and get back product information (image, title, price, availability, etc). It takes ~10 lines of code to write a driver for a merchant, and they are fairly stable if written properly.

My goal is to open source this, and allow anyone to contribute new / update existing drivers. Would there be interest in something like this?


Please do. I'm tired of going to multiple sites to compare prices.


Awesome!

I tried writing a CLI tool for ordering Chipotle, but I ran into some dynamically generated headers, that made it near impossible to authenticate. In the end, I gave up.

Would love to see more examples and see how issues like this can be handled!


I’ve had this come up a few times when doing API snooping, and what I’ve done is decompiling the Android version (if available) of their app using jadx [1]. You get pretty legible Java, sometimes with Kotlin artifacts, sometimes with obfuscation, but in combination with some MITM snooping you can see how auth/headers/hashes are done. Sometimes you even find internal API endpoints with a security issue or two.

[1]: https://github.com/skylot/jadx


Thanks!

I've used jadx once before to decompile a steamship line app, but it was just curiosity with no end goal in mind. Didn't try it with the chipotle app.

Next project!


I wrote a Ruby one for the NBA.com API but not sure if it still works though as I haven't used it in a while (and also slightly embarrassed at some of the code I wrote back then, which I guess is good!)

http://github.com/FanaHOVA/nba_rb/


“Anything can be an API - if you are stubborn enough.”


This is distinctly more SFW than the similar aphorism I've heard.


I tried playing Final Fantasy XI on a private server for a week, but found the game to involve too much grinding. Then I noticed an SQL injection in their auction house web page, which allowed me to dump prices of items at various times and build a private API. I used this to inform my investment, got quite wealthy and bought equipment for my toon, then quit the game shortly after.


Along similar lines, and unfortunately quite out of date at this point(!) but I've maintained a list of Monzo Bank APIs for a while:

https://github.com/billinghamj/monzo-api


I love the tinder API. A few years ago I was planing to shift to another locality in my city but couldn't figure out where. I eventually decided to move to an area with the best looking women. Used the python tinder API to move around the whole city with GPS coordinates while setting the shortest possible radius and saved profile pictures tagging them with geo location. The end result was a database of geolocation mapped to profile pictures of women in that area.


That's really freaking creepy...


BRB: Going to pitch some real estate agents with an automated system to add Tinder women with super hot profile pics tightly geolocated to their realestate listings...


There are hot women in your area

Here's the data in a spreadsheet


You really can't do anything with that data unless you are matched with the other person.


what's it like living above a yogurt shop?


Personally I’ve found using ‘private’ web APIs to be a bit of a mixed blessing. Having side projects break with no notice is very frustrating, at the same time they’re a huge improvement on scraping for ad-hoc tasks.


Does using an unofficial API authenticated with someone else (e.g. a customer)'s account count as a violation of the ToS? In other words: who gets reprimanded/sued? The final user or the messenger?


Someone please correct me (and someone for sure will), but I don't think you can get sued in most places for breaking the terms of service. The only thing (which is hinted in the name) they can effectively do is boot you off the platform.


Don't all of these libraries require you to provide your own credentials?

I do think this violates their ToS anyways though.


I'm sure it would, in some cases at least. I'd be more interested in understanding who gets the blame.


The few things I clicked on are scripts/modules you'd run yourself, so they're not really APIs in the commonly used sense, but website-wrappers that let you use the website as if it was an API. In that case, the user and the messengers are one and the same, the developer of the wrapper isn't involved at all in the transaction.

If it was Unofficial-API-As-A-Service, I'm pretty sure both would be in violation for most services. The user at least for sharing their account credentials, the UAAAS provider likely for some thing in the fine print about only being allowed to use the website for the intended purposes. I doubt either will get sued, the user will get their account cancelled and the provider will get their servers blocked and an angry letter from the lawyers telling them to stop.


Considering that Plaid, which is "please enter your bank username and password so we can validate you are who you say you are" is very common, just assume nobody cares about anything


Hacker News also has an unofficial API: https://github.com/cheeaun/node-hnapi


It also has an official API: https://github.com/HackerNews/API


HN has unofficial write APIs, since they don't prevent cross-site request forgery. This enables HN apps that upvote/comment/post to work. Thanks, HN ;)


Your exemple is more a third-party API than an unofficial api as the one from this post.

The one from this post are self-hosted scrapper-wrapper library API.



If too many people are using the unofficial API for a product, is it right to assume that building a competing product offering a better official API, would be successful?


My guess is that the two things are unrelated, at least looking at this list.

The API itself isn't the reason people use the product.


That Notion API is pretty slick. His example of using it for task management and dashboards is wild. May have to give that a whirl.


Strange they list Coinbase as an "Unofficial API".

The trading API is very much so official, and dates back to the GDAX days.


I think you misunderstand what this repo calls an "API".

The Coinbase "API" in this repo isn't the protocol (eg an HTTP REST API implementation), but the 3rd party Python library which speaks the protocol[1].

[1] https://github.com/danpaquin/coinbasepro-python


I must be lost then.

The python code is an implementation of a program that uses the Coinbase API to make trades, fetch market data, etc.

In my day, we'd call that a Program, Library or an SDK. Definitely wouldn't call that an "Unofficial API".

When I think of "Unofficial API", I think of an API that wasn't intended for public use and is undocumented and supported by the company. Like the Pandora API some music players have reversed... or the Pokemon GO API people are talking about in this very thread.

Coinbase released, documented and supports their API. Anything that uses that API to do things is just a program, or library.


Coinbase may have an official API, doesn't mean they support the Python implementation of their API, hence the 'unofficial' as in 'not maintained by the coinbase dev team'

These implementations are there for devs to save time and simply import the API implementation as a module and use already made functions to do the API calls


OK, but what you described isn't an API then. It's a library.

The Coinbase team, to my knowledge, doesn't maintain any implementation... making all implementations "Unofficial" in that sense.

I still fail to see how this python code is considered an "Unofficial API".


I agree it's a library but these days anything is being called an 'app' it's kind of the same with API and API implementation (library). It makes it confusing


Been meaning to do something with the Robinhood one


take a look at Alpaca! particularly, the paper trading is a nice and easy way to implement strategies w/no money on the line


How does this work? Using scrapping?


oh this is awesome! would be great for new data scientists as well


From a practicality standpoint, I strongly discourage using unofficial APIs, as tech companies nowadays are very prone to sending out C&Ds. If you have a use case where the official API doesn't suffice, then do not redistribute anything obtained from the unofficial API, and definitely do not attempt to commercialize it.


Max remember this is Hacker news. You sound like you're on Nextdoor complaining about the neighbors who are in clear violation of HOA yard maintenance policies.


It's a warning to avoid wasting valuable time, speaking from experience.

I used to use unofficial APIs years ago. There's a reason I don't anymore.


Its hit or miss, the NHL for example powers their website from their API entirely yet its 100% open to the world and tons of people build all manner of things off of it yet they seem to give zero shits about people using it at all.


i sort of agree here. it would really suck to build something you love/proud of and maybe instead of getting a C&D, it simply changes or gets shut off and your project is dead. Rather put my efforts into something that I know will have more of a chance of surviving. at the same time i totally get if someone has an itch they just want to scratch on a weekend at the same time


I'm actually the current owner of this repo!! I started it because I love working with unofficial APIs for fun hacks that improve my life. During college, I had an alert that told me when the Krispy Kreme donuts were fresh built off an unofficial API that promised enjoyable donuts.

If you're interested, I have a newsletter to update you when new repos are added https://forms.gle/e8nCivpTBNftNtgGA and to feature interesting stories from the community.


Please don't use multiple accounts to promote things on HN. The community feels extremely strongly about this and will bring out the Searzalls and use much less kind words than "using multiple accounts to promote things".

I'm going to leave the submission up because it struck a genuine chord with people, but that's a lucky escape. Normally, the likelier outcome is bannage.


Thanks for creating this - super cool!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: