
Hacker News API - kevin
http://blog.ycombinator.com/hacker-news-api
======
christiangenco
Oh man you guys, patio11 has generated _massive_ amounts of content:
[https://hacker-
news.firebaseio.com/v0/user/patio11.json?prin...](https://hacker-
news.firebaseio.com/v0/user/patio11.json?print=pretty)

I count 8,483 submissions. I'm sure there's something interesting to be done
with all of this data. A word frequency chart?

\---

Edit: So apparently there's a ruby gem that lets you feed it a body of text
and generates pseudo-random phrases based on that text.

I present to you the patio11 impersonator:
[https://gist.github.com/christiangenco/e8d085e47479be0131e1](https://gist.github.com/christiangenco/e8d085e47479be0131e1)

One of my favorites:

    
    
        A nice set of challenges -- kitty at a school with tens of thousands of bucks a year or less immediately.
    

\---

Also, a word count on patio11's submissions: 1,052,351. For comparison, all 7
Harry Potter books total 1,084,170 words. patio11 has written the entire Harry
Potter series worth of content on HN. Just... wow.

~~~
minimaxir
In February, I did an analysis on all Hacker News stories:
[http://minimaxir.com/2014/02/hacking-hacker-
news/](http://minimaxir.com/2014/02/hacking-hacker-news/)

Yesterday, I published an analysis on all Hacker News comments:
[http://minimaxir.com/2014/10/hn-comments-about-
comments/](http://minimaxir.com/2014/10/hn-comments-about-comments/)

There's a lot of interesting trends in the data. Let me know if you want to
know anything in particular and I'll get back to you. :)

~~~
danieltillett
I would be interested to know the average hourly cost of HN in lost
productivity :)

~~~
skierscott
I'd like to know the average hourly amount _earned_ from HN. I know personally
HN has helped me greatly in the projects I do at work.

~~~
danieltillett
Yes this is an interesting question too, but one that is much harder to
answer. I love HN, but it sure gets a lot of activity during work hours :)

~~~
pessimizer
We seem to think that time spent learning is lost labor, rather than that time
spent laboring is lost time for learning, although we know that both learning
and laboring are required for productivity, and that learning is capex.

------
airlocksoftware
This... is cool, but also kinda sucks for me. I've invested dozens of hours
into writing an extremely complicated scraper for my Android version of HN.

[https://play.google.com/store/apps/details?id=com.airlocksof...](https://play.google.com/store/apps/details?id=com.airlocksoftware.hackernews)

The newest version (still under development, probably a month or two from
release) adds support for displaying polls, linking to subthreads, and full
write support (voting, commenting, submitting, etc). I'm fine with switching
to a new API (Square's Retrofit will make it super easy to switch), but
without submitting, commenting, and upvote support I have to disable a bunch
of features I worked really hard on. Also it would've been cool to know this
was coming about 3 months ago so I didn't waste my time.

Anyways, quick question on how it works -- when I query for the list of top
stories

[https://hacker-
news.firebaseio.com/v0/topstories.json?print=...](https://hacker-
news.firebaseio.com/v0/topstories.json?print=pretty)

it just returns a list of ids. Do I have to make a separate request for each
story

[https://hacker-
news.firebaseio.com/v0/item/8863.json?print=p...](https://hacker-
news.firebaseio.com/v0/item/8863.json?print=pretty))

to assemble them into a list for the front page, or am I missing something?

~~~
piyush_soni
So why, in the first place, would I want another mobile app rather than just
opening the fully functional website (which is pretty simple & basic already)
on my mobile browser?

~~~
airlocksoftware
Because it can be better designed, use common design / navigation patterns of
your mobile OS, notify you when you get a reply, change the text size, change
the theme, have richer animations, and allow you to automatically share
content from other applications directly to HN?

~~~
piyush_soni
Yeah. I'm just a bit averse to apps scraping data for the reasons you
mentioned. It should have been the work of the mobile website, not an app.
Speaking purely from the user's point of view (not the developer - I realize
this is a community full of app developers) - one can't just keep installing
apps for every website which is not mobile efficient yet. You all must have
seen a lot of websites showing messages like "Welcome, we have an app, pess OK
to install that, or Cancel to continue". Most of those websites don't do
anything which a mobile website couldn't.

~~~
enobrev
I agree with you in theory, but most mobile websites are poorly thought out
and implemented - if at all. I definitely don't download apps for every site I
use, but for the ones I use daily, I generally find I need to. Native OS
interactions seem to be difficult to get right in the browser.

HN is definitely an example of a site that isn't ideal in a mobile browser.
For instance, if you have the ability to downvote, it's incredibly easy to
mistakenly downvote when you mean to upvote because how close and small the
buttons are. There's other added functionality, like tracking who I've upvoted
/ downvoted in the past as well as tracking un/read comments when returning to
a thread. In the browser, I use a chrome extension for this, and on my phone,
I use airlocksoftware's app. (side-note, I wish said state carried between the
extension and the app)

The developers of HN are surely capable of creating a mobile website that
could work just as well, or even better than a mobile app. But currently, it's
not ideal. And for that reason, I completely appreciate airlocksoftware's (and
the devs of other HN apps) for their efforts.

------
Livven
I've been working on a Hacker News client for Windows Phone over the past
several weeks and am very close to an initial release, so I feel somewhat
ambivalent about this.

On the one hand, of course it's great that HN is finally getting a proper API
and also modernizing its markup (which is a mess even if you ignore all the
tables – for example, the first paragraph in a comment usually isn't wrapped
in <p> tags), but on the other hand this current v0 version is very lacking
and impractical for a regular client application.

Since the top stories (limited to 100) and child comments are only available
as a list of IDs a client app would have to make a separate HTTP request for
every single item, which is obviously not something you'd want to do
especially in a mobile environment. Other lists apart from the top stories
(new, show, ask, best, active etc.) don't seem to be available at all right
now.

Of course this is just the first version, and the documentation promises
improvements over time – which I don't doubt at all – but there's no clear
indication that the API will be at feature-parity with the current website,
even excluding anything that requires authentication, by October 28. So this
means that I – and other developers of client apps or unofficial APIs – will
probably have to write new scraping code once the new rendering engine (which
I assume refers to the website) arrives instead of being able to switch to the
new API immediately.

Now I guess I might just be needlessly worried, especially since the blog post
explicitly says that the new API "should hopefully making switching your apps
fairly painless", but then why not wait until it's actually ready for that
before making the announcement? Putting a half-baked API out there a few
days/weeks (?) in advance before it's fully fleshed out doesn't seem all that
helpful, at least to me.

~~~
kevin
Use the Firebase libraries rather than the REST one to efficiently handle
requests. I believe it uses a websocket internally. "It does all the work for
you and is awesome." to quote Nick.

~~~
kogir
Also, while they don't have a Windows Phone API I previously wrote a COM
Windows Scripting API to use it via Chackra from an ASP.Net app.

I think it's even easier now that JS is a supported app language.

~~~
Livven
Are you referring to the WebSocket API that the Firebase SDKs use internally?
It doesn't seem to be documented anywhere so I guess it's only slightly better
than scraping HTML ;)

~~~
kogir
I set things up so I could actually run their whole JS SDK and talk to it from
C#.

~~~
Livven
Thanks for the tip, I actually just figured that out myself a few minutes ago.
Should be good enough until a proper SDK arrives.

With access to a Firebase SDK the only major additions the API needs to become
a viable replacement for existing read-only client apps would be support for
all the other lists apart from top stories (new, show, show new, ask, jobs,
best, active) and more than 100 items for each. For apps that need write
access I'd suggest keeping the current website on a separate subdomain until
that is implemented into the API.

------
shill
I wrote a stupid simple wrapper and pushed it to PyPI. My excuse is that I
needed to learn how to use setuptools today.

    
    
        pip install hackernews-python
    

Usage:

    
    
        >>> from hackernews import HackerNews
        >>> hn = HackerNews()
        >>> hn.top_stories()
        [8422599, 8422087, 8422928, 8422581, 8423825...
        
        >>> hn.user('pg')
        {'delay': 2, 'id': 'pg', 'submitted': [7494555, 7494520, 749411...
    
        >>> hn.item(7494555)['title'])
        Hacker News API
    
        >>> hn.max_item()
        8424314
    
        >>> hn.updates()
        {'items': [8423690, 8424315, 8424299...], 'profiles': ['exampleuser',...]}
    
    

[https://github.com/abrinsmead/hackernews-
python](https://github.com/abrinsmead/hackernews-python)

~~~
BaryonBundle
Can't get it to work, e.g., for `hn.top_stories()` I get:

    
    
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
          File "hackernews.py", line 23, in top_stories
            return r.json()
        TypeError: 'list' object is not callable
    

Tried in both python 2.7.3 and python 3.2.3

EDIT: You need a relatively new version of requests for this to work. The
version packaged with Debian Wheezy is too old. Use pip.

~~~
shill
Thanks for the feedback!

Prior to version 1.0.0 response.json() was a property, and was not callable.
That's probably what caused your error.

I'll add a minimum version to requirements.txt for the folks that install this
package manually.

------
dstaley
Decided to recreate the Hacker News homepage using Ember and the new API. I
was really pleased with how easy it was!
[https://realtimehackernews.firebaseapp.com/](https://realtimehackernews.firebaseapp.com/)

~~~
ryanseys
Very cool. Would be neat if changes were more pronounced when they happen.
Will the posts change order in this demo if they change on the homepage?

~~~
dstaley
Sadly no, but I'm looking into how to make that happen! It's a result of my
particular implementation, not a fault of the API.

------
dang
To everyone asking about logged-in access and write access: this is just a
first release! Where it goes from here will depend, in good iterative fashion,
on what people want.

~~~
jffry
I think part of the angst centers around the bit where the markup is going to
change. Lots of apps for HN that scrape the UI rely upon the current site to
enable things like voting and other signed-in-only actions, for which there is
(currently) no first-class API way to do these things. Even if the endpoints
for voting change, getting other information (i.e. which items you voted on,
so the vote arrows hide) is still markup-dependent.

------
minimaxir
How does this differ from the Algolia HN API in terms of data access?
([https://hn.algolia.com/api](https://hn.algolia.com/api)) I was able to
download all HN data recently with ease using that endpoint. Authentication?

EDIT: After looking at the documentation there are two new aspects of the
Firebase API not in the Algolia API:

1) Ability to see deleted/dead stories.

2) Endpoint for user data.

Question to kogir/dang: Has the "delay" field (Delay in minutes between a
comment's creation and its visibility to other users) always been there?

~~~
gwern
I'm also curious whether it removes some of the limitations of the Algolia
version; I wanted to download my content for some statistical analysis (notes
at [http://www.gwern.net/HN](http://www.gwern.net/HN) ) but I discovered that
it seems there's some hard limits to how much of my data I can reach:
[https://github.com/algolia/hn-search/pull/36](https://github.com/algolia/hn-
search/pull/36)

~~~
minimaxir
There's a cheat to work around that limit: use the created_at_i parameter.

Example: [https://github.com/minimaxir/hacker-news-download-all-
storie...](https://github.com/minimaxir/hacker-news-download-all-
stories/blob/master/hacker_news_getallstories.py)

~~~
gwern
I don't know Python, so I'm not sure what your source code is doing. At a
guess, you've hacked together some sort of repeated queries thing with a time-
window?

~~~
minimaxir
> Get 1000 entries and process them. > Take the timestamp of the last entry. >
> Requery the API, asking for articles made before that timestamp

Repeat.

------
jamest
[Firebase founder here] This is pretty exciting for us, we're glad kogir,
dang, kevin and sctb chose to expose HN's data through Firebase. We're seen
quite a few startups (and big companies like Nest) do this, since building,
maintaining, and documenting a public API often isn't a easy task.

~~~
nacs
How does this API work with Firebase?

Is HN data already in Firebase (as its primary data store) or is content from
HN's DB getting 'mirrored/cloned' on-demand to Firebase for the API?

------
jcampbell1
This makes it really easy to add average karma to the comment section for
every user. For instance, you can paste the below into the console, and should
add average karma data for each user.

    
    
        Array.prototype.forEach.call(document.querySelectorAll('a[href^=user]'),
        	function(v,k) { 
        		var s = document.createElement("script");
        		s.src = '//hacker-news.firebaseio.com/v0/user/' + v.innerHTML + '.json?callback=ud_' + k;   
        		document.head.appendChild(s);
        		window['ud_' + k] = function(user_data){
        			var avg_karma = user_data.karma / user_data.submitted.length;
        			v.innerHTML += ' (' + avg_karma.toFixed(1) + ')';
        		}
        	} 
        );

------
jaredsohn
Here is something I built with the Algolia API awhile back and just haven't
gotten around to cleaning it up to post here.

It lets you download all comments/stories for a user as a JSON or CSV file,
breaks down karma between comments and stories, and plots comment/story
counts, karma, etc. over time on a line chart (clicking will show you the
details via an hnsearch).

Also I built some npm modules so you can get this information via the
commandline.

[http://hnuser.herokuapp.com/](http://hnuser.herokuapp.com/).

Example:
[http://hnuser.herokuapp.com/user/tptacek/](http://hnuser.herokuapp.com/user/tptacek/)

The Chrome extension hasn't been updated for awhile (it just superimposes a
small amount of this information on the user page).

------
josephwegner
I really appreciate giving a 3 week heads up before moving to a new frontend
structure. It's a nice gesture, but I have this horrible feeling that there's
only about a 10% chance that my Hacker News app gets updated in time.

I know you can't not iterate because people are scraping, but it does stink.
At least this will make everything more future-proof going forward.

However, it may be nice to give a bit more heads up than 3 weeks. I know a lot
of apps can take ~2 weeks to get through the review process for iOS.

~~~
dang
As someone who poured countless hours into meticulously scraping the HN markup
and faces the prospect of having to port all my code with dread, I'll probably
be pleading for an extension alongside you.

~~~
ukd1
A preview release / staging version would help those of us with scrapers
update it, without having a so much downtime / scramble when it's finally
released.

------
bennyg
I've been waiting forever for an API from HN, but unfortunately I will not be
using it for my app ([https://github.com/bennyguitar/News-YC---
iPhone](https://github.com/bennyguitar/News-YC---iPhone)).

I've built a library for iOS
([https://github.com/bennyguitar/libHN](https://github.com/bennyguitar/libHN))
that handles scraping, commenting, submitting, voting, etc pretty well and
allows me to make as few web calls as necessary to use HN. It looks like I'd
have to drop functionality and completely change the networking scheme to
match this API - something I'm not willing to do yet.

Correct me if I'm wrong here, but to get every comment on a post, I'd have to
recursively get each item for each child. Instead, right now, I can make one
network request and get all comments for a story. Granted, I have to parse the
HTML (which I hate), but it's a much cleaner solution than going through every
item, checking the children and then getting those items ad infinitum. Again,
I just glanced over the documentation, but that seems untenable to me.

~~~
dailen
Check the article btw

"Most importantly, the reason we released an API is so that we can start
modernizing the markup on Hacker News. Because there are a lot of apps and
projects out there that rely on scraping the site to access the data inside
it, we decided it would be best to release a proper API and give everyone time
to convert their code before we launch any new HTML."

~~~
bennyg
Yeah, the only problem is that I don't want to cut major functionality just to
use their API. Things I do, that it doesn't look like the API handles:

    
    
        - More than 100 stories
        - Best, Top, Ask, ShowHN, Jobs, User Submission Posts
        - User Management (logging in/out)
        - Commenting
        - Submitting
        - Voting
    

My app doesn't just function as a reader, which is what this API seems geared
towards with the v0 release, it functions as about as full-fledged of an HN
client as you could get. There's a couple things that I haven't built in yet
like changing your about me text, but those were on the roadmap.

I'm actually thinking about storing the configuration of how my app scrapes
online such that if the HTML markup changes, I won't have to push huge
sweeping changes to the App Store to get my app online again. I just deploy to
Heroku and the app will handle that configuration and scrape correctly sans
pushing to Apple.

------
s9w
I welcome the idea, but this barely qualifies as an API. The most useful part
is the "current top stories" \- but what timeframe exactly? Seems to be over 3
days at least and can't be customized. And even my test parsing of the 100 top
stories took a good minute.

And that returns only the ids, nothing else. To get basic information like the
score, title or url you have to lookup the ids individually. And even the
story items do not contain such basic information as the number of comments.
And you can't calculate it yourself since only the top comments are even
returned (as ids of course). So you'll have to recursively dig through the
comments to get the number.

This is even more curious as there is a very solid Algolia API where you can
filter for submission time, story score, number of comments and even return a
greater number of results + access page numbers to get even more.

To get the information of a single algolia api call you will need hundreds or
thousands (in case of nested comments) "official" API calls. Hoping for
updates

------
eevilspock
If _up /down vote data_ were included in the API, much needed experimentation
on collaborative filtering would be made possible! This is _Hacker_ News after
all.

Right now one team, Ycombinator, is trying to fix important issues in the
ranking and moderation of posts and comments. Many of us are frustrated by the
increasing domination of popularity (and hatred) over quality and relevance. A
lot of good submissions and comments are simply buried, never to be found.
There is too much muck to have to wade through. The timing of posts and
comments plays a much larger role than quality. I could go on and on.

Imagine a Netflix Prize-like flowering of experiments and collaboration,
leveraging the hacker community's collective smarts and enthusiasm. Many of us
have ideas, but right now are unable to test them. What a shame if a great
idea dies on a notepad.

There are two possible issues with opening up voting data: gaming and privacy.
If having vote data allows someone to game the front page, then only include
it with some delay (2 days?) so that it could't be used to game the front
page. This will still allow experimentation with collaborative filtering
algorithms and the like.

My take on the privacy issue is that anonymity isn’t that important for a site
like Hacker News:

1\. Startup culture is about straight talk, putting your money where your
mouth is, and open critical feedback, both in the giving and receiving. There
are precedents for exposing voting data (e.g. Quora, Facebook, Stack
Exchange).

2\. HN is not aimed at political discussions or other topics where anonymity
can be paramount.

3\. Pseudonymity is sufficient for those who don’t want their votes and
comments tied back to their actual identity.

Thoughts?

I would love to hear from others who yearn to experiment with alternate
algorithms and strategies for improving Hacker News.

~~~
dang
There are many legitimate views on this, but FWIW mine differs from yours. I
believe that anonymity actually is important for a site like Hacker News, and
the odds of us ever publishing the vote data—even pseudo-anonymized—are small.
Sorry to disappoint.

~~~
eevilspock
Daniel, I understand. Do you or any others at Y-Combinator have any thoughts
on how the hacker community could experiment in the areas I mention above, or
whether you guys even think such experimentation would be valuable?

------
comeonnow
I built a scraper around 3 years ago (been through a few usernames since
then), and I've had to change it once 3 months ago because the HTML output
added quotes around HTML attributes.

Even though it's read only, I'll continue to use my scraper rather than the
API simple because it's one request, rather than the API would require one
request for the top IDs and then one call per story, so it would be 31 calls
instead of just 1.

Unless I'm missing something, it seems fairly poorly designed for top stories,
and non existent for new stories.

\------

EDIT: Looks like I missed the text about updating to a new rendering system in
3 weeks time, and to iterate designs faster to allow mobile friendly theming.
Looks Like I WILL be updating to use the API

~~~
tomw1808
yeah, I just have the same problem here... and then I have basically the same
question as someone mentioned below... new stories through the api? do we have
to get the max-id and then get everything below the max-id and check if its a
story? and other ideas?

------
jxm262
Yay! I've been wanting something like this to come out. I've been playing
around with some new tech stacks and built a css replacer of hacker news, but
always wanted an actual api to make it easier.

[http://jmaat.me/hn](http://jmaat.me/hn)

There's a bunch of css pages that come out for hacker news, but I couldn't
find anything that aggregates them. This will be alot easier to extend and
customize the site.

I'm not seeing any api's for the jobs or show sections though? Hopefully this
might come in the future?

~~~
minimaxir
Why do you need an API to replace _only_ CSS? You can do that with an
extension like Stylish/Greasemonkey.

~~~
jxm262
Well personally, I didn't want to install any browser add-ons. I also had some
other ideas, like aggregating reddit and hackernews posts, but would need to
scrape for that (unless there's an external api I'm missing).

------
kogir
Just pushed the ability to subscribe to all changes and new items:
[https://github.com/HackerNews/API/blob/master/README.md#live...](https://github.com/HackerNews/API/blob/master/README.md#live-
data)

------
ssorallen
The Firebase JavaScript library makes make this impressively straightforward
to use. I built a clone using React.js and Firebase's library. Because v0 of
the API requires a request for each news story, it's not possible to use
Firebase's React mixin yet.

[https://github.com/ssorallen/hackernews-
react](https://github.com/ssorallen/hackernews-react)

~~~
insin
Here's another React version which also does comments (and allows you to fold
them - I needed to write a userscript for that before!). It's using react-
router to switch between top stories, comments, individual comment and user
profile pages.

[http://insin.github.io/react-hn](http://insin.github.io/react-hn)

I've just gone for it with Firebase's React mixin, binding everything as an
object, since their devs in this thread don't seem concerned about rate
limiting. The mixin seems to throw an error every time I try to unbind, which
I'm just catching and logging for now.

Edit: I just watched this comment pop up live in my version - pretty neat :)

~~~
ssorallen
Nicely done! Binding every HN story as an object would have been the simplest
approach, that makes sense.

Is the source available somewhere?

~~~
insin
Original source is here until I set up a proper repo (after modularising and
setting up a build process):

[https://github.com/insin/insin.github.com/blob/master/react-...](https://github.com/insin/insin.github.com/blob/master/react-
hn/app.jsx)

------
andrewstuart2
I'm definitely excited about the API and the future possibilities with it.
Looks like a great start. I do have a few questions and suggestions, though.

Is there any chance of getting more than just the top 100 stories returned? I
think it will be a lot more useful for api consumers if you can use a query
parameter to set the limit (within reason, usually 1,000) and a number of
results to skip. For now, scraping is still more desirable to me since I can
retrieve any number of results in their current order.

Better yet, but more complex: a number to skip and a certain timestamp so I
don't see the same article on two pages due to natural upvoting, downvoting,
or rank decay.

Also, if there's any flexibility still with property names, I'd suggest these
changes for clearer semantics: "deleted" -> "hidden" (since they're obviously
not deleted) "by" -> "author" (for more clarity) "kids" -> "children" (the
common convention)

------
paulsutter
Please do allow other sites to use HN logins. Then the community could develop
useful sister services.

For example, a site where HN members can upvote and rate different development
tools, libraries, IDEs, management tools, etc. All with backlinks to HN
discussions. It's a great community and there are many ways we could share
knowledge and experience.

------
bambax
Any kind of rate limitations we should be made aware of?

~~~
sararob
[Firebase Dev Advocate here] We don't currently have rate limits. I'd
recommend using our SDKs, they handle connections more efficiently than
dealing with the REST API. You can also run your own server process using our
Node
([https://www.npmjs.org/package/firebase](https://www.npmjs.org/package/firebase))
or Java libraries
([https://www.firebase.com/docs/android/quickstart.html](https://www.firebase.com/docs/android/quickstart.html)).

~~~
dmunoz
Any kind of rate limitations with the REST API we should be made aware of?

~~~
kevin
You really want to use the Firebase SDKs.

[https://news.ycombinator.com/item?id=8423055](https://news.ycombinator.com/item?id=8423055)

------
kevin
Oh and if you guys build anything cool with the API, please let us know at
api@ycombinator.com.

------
ossama
Here's a simple example that displays the top story and votes using the
Firebase JS SDK (and updates in realtime):
[http://jsfiddle.net/firebase/cm8ne9nh/](http://jsfiddle.net/firebase/cm8ne9nh/)

~~~
ossama
and one showing all top stories updating in realtime. Obviously, in JS, but
the other Firebase SDKs are similar:
[http://jsfiddle.net/firebase/96voj1xh/](http://jsfiddle.net/firebase/96voj1xh/)

------
benhamner
Suggestions for improving the API, to make it more valuable for data mining
and analytics. This assumes more historical data is available.

1\. Provide a way to bulk download the data (that's a click, instead of
scraping the API)

2\. Add a field for the maximum position a story reached on the front page

3\. Add the numerical score for the comment (at least on comments that are N
days old, which won't interfere with the reason to hide the scores on the main
comments)

Some other changes that would be awesome (but are less realistic) include:

4\. A historical event log of votes (even better would be relating those votes
back to users, but I imagine that's not going to happen for privacy reasons.
An intermediate possibility would be a vote log connected back to anonymized
user ids, assuming the anonymous id -> real user id mapping is difficult)

5\. A historical event log of display position changes for stories & comments

6\. An event log of pageviews with as much metadata as possible to release
without infringing on privacy

------
elwell
> the reason we released an API is so that we can start modernizing the markup
> on Hacker News

This is bigger news. No more tables!

------
so_
I think quality of the v0 API is not so good for the following reasons.

1\. Major functions are dropped (best, new, job, ask,...)

2\. Useless response schema of the top stories API (should learn from
Netflix's internal/public API design)

3\. A short transition period

It's very hard to provide same responses to apps.

------
ChuckMcM
Oh this will be cool, and I really look forward to being able to read HN on a
phone or tablet in the future without all the zooming and scrolling!

It will be interesting to see if it has an impact on site traffic, how much of
that traffic is scrapers today?

------
wvl
Would it be possible to cache the number of comments a story has? Or am I
wrong in my understanding that the only way to find the number of comments a
story has is to walk the tree of child items and maintain my own count?

~~~
calebm
+1

------
davexunit
Glad to see that HN will still be written in a lisp after these major
upgrades.

~~~
dang
I regard that as part of the site's DNA. It permeates so many things about its
design, both internal and external.

------
quesera
Great news! Will require a fair bit of retooling, but, ya know -- omelets,
eggs, etc!

I see the topstories query to replace scraping /news: [https://hacker-
news.firebaseio.com/v0/topstories.json?print=...](https://hacker-
news.firebaseio.com/v0/topstories.json?print=pretty)

And there must be a query to get the newest items instead of the current
/newest.

Are there also new equivalents for /active, /best, /classic, /show, and
/shownew?

I'll be happy to replace dodgy XPATH parsing with a proper API. Hope we won't
lose these other views though!

------
dimillian
So, does it mean I can get the top stories, and then get a top story item with
all the comment expanded ? I mean, at first it look like it just send me the
id and I need to fetch the detail for each of them. Again, this is just
looking at the REST API, not the iOS SDK for example.

I'll need to "convert" SwiftHN
([https://github.com/Dimillian/SwiftHN](https://github.com/Dimillian/SwiftHN))
either to this new API or adapt my scrapping engine to the new site layout.

~~~
matznerd
Please do convert it, I am working on a project with some of your code as a
base!

~~~
dimillian
You're using the app, or the scrapping engine (Hacker Swifter)?

The nice thing is that HackerSwifter public API is already in a quite finished
state for the available functions, even if I switch to the API, the method
calls will be the same.

~~~
matznerd
I am actually using the app because I am doing a mashup of HackerNews and
another piece of software. If I decide to end up changing the UI a lot (which
looks like it is trending towards), I was thinking of switching to the
scraping engine. It's awesome that I won't have to change a lot if you do
switch to the API. I'm following on GitHub, keep up the great work!

------
awwstn
This is so awesome – albeit a bit overdue ;)

I'm going to start diving into the API to build a simple, powerful "Google
Alerts for HN" app on Assembly, and I'd love help from anyone who's
interested: [https://assembly.com/hn-monitor](https://assembly.com/hn-monitor)

There are some products like this out there, but they had to rely on scrapers
and the HNSearch API, so I've always found them to be spotty. I think we can
make something better.

------
Lekoaf
Chiming in here and presenting my try at a Hacker News website using Angular
JS and Bootstrap.
[https://github.com/lekoaf/HackerNews](https://github.com/lekoaf/HackerNews)

My only problem with the API is that you need to do an awful lot of Ajax calls
just to get something out of it. The topstories endpoint just gives you an
array of IDs and then you need to do one Ajax call for each ID to get the
story.

Oh well, the site is a work in progress. Not done yet.

------
stevenspasbo
I spent about 20 minutes throwing this together, so it's VERY rough, but maybe
it will turn into something useful. I'm primarily a java programmer but I've
been wanting to teach myself more ruby, so here it goes.

[https://github.com/stevenspasbo/Hackernews](https://github.com/stevenspasbo/Hackernews)

EDIT: I should say, I started on this to create word clouds, but if anyone has
any ideas, contribute!

------
tlo
For my twitter bot
[https://twitter.com/hn_bot_top1](https://twitter.com/hn_bot_top1) I use
[http://api.ihackernews.com/](http://api.ihackernews.com/) at the moment. This
works but the site (and the API) is quite often not available. So I'll
probably switch to the new API as soon as I have some spare time...

------
ryanseys
Smart move using Firebase. This instantly gives developers traction with the
API and as a big API client guy, I love making clients but also very happy
we'll have all the tools we need to get started using the API right off the
bat. Considering Firebase doesn't have rate-limiting too, this means that the
things people can build with this API are limitless.

------
hardwaresofton
"Let there be apps".

Anyone with a Firefox phone that wants to work together on a HN client for it?
FFOS has been lacking a good HN app.

------
IgorPartola
I am excited about this because now I can finally build a way to query my own
posts and comments. I often times come across products/services/cool hacks on
here that I vaguely remember, but cannot always locate. Using built-in search
is kludgy, and I'd like to be able to do something more complex. Thanks for
doing this!

~~~
DanBC
It's interesting that you mention the phenomenom of "I remember seeing it (or
maybe even saying it) but I cannot find it". This has happened to me a lot. It
is intensely frustrating to me because it's blocking - I don't get much done
until I re-find whatever it was.

I wonder how many other people have this? And what their techniques are?

~~~
terhechte
I have the same problem, and I think the best solution would be to index my
web browser search history and be able to search through that index. However
that would probably generate a lot of junk. My current solution is to bookmark
anything that's vaguely interesting. And when I remember something, I search
through the bookmarks. I never look at them, they're just for keeping the
stack of stuff to search through.

~~~
raldu
I am using the same approach too. However, most of the old stories I had liked
at HN were not bookmarked, so I had to struggle writing a scaper for them. It
would be very cool to be able to authenticate and grab all my saved stories
and the comments for archiving and fun!

------
bajsejohannes
Awesome!

On the top of my wishlist: Look up HN story by URL.

(From the comments, I learned about a different API that will suffice for now:
[https://hn.algolia.com/api/v1/search?query=google.com&restri...](https://hn.algolia.com/api/v1/search?query=google.com&restrictSearchableAttributes=url)
)

------
thekingshorses
I added upvoting stories and comments feature to my ios/android app few weeks
ago. And thinking about adding commenting.

[http://hn.premii.com/about](http://hn.premii.com/about)

Now I will have to remove that. Please keep current version of the YC site on
a different subdomain.

~~~
Livven
I use your app, it's awesome! Also agree with your request, as I said
elsewhere in this thread...

By the way, animations (the slide between list and comments) have been jerky
since a recent update. I'm using the Android app on an HTC One M7 with the
latest stock ROM (4.4.3 and Sense 6.0). A few other people have mentioned this
in Play Store reviews. Is this a known issue?

------
bonzoq
If someone would be interested in contributing to an OSS project to build an
iOS HN client, please have a look at
[https://github.com/bonzoq/hniosreader](https://github.com/bonzoq/hniosreader).

~~~
LazyGhost
OSS under what license? I don't see anything in a quick look over the repo.

~~~
bonzoq
MIT. Added it to readme now. Thanks.

------
wslh
I would love a personal API too where I can download my own data (i.e: saved
articles).

------
BigChiefSmokem
Does anyone have a good solution on how to get around making a separate
request for each Item? Is there somewhere we can pass an array of Item Ids? Is
this planned for the future?

Thanks HN!

EDIT: I just read the post about using the Firebase SDK to do this
efficiently.

------
markrickert
Nice! I'll be changing my Mac system tray app HackerBar over to use this
instead of some Objective-C scraping magic that it uses now:
[http://hackerbarapp.com/](http://hackerbarapp.com/)

~~~
Igglyboo
Wow I've never heard of this but that is a really cool app, I'll definitely
check it out. Do you know of anything similar for reddit?

~~~
markrickert
There's way too many reddit things already out there... I was trying to tap a
market that doesn't have lots of enthusiastic software developers.

oh.... wait...

------
andreash
Give me a HN trend chart for the words "pivot" and "mvp" please:)

------
fosk
If you guys are already using Mashape, it has been released as a community API
at [https://www.mashape.com/community/hacker-
news](https://www.mashape.com/community/hacker-news)

------
calebm
Why could you not have released it 1 day sooner? :) I just wrote a scraper:
[https://github.com/calebmadrigal/hn-
tracker](https://github.com/calebmadrigal/hn-tracker)

------
calebm
This story via the API: [https://hacker-
news.firebaseio.com/v0/item/8422599.json?prin...](https://hacker-
news.firebaseio.com/v0/item/8422599.json?print=pretty)

------
BenSS
Great news, I've avoided touching my iPad reader because of the whole scraping
issue to get at some of the data. Now I can justify updating for iOS8 + the
new API. (just hope it gets approved in time)

------
deutronium
I'm just wondering if you'd consider releasing a regular data dump, possibly
via bittorrent.

As I'd like to do some analysis, but don't really want to thrash the API,
downloading all data.

------
aetherspawn
I managed to take a picture when this got 1337 upvotes.

[https://hostr.co/file/cHCoBCgwsjEc/2.PNG](https://hostr.co/file/cHCoBCgwsjEc/2.PNG)

------
burntcookie90
Very cool, I'm glad I prepped my android app for this :D

One question: why the choice of returning everything as an ID, on mobile, this
will require a lot of very small network requests.

~~~
sararob
If you use the Firebase SDKs the request is sent over a reused full duplex
socket rather than creating new/individual connections.

~~~
burntcookie90
Yeah, unfortunately, I'm trying to not use any closed source stuff in the app.
Going to write some service layer stuff for it

------
astevens
I whipped up a quick test in Ruby

[https://gist.github.com/astevens/627756eafbc6ca07e3dd](https://gist.github.com/astevens/627756eafbc6ca07e3dd)

------
xtrumanx
Perhaps keep the old site still running on a different URL so scrapers who
can't get their act together in 3 weeks can just change the URL they're
scraping from.

------
induscreep
Finally, an AlienBlue caliber app for HN is now possible. Soon.

~~~
twodayslate
news:yc is the best I have found. Just wish it has collapsable comments.

------
lnanek2
Disappointing they are still going to use Arc even with the update to modern
HTML. I despise how with Arc if I leave the page open during the day and try
to click a link later the link has expired.

They should really update to a modern web framework at the same time. Big
modern frameworks like Rails are making ridiculously awesome improvements like
replacing page loads with XHR (quicker loading since JS/styles/etc is all
loaded already, no screen flash, etc.) in a progressive enhancements manner.

So Arc can't even generate fully functional links, let alone keep up with
modern web advancements.

~~~
danielhughes
What is it about Arc that causes the unexpired link issue? I know very little
about Arc. But I understand it to be a language whereas the issue you describe
appears to be the function of some implementation decision. Can you please
elaborate?

~~~
dang
(I'll take a crack at explaining this momentarily...)

Good grief, that took a long time. Here you go:
[http://pastebin.com/bSW5dfRQ](http://pastebin.com/bSW5dfRQ) [1]. I'd better
stop neglecting my duties now!

Edit: One thing I forgot to put in there: one reason the closure technique is
powerful is that you're leveraging the programming language and runtime to do
most of the book-keeping for you. Whatever data is handy, you just reference.
The system will remember all the references. That's why using things like
query strings and hidden form fields is more complicated: you have to handle
all those details yourself (not to mention serialize and deserialize them if
you're passing through any other format than what your program keeps in
memory). That is tedious, and when your app has many kinds of request, the
complexity quickly piles up.

Of course there are other abstractions you can build over this, but closures
are an elegant one—especially in cases where programming simplicity is more
important than scalability, which is most cases.

Edit 2: A few people thought this should be its own post, so I made
[https://news.ycombinator.com/item?id=8425011](https://news.ycombinator.com/item?id=8425011).

[1] Originally [http://pastebin.com/dETyYtpX](http://pastebin.com/dETyYtpX),
but I added the above bit etc.

------
r0nin
I started a small package for golang here:
[https://github.com/cryptix/gohn](https://github.com/cryptix/gohn)

------
tonysuper
HN is getting a mobile-optimized site?

Best news I've heard all day!

------
chrismccoy
[http://jsfiddle.net/chrismccoy/cuz7ugL8/](http://jsfiddle.net/chrismccoy/cuz7ugL8/)

quick fiddle to dump the json

------
jingwen
Wrote a simple ruby wrapper for this.

[https://github.com/infinitus/hnrb](https://github.com/infinitus/hnrb)

------
pbnjay
Great news! Can we add endpoints for "Show HN" and "Ask HN" too though? I
don't see an easy way to get them.

------
kasperset
This is a very good news. Made my day. Thank you.

------
tsenkov
Are there API methods for New, Show, Ask, Jobs and Comments page items?

Or should I be picking them up from "/v0/topstories"?

------
EGreg
This is excellent news, and allows me to make an app for YCombinator using our
social app platform!

This should really energize the HN community.

------
hrjet
On a related tangent, since you mention modernizing the markup, can we please
have _markdown_ support in comments?

------
stevekemp
Until now I've been using the RSS feed to write a Maildir archive of stories.

It will be nice to write that directly via the API :)

------
x201s
Glad to see there is an API. What about the ability to delete accounts? Why do
I have to beg to delete an account?

------
izietto
Yes!!! Now I can make my HN app! I don't like the ones that are on the market
now (on Android platform)

------
oftenwrong
I have been working on a toy project that uses the Algolia API. Maybe I'll
switch it to this now.

------
andrethegiant
Any plan on having another field on the items route that can be queried
besides id, e.g. url?

------
bibonix
What about search? Would be great to have an ability to search by URL or title
fragments

------
bch
Does this encoding: &#x27;

have a name ? I can't actually find anything useful on the 'net.

~~~
tlb
It's part of HTML.

[https://en.wikipedia.org/wiki/Character_encodings_in_HTML](https://en.wikipedia.org/wiki/Character_encodings_in_HTML)

~~~
bch
Thanks. I'm trying to get cURL to decode, but it doesn't seem to natively
handle. Now I'm digging into .../escape.c

I feel like I must be missing something... :/

~~~
tlb
You can't simply decode each character without losing information. For
example, &#x3c; means a literal < character to be shown on the page, as
opposed to a < in the stream which starts an HTML tag.

If you're just planning on displaying the text in a browser, no decoding is
needed. If you want to parse the text to do some sort of textual analysis, an
HTML parser library might be best.

~~~
bch
I understand what you're talking about re: &#x3c; and '<' \-- the json -looks-
page (terminal in my case) displayable, barring the &#xhhhh; encoding. cURL
has facilities for decoding %20 (for example), but not what we're getting back
w/ this json.

You've given me an idea though, so back to vi for me.

Thx.

~~~
mh-
not sure if you figured something out already, but just saw your comment and
remembered that this exists in PHP:

[http://us1.php.net/manual/en/function.get-html-
translation-t...](http://us1.php.net/manual/en/function.get-html-translation-
table.php)

absent another source, you could dump it out for your usage elsewhere.

    
    
      % php -r 'print_r(get_html_translation_table(HTML_ENTITIES, ENT_QUOTES|ENT_HTML5));'
    

or

    
    
      % php -r 'print json_encode(get_html_translation_table(HTML_ENTITIES, ENT_QUOTES|ENT_HTML5));' | jq .
    

edit: just found [http://dev.w3.org/html5/html-
author/charref](http://dev.w3.org/html5/html-author/charref) (but might be
harder to parse..)

------
jtwebman
Nice time to makes some awesome data charts like most common subjects, etc...

------
zx33
Is there a way we can get all the submissions for a month or a date range?

------
me_bx
I hope they'll add an article search (by keyword) feature soon.

~~~
eevilspock
The API will allow others to come up with competing designs for HN search.

~~~
me_bx
Yes, but someone willing to get the HN posts about a certain topic from the
API will need to keep polling the API to get all the new submissions, and
filter out the content irrelevant to her purpose. Search directly from the API
would be so much more convenient.

------
bambax
Can we query new stories?

~~~
bambax
I meant the list of newest stories as on this page
[https://news.ycombinator.com/newest](https://news.ycombinator.com/newest)

Any idea?

------
na85
Please please please no javascript in the next HN iteration.

~~~
hmsimha
Do you want to have to reload the page every time you upvote someone?

------
oldboyFX
Ummm I have a question.

Why is so much preparation necessary to redesign like 3 simple templates.
Shouldn't it take like ~10 hours for one person to do this? I'm talking about
the front end.

Why wasn't hacker news optimized for mobile a long time ago?

:|

------
sinwave
It would be fun to whoop up an LDA analysis of this data.

------
calebm
How can I find the current rank of a story via the API?

~~~
calebm
Nvm, I see: [https://hacker-
news.firebaseio.com/v0/topstories.json?print=...](https://hacker-
news.firebaseio.com/v0/topstories.json?print=pretty)

------
yawniek
is there a simple way to stream all the updates or at least the new
submissions/deletes by just subscribing to a Websocket?

------
kolev
At last! A hacker-friendly Hacker News! Yay!

------
dabit3
I love firebase!

------
dpcan
As soon as someone uses this API to create a replica of the site where
everything else is the same, but the design is responsive, I want to know
about it :)

------
jgill
A very welcome addition.

------
danieldisu
Can't wait to see all the apps that will be built with this, now that you
don't have to scrape the horrible HN HTML!

------
angersock
Why do you have all the times in Unix time instead of ISO-8601 UTC?

EDIT: Also, the "about" value in the users api appears to be truncated.
Compare...

[https://hacker-news.firebaseio.com/v0/user/angersock#](https://hacker-
news.firebaseio.com/v0/user/angersock#)

...with...

[https://news.ycombinator.com/user?id=angersock](https://news.ycombinator.com/user?id=angersock)

EDIT2: Note that the JSON is correct, but the preview in the firebase API
seems to be broken.

EDIT3: No issue tracker on the Github? Laaaame.

~~~
Someone1234
UNIX time is actually much easier to parse and more accurate. Almost all
platforms (even Windows[0]) have a way to convert a UNIX time into a culture
specific local time.

What you're asking for is a string you have to parse. That's a lot more work
and there is a lot more that can go wrong.

[0][https://stackoverflow.com/questions/249760/how-to-convert-
un...](https://stackoverflow.com/questions/249760/how-to-convert-unix-
timestamp-to-datetime-and-vice-versa)

~~~
angersock
Um, how about, no?

    
    
      var time = new Date("<ISO-8601 string>") // in Javascript
    
      time = DateTime.iso8601("<ISO-8601 string>") # in Ruby
    

Aaaand it's human readable! Aaaaand it works before the Unix epoch!

EDIT:

Changed DateTime.parse to DateTime.iso8601, to be even more retentive.

Look, parent claimed that parsing ISO strings is hard (it's not, _especially
if you 're consuming a web api on a modern web language_) and that it was more
readable (which is so clearly wrong I have no words).

As for being more accurate, again no. The range is worse (lol wraparound if
you're using a 32-bit int), there isn't explicit support for fractional
seconds, doesn't map onto UTC cleanly, doesn't handle leap seconds, and so on.

It's only "easier" if you don't actually care about a human-readable timestamp
that is robust and if you desire to do date parsing yourself instead of using
any of the well-established libraries out there. Ugh.

~~~
teach
That's two programming languages. Not all languages have a nice ISO-8601
parser built-in.

Human-readable is a slight advantage, agreed.

~~~
angersock
It's not a _slight advantage_ if you've ever had to parse through logfiles by
hand or debug APIs with curl--it's a great deal more than that.

Why not, let's have some more examples:

In PHP (with a bug, because PHP is stupid and doesn't handle decimal
fractions):

    
    
      $dateTime = DateTime::createFromFormat( DateTime::ISO8601, '2009-04-16T12:07:23Z');
    

In C#:

    
    
      DateTime time = DateTime.Parse("2010-08-20T15:00:00Z",  null, DateTimeStyles.RoundtripKind);
    

In Python (stupid that it isn't in the standard library, see
[https://wiki.python.org/moin/WorkingWithTime](https://wiki.python.org/moin/WorkingWithTime)):

    
    
      import dateutil.parser
      dateutil.parser.parse('2008-08-09T18:39:22Z')
    

In Perl: my $dt = DateTime::Format::ISO8601->parse_datetime(
"2008-08-09T18:39:22Z" );

~

Interchange formats, say JSON blobs over a wire, should very clearly express
what's in them, perhaps by using a very well-known standard which is human-
readable, whenever possible. The fact that some languages haven't yet realized
that this is an important-enough feature to put in their standard libraries
compared with whatever esoteric academic shit they think is necessary (C++1x,
for example!) is not the format's problem.

Hint: if you're consuming a web API, you are probably using one of the
languages I gave examples for, or a very close relative. Just because your
Haskell-on-M68k package doesn't know how to 8601 doesn't mean that using a
nondescript number is a good idea.

Why, in the year of our Lord 2014, _is this even a fucking question_?

EDIT:

I'm sorry to be so mean in my language about this, but I've had to fight a lot
of raging stupid with regards to storing timestamp data. I do not wish to see
anyone else suffer unduly.

~~~
Someone1234
> It's not a slight advantage if you've ever had to parse through logfiles by
> hand or debug APIs with curl--it's a great deal more than that.

Funny you should say that, UNIX time works great for log files. Since now even
a dumb tool can search for values in the range, 1388491199-1420027199 to see
all 2013-2014 lines.

Your way requires specific support for the date format, dealing with "2014"
false positives, or searching through all 12 months individually (2013-01,
2013-02, 2013-03, etc).

You keep drumming on about library support, which is important for a format
which isn't universally supported. Fortunately for UNIX time there's no value
in listing them off one by one, you can just assume it is all of them...

UNIX timestamps are also listed in several international standards. Including
POSIX.

~~~
angersock
ISO-8601 is designed to be sortable lexicographically, so the same benefit
stays there when using dumb tools.

Again, no benefit to not using it, other than you are scared of strings. If
you are handling JSON from a web API, _you are already dealing in strings_.

------
zuck9
What's the Arc rendering engine? Google didn't help

~~~
Goopplesoft
[http://www.paulgraham.com/arc.html](http://www.paulgraham.com/arc.html)

------
yefim
The GitHub repo is inaccessible because it is private.

~~~
kogir
Oops. Thanks for letting me know!

~~~
tokenizerrr
The issue tracker on
[https://github.com/HackerNews/API](https://github.com/HackerNews/API) appears
to be disabled, though the summary says "Documentation, Samples, and Issue
Tracking for the Official HN API."

~~~
kogir
We decided email would be better until we got a feel for things. Updated.

Examples are coming soon, but we didn't want to make people wait.

~~~
wvl
Please consider opening up the issue tracker. Other people might benefit from
seeing these issues, such as the issue about the comment count that I just
raised as a comment here.

------
projectileboy
Listening to programmers complaining about the markup used to implement HN is
a perfect example of tech folks not focusing on the users.

~~~
NewHermes
Could you elaborate? I assume you are criticizing HN or the complaining
programmers but I can't tell which.

------
Achshar
I always thought HN was more of a one man job. I didn't realize there is an
entire _team_ working on it (full time?). Honestly there doesn't seem to be
much to work on here. It's a simple site that rarely changes (at least on the
front end).

~~~
tdicola
'Simple site' that gets probably tens of thousand if not hundreds of throusand
unique visitors a day.

~~~
Igglyboo
HN uses 12 lines of javascript total, it's pretty damn simple.

~~~
tdicola
And what about the backend? Last I heard it's all stored as files on a handful
of machines, so it's not like it's some off the shelf database you can just
keep throwing machines at to scale. Also it's written in lisp originally by
one person, so have fun figuring out how it all works and ramping up to it
quickly.

There's also certainly a big support overhead dealing with the moderation of
stories and comments.

I'm not saying it should take hundreds of dedicated engineers to run the site,
but I think it's silly to look at it and think you could run it in your spare
time because it's just a list of stories.

