

Show HN: Hacker News Comments - nathancahill
http://hncomments.nathancahill.com/

======
jere
>Although many websites and blogs have a comment system, the quality of the
discussion on Hacker News tends to be much better.

Probably because your random internet stranger has never heard of HN.

It's a shame, for instance, that I was ever allowed to find this place.

~~~
seagreen
If that ever changes and HN is overrun (insert dramatic and sad Hans Zimmer
score) what about starting a new HN just for programmers? I wouldn't be able
to get in at the moment, but someday I'd hope to be able to.[1]

[1] I'm assuming you'd have some kind of rotating or individualized Project
Euler style problems that you have to solve to get in, and that they would be
pretty tough. It occurs to me that you could also have temporary student
accounts for people who are just getting into the software world as well
though.

~~~
mapleoin
Passing a programming test is no guarantee for having anything interesting to
contribute to an online community, just look at your company's internal IT
mailing lists.

~~~
seagreen
Totally! That's why I'm not suggesting anyone do it _now_. There's a place for
kludgy solutions, but not until you've tried everything else.

------
quesera
Scraping comments on HN hits the robots.txt wall rather quickly.

At a maximum 2 scrapes per minute, your cache will be 60 minutes old when
you're watching 120 (single-page) comment threads. That might be tolerable.
Now try 1200 or 12,000 comment threads.

You can do smart cache refreshes, but you will miss bursty activity. You can
ignore articles older than X days, but you'll miss the "dupe, comments here:"
resurgences.

HN gets about 150 posts per day that hit the top 60, at least briefly, as a
point of reference.

This is a neat thing, but if it gets popular you will have a problem. On the
plus side, it's self-limiting. As the usage gets higher, the experience will
degrade and usage will decline. No intervention required by you. :)

I've wished for a policy-compliant way to cache comments in a timely manner,
but there's just no way to stay current under the existing rules.

~~~
rcavezza
What if you're scraping with ajax? Wouldn't each individual user's IP take the
hit and not the domain's IP?

~~~
NathanKP
You can't scrape with AJAX because of cross domain security restrictions.

One potential solution to obey robots.txt might be to spawn multiple small EC2
instances with different IP's and have them coordinate with each other to
share the crawling without individually running over the limits. (This is also
useful for scraping from sites that have rate limits)

~~~
typpo
robots.txt doesn't enforce itself so there is no IP limitation; this is still
a violation and no better than simply lowering the delay on a single scraper.

------
sgdesign
I'd definitely use this on my blog (<http://sachagreif.com>) if it was
officially supported (or at least tolerated) by HN somehow.

And the fact that people can only comment if they have an HN account makes it
even better!

------
jacobr
> Although many websites and blogs have a comment system, the quality of the
> discussion on Hacker News tends to be much better.

Wouldn't this risk changing that?

~~~
danmaz74
Probably not, unless your blog has an organic audience that is bigger (or at
least comparable) than that of HN homepage...

~~~
nathancahill
Blogs this big are unlikely to implement a third-party (and unsupported)
discussion feed.

------
unimpressive
In my preferred model of Internet content, stuff like this would be available
for every element without scraping.

Good job.

~~~
jerf
The word for that is <http://en.wikipedia.org/wiki/Transclusion> , and it's
interesting that it was a core component of Xanadu (see the Wikipedia link
there).

~~~
unimpressive
Yeah. I know.

We also need a micropayment processor that can do sub-penny transactions.

EDIT: Actually, I would be fine with penny transactions considering that the
current state of things with say, paypal; is 30+ cents per transaction.

------
Davertron
How hard would it be to have your script look for the submission on HN by url
instead of having to manually put in the id? Maybe it could use the one with
the highest score if there's more than one?

~~~
nathancahill
Yes, that's the plan! A new API endpoint will use the referring page to
capture the URL, then link it with the appropriate discussion on HN.

------
nickwoodhams
Great contribution Nathan, looking forward to using this to embed comments
after my next blog post. Can you let me know how the comment system works? Are
you scraping HN?

~~~
nathancahill
Thanks! I'm respectfully scraping. I've rate limited all calls to YC's servers
and am caching all comments on my side.

~~~
nickwoodhams
Seems respectable. Out of curiosity, why did none of the existing 3rd party
API's suit your purpose?

~~~
redslazer
Probably the same reason no one should you use this to power their comment
section there is no revenue, and no real reason to keep it up. Things like
this come and go.

------
sktrdie
This would make more sense if it was entirely client-side code towards an
official JSONP endpoint, such as the official hacker news API.

~~~
mattacular
Not saying this is necessarily true for Hacker News, but unfortunately a lot
of sites don't have the time/resources to develop and then maintain a public
API for the majority of their content. And mostly see an RSS or XML feed of
their news/articles/stories as good enough.

------
sp332
Do you think people are going to wait for someone to submit each post to HN? I
think this will encourage writers to submit every single story they write to
HN, just to get a comments page.

~~~
samjc
Even if someone put every single story they write on HN, that's not a
guarantee that it will get comments, since it might not get up-voted.

I think if I were to implement this on my own page, I would probably have two
tabs that expand if clicked. Something like this:

Show [your site's name] comments | Show HN comments

And then for the HN comment tab, you would use the OP's awesome submission.
Thanks, btw!

------
joshaidan
Here's a question, is it possible to put a Hacker News upvote button on your
blog post? Or is that against the HN rules, or something that's frowned upon
in the community?

~~~
sbashyal
I created hnlike.com about a year ago that helps you to setup an upvote button
for your blog. I have disabled the upvote feature as the community seemed
worried. See the HN discussion on my story for details.

------
bherms
I hope this doesn't further impact the quality of discussion on hacker news.
It seems that every day I notice more and more people posting completely non-
contributing crap in the comments that you'd expect from Reddit or something
(one-liners, stupid jokes, etc). I go to Reddit for that, I come here for
insight (which, granted, you can find a lot on Reddit also, but HN is supposed
to be a bastion of intellectual thought and discussion).

------
progrock
This idea is kind of alluring. But you can't really include discussions on
your page that might be happening all over the Internet. Reddit etc.

This is where the browser should really be smart enough to do a search engine
lookup, and propose smart links alongside the page you're reading. Or even
summarise and merge discussions found elsewhere.

~~~
personlurking
I'm going off topic a little but...

An element I really like from random blog posts I've seen off of HN is using
the sidebars for additional, short commentary/details. Your comment reminded
me of that, but sadly this might not work well if an article were all chopped
up with tons of commentary (I'm thinking of used books where 'important' parts
are highlighted and comments put beside them by a random reader, which I
dislike). Well, then again, perhaps for people who like to dive into subjects
and really digest an article, there could be a straight text version and a
feature to see it marked up w/ smart commentary.

This also reminds me of a small 'movement' a year or two ago for citing in-
article links at the bottom instead of linking to them throughout the article
itself.

~~~
progrock
Does it really matter where the links are listed - in the article or at the
end?

Opera has a side panel, one of which is the Links panel. That pulls links out
of the page and lists them alongside. But this alone isn't that much use. The
browser however could do something more interesting with them.

I think a lot of pages seem dreadfully wasteful by placing in the same old
accessory content on their pages. Buttonitis etc. I don't think this
necessarily should live 'inside' the 'page' at all. The content should remain
dumb. The inbound links, and chat around the content augments it. And this
could be summised somehow by browser tools / other services.

I understand the author's want for embedding HN comments on a page. There's a
simple and elegant solution to the problem. Simply open up another browser
window/tab and point it to the relevant HN comment page. To do so, the author
could include a simple link in the page, leave it to the browser to infer, or
just let the user discover it for themselves (pointers do help though.)

Why the obsession of stuffing our pages with content?

It also feels a little snooty doing something like this (embedding HN comments
on your page that is.)

------
irrationalfab
Why isn't this open source?

~~~
nickwoodhams
Why would you want open source for this? Do you want to spend the time to
deploy your own instance just to have the exact same result?

~~~
muellerwolfram
yes i would. to implement the whole thing, instead of only this little jquery
snippet shouldn't take much longer. but then i'm not dependent on someone
elses server being up/fast.

~~~
gcmartinelli
If everyone could implement their own scrapper they would be able to follow
robots.txt guidelines/limits, but HN servers might take a big blow it it gets
popular.

------
mootothemax
Wow, I really like this, great work!

I've previously thought that comment threads on blogs tend to die down fairly
quickly, whereas discussions on HN tend to last a bit longer, so this is
doubly cool.

I guess the only downside is the near-impossibility of letting people comment
directly from the page without having to log in again.

~~~
nathancahill
Not a downside. I'm not allowing posting comments or upvotes through this
system to maintain the integrity of the discussion on HN.

------
mattjball
Just a heads up, I tried to test out your inline "Give it a try below!" but
it's not working right now. In the browser console: "GET
<http://hncomments.nathancahill.com/comments/4285019/html/> 500 (Internal
Server Error)"

~~~
nathancahill
Thanks, there were a couple hiccups handling the large amount of traffic.
Should be in working order now.

------
sirbrad
Nice work, this is pretty cool! I've been working on a similar approach but
using Github (mainly because I host my blog there using Jekyll). I'll try and
get it released tonight :)

------
danmaz74
Very, very interesting! So you already calculated that the cost of this will
be insignificant for you even with millions of pageviews? Blogs have a long
tail of visits... ;)

------
biot
How do people opt-in to allow their comments to be shared on sites other than
HN? Or is this strictly opt-out and, if so, what is the opt-out mechanism?

------
beshrkayali
This is very interesting! Great work.

~~~
swah
This comment could be an upvote to the original story :)

~~~
nilved
Couldn't your comment be a downvote? ;)

~~~
swah
The downvote arrows don't appear sometimes! I don't know how the software
decides....

------
danjessen
looks good :)

~~~
swah
Upvoting gives the same result as "nice job!" comments like yours :)

