
New, Bigger HN RSS Feed - pg
http://ycombinator.com/newsnews.html#5oct2011
======
snprbob86
> HN now gets over 120k unique ips on a weekday, and serves over 1.3 million
> page views.

Still off just one server? What are its specs?

How up to date is the news.arc file in the Arc distribution?

I'm considering using the simpler, more functional, single server, all in
memory, filesystem as a database approach in an upcoming project. Would really
love to see more details from PG and others who have had success with that
back-to-basics simple approach.

~~~
pg
One server. An Intel Xeon E5450 with 12 GB RAM.

The currently available news.arc is quite out of date.

Operating out of memory works very well for an application like this, where
most requests are for recent stuff.

~~~
snprbob86
Thanks for the info! I hope you don't mind me asking a few more questions...

> The currently available news.arc is quite out of date.

Any chance of getting an updated version?

> Operating out of memory works very well for an application like this, where
> most requests are for recent stuff.

Do you ever release cached resources? Or does the process simply die with an
out-of-memory error, only to be restarted with empty caches?

If the former, what is the cache invalidation strategy?

If the latter, how often does the process die? What are the implications on
availability and performance, particularly surrounding process initialization
and warming the caches.

~~~
pg
We randomly throw older items out of memory. If they're needed again they'll
get reloaded from disk, but it's unlikely they'll be needed soon because older
stuff is mostly visited by crawlers or random Google traffic.

------
jerrya
I read HN in Google Reader in a tabbed browser, and I think I would prefer a
feed that reversed the order of the two links. So that the first link goes to
the HN comments, and the second link goes to the article itself.

That would allow me to use Google Reader's List view which does not display
the expanded, secondary link, and then go down that list of HN articles, open
the intriguing HN comments in a background tab, and then when I get to the
tab, click on the link in the comments that takes me to the article itself.

~~~
juxta
same goes with their twitter feed. I like seeing the article but I come to HN
for the comments as well (come for the articles - stay for the comments ;) )

~~~
juxta
better yet - I would be cool if they implemented a "reddit bar" for their
twitter feed links so this way you can they have HN Bar which goes on top of
the article

------
pasbesoin
I'm one of those people who might be perceived as "crawling" against the
"More" link.

I use a Firefox extension to pull the next 10 pages, inline. I then convert
all the links to open in new tabs. I'll slowly work my way through the
resulting Frankenpage.

When I first started doing this, I only pulled 5 pages, sometimes waiting a
bit to then pull 5 more/deeper. I was worried about excessive use -- including
triggering whatever blocking you have in place. Eventually, I tried 10 at
once. I've found that 10 is about as deep as I want to go into the recent
history and keeps me fairly current with daily browsing.

I hope this use isn't considered excessive. It's run just once or twice in a
day, and the results are simply for personal consumption.

As I (somewhat vaguely) recall, I began doing this when and because a change
made the "More" links expire more -- as in, rather -- quickly. That was a time
when you were working to keep HN from bogging down and choking on its memory
usage. I wanted to see a few pages into the history without a bunch of overlap
and having to start over from the first page; and pulling the subsequent pages
all at once worked well for me as a solution.

I'll have a look at the revised RSS feed.

~~~
wvl
Perhaps my <http://hckrnews.com> will suit you, since it archives anything
that makes it onto the homepage (without crawling against the 'More' link).

I've resisted putting RSS feeds on hckrnews, because I think RSS feeds are a
poor match for the format. Number of points and number of comments combine
with the link to provide information that you don't get with an RSS feed.
Often the utility of a link is not obvious from the source and title only.

~~~
pasbesoin
Well, I also enjoy finding and promoting good/relevant links that escape
attention "amidst the deluge".

I also take your point that the comment links and point counts are pertinent.
In particular, I will often start off with the comments rather than the linked
page itself. Again, I'll have to look at the revised RSS feed, but as I
recall, when I looked at it in the past, I, too, found it less useful for
these aspects (unless I was viewing one of the third-party feeds that I seem
to recall being created, some time back).

I suppose one could view my use / time on HN as high. OTOH, I pull the pages
(i.e. this "Frankenpage") once and then slowly pay attention to them. I guess
one could contrast that with another user hitting the top page every 15
minutes throughout the day.

(I do also check the new/ and classic/ front pages a few times, often later in
the day. I used to drill further into new/, but I've been trying to limit my
time on HN somewhat. Also, new/ is so full of spam, these days, that it's kind
of discouraging -- although people need to continue to hit it and aggressively
kill off the spam and other crap.)

\----

Completely aside, I'd like a comprehensive list of post links. I used to have
some old favorites saved, but some of those were lost in a theft. I guess it
might not be the healthiest thing for the HN ecosystem including system load,
but I'd enjoy journeying back in time to the early days. Many of those
discussions were incredibly informative and focused.

I know there are and have been a third party archives and/or views into HN,
but either I didn't take the time to learn or the interface itself didn't seem
to offer a simple, time-based listing.

------
antoncohen
With 120k unique IPs a day, it is really surprising how few votes stories get.
Most are under 100 points, only about once a month is there one over 1000
points. Do most of the readers not have accounts? Do people with accounts not
up vote?

~~~
adhipg
There was a statistic on Reddit yesterday that 90% of the people on that site
did not have an account - and, 90% of the people who had an account have never
voted.

Maybe it's a similar scene here.

------
juiceandjuice
PG, how many users are there on hacker news?

~~~
pg
There are 234,301 accounts, but the great majority were created by spammers.

2531 accounts have voted in the last 20 minutes.

~~~
Sukotto
Do you post a stats page someplace? I would find it really interesting to
watch them over time.

------
endlessvoid94
I'd love to know if HN is run on a single machine somewhere. If so, I
definitely need to give Arc a try.

~~~
pg
Not merely a single machine, but a single core.

~~~
endlessvoid94
Yikes. Time to get to work.

------
jcurbo
Is there anything like alterslash.org for HN in an RSS feed? A link to the
article plus the top 5 comments or so would be great to pull in an RSS reader
for on-the-go reading.

------
krishna2
Is there a way we could also get the total points, submitted by, time
submitted and no. of comments ? (essentially the second line). Thanks very
much in advance.

------
ronnier
PG, do you want me to close <http://api.ihackernews.com> ?

~~~
jorde
I hope this doesn't happen. As a frequent Android/iPad user I have grown to
love iHackernews ever since it was published. Unfortunately the lack of
comments in most of the posts is really hurting the service and it would be
nice to see some collaboration between HN and iHackernews.

I'm already using my custom CSS for HN in Chrome but sadly this isn't possible
on mobile and iHackernews fills the need perfectly.

------
ivank
Any chance of getting per-user RSS feeds? I've been missing this functionality
ever since SearchYC went down.

