

Ask PG: Would you accept donations towards buying a monster server for HN? - jacquesm

HN is pretty slow again, and has been for some time now, connections timing out the rule rather than the exception when posting an article or commenting.<p>I can see why it's hard to scale HN using multiple servers, what with ARC not being the most cluster friendly platform, but there is a different solution, simply getting a really badass server.<p>I'm sure that you're not wanting for cash, but since we all use this resource and we all profit from it in various ways would you accept donations that would go to buying a server large enough to see HN through the next phase of growth?
======
pg
We have the fastest server we could conveniently get about a year ago. But I
just asked Rtm if we could get something better.

Unfortunately processors don't seem to get faster at the rate they used to.
Now they just have more cores, and I believe MzScheme can only use one core.

I think the solution is to make the software faster. There's probably some
small hack we could make somewhere in the guts of the system that would make
it 10x faster. I'm thinking of having a make-HN-faster contest, but first I
have to release a more current version of the code.

~~~
jacquesm
Would it be possible to layer arc on top of clojure or is that too far distant
from each other?

~~~
alnayyir
That's a _bit_ much to resort to when AFAIK caching hasn't been pursued
aggressively as it could be.

~~~
jacquesm
Typically if you start the work when you need it you're actually too late, if
this is a direction that is viable the work could get underway long before
there was an immediate need and it would probably be done that much better for
that reason.

Again, I have no idea if this is even feasible, the differences between the
various dialects of lisp seem to me (as an outsider) to be considerable.

~~~
pmjordan
To my knowledge, Arc, like Scheme, uses mutable conses as its main data
structure; Clojure doesn't use conses in the sense that the cdr-equivalent
element of a list cell can't be just anything. Clojure's data structures are
also immutable. You'd have to do a lot to re-engineer the basic data
structures just for that; I doubt Arc would be the same language at the end of
it.

I'm sure Mz/Arc can be made to run workers in other threads or processes in
some way, even if it's not trivial. I also get the impression the caching is
implemented in Arc code rather than as a HTTP proxy. The latter could run
across multiple cores or even servers. Finally, it may be possible to decouple
the user-specific data from the generated pages themselves and cache that
information on the HTTP level. I admittedly haven't given it a lot of thought,
and I also haven't read through the public HN source yet.

------
roschdal
PG: Have you considered setting up Varnish on HN? I think this would solve
most of the current issues with the performance.

<http://www.varnish-cache.org/>

~~~
sigil
Varnish is indeed crazy fast...for things you can cache. The complaints I've
been seeing center on slowness and timeouts for posting stuff, in which case
Varnish (like the recent pagination change) might not help.

I think PG opening up the latest code and giving people a chance to actually
diagnose the causes of the slowness is a great a idea. Anything else --
premature optimization.

------
coderdude
HN is becoming crazy mainstream. Might as well start figuring out how to use
multiple servers now before we're really in a bind.

