

Scaling Reddit from 1 Million to 1 Billion – Pitfalls and Lessons [video] - veszig
http://www.infoq.com/presentations/scaling-reddit
Jeremy Edberg shares some of the lessons learned scaling Reddit, advising on pitfalls to avoid.
======
itry
What a horrible website. Made an account to get the mp3. Just to discover that
the mp3 download link doesnt work even then. Thank god I used a throwaway
email to sign up. Otherwise I would be even more pissed.

~~~
paraxisi
[http://bugmenot.com/view/infoq.com](http://bugmenot.com/view/infoq.com)

Bugmenot is great for sites like this.

~~~
wjk
[http://10minutemail.com/](http://10minutemail.com/)

is what i use, much simpler.

~~~
amitparikh
Another similar option is Mailinator:
[http://mailinator.com/](http://mailinator.com/)

~~~
Houshalter
Some websites block emails from mailinator which is pretty annoying.

------
russelluresti
Disappointment in the HN community has reached a new high today.

So, instead of discussing the topics in the video, the majority of commenters
here are discussing the flaws of the website it's hosted on or debating
whether or not reddit is profitable. Neither of which has ANYTHING to do with
scaling.

I expected better, people. Seriously.

~~~
diminish
The dominance of non-relevance is interesting for HN. Was writing exact the
same thing, and just saw yours.

In this and some other technical topics, people end up discussing their
personal tastes with web site's design, their individual UI frustrations with
some button on the web site, the font, the color, and other random non-
relevant topic; like now the profitability.

~~~
kerkeslager
It's no excuse, but I see a reason for this: most people feel they understand
these irrelevant topics better than they understand the scalability features
of the infrastructure Reddit has built. People talk about their comfort zone.

~~~
alanctgardner2
This is the fundamental rule of bikeshedding, which HN has discussed, decried
and regurgitated since time immemorial.

------
billb2112
"ask me anything about running a profitable social media company".

Except for that reddit is not profitable.

[http://blog.reddit.com/2013/08/reddit-myth-
busters_6.html](http://blog.reddit.com/2013/08/reddit-myth-busters_6.html)

[http://www.reddit.com/r/TheoryOfReddit/comments/1ihwy8/rathe...](http://www.reddit.com/r/TheoryOfReddit/comments/1ihwy8/ratheism_and_rpolitics_removed_from_default/cb50c8c)

~~~
jedberg
Funny story about that. When we were owned by Conde, the accounting was a
little different (they took on some of the charges, like Akamai), and so we
were actually told we were slightly profitable.

When that blog post went up I was as surprised as you to see it wasn't
profitable when I was there.

~~~
danilocampos
But weren't you "deeply involved in the business?"

~~~
jedberg
Yes. What point are you trying to make?

~~~
300bps
I think his point is that he doesn't understand accounting and how complicated
it can be so he thinks that if you were deeply involved in the business you
should know every facet of every debit and credit.

If he's right I guess I wasted about 300 hours studying for Financial
Reporting and Analysis section in the Chartered Financial Analyst (CFA)
program.

~~~
jedberg
Since this isn't reddit, I won't just say: ^^this.

But yes, I believe he was trying to point out that I couldn't have been
involved if I didn't know, but doesn't understand the ins and outs of G&A and
other such accounting practices.

------
dap
I was confused about the memcached problem after moving to the cloud. I
understand why network latency may have gone from submillisecond to
milliseconds, but how could you improve latency by batching requests?
Shouldn't that improve efficiency, not latency, at the possible _expense_ of
latency (since some requests will wait on the client as they get batched)? And
while maybe efficiency is valuable, why would that be an improvement for a
problem they didn't have before?

~~~
jedberg
Sorry that wasn't clear. The latency didn't get better, but what happened is
that instead of having to make a lot of calls to memcache it was just one
(well, just a few), so while that one took longer, the total time was much
less.

~~~
robotmay
I actually did some (simplistic) examples of this in a small presentation to
illustrate the performance improvements of batching memcached requests, if
anyone's interested: [https://speakerdeck.com/robotmay/a-simple-introduction-
to-ef...](https://speakerdeck.com/robotmay/a-simple-introduction-to-effective-
caching-in-ruby-on-rails?slide=11) (slides 11 to 14)

~~~
jedberg
That's a better explanation than mine. :) Thanks for the link.

------
hardwaresofton
So did anyone else take the most important lessons in the video as:

1\. Use AWS

2\. Use Postgres

3\. Use AWS

4\. Use Cassandra

5\. Use python, so later you can write C when shit needs to go super fast

That's what I got.

~~~
jedberg
I find this interesting, this is really great feedback for me actually.

Those are some of the important lessons, although use (postgres|cassandra) are
really too prescriptive. More like "use the right tool or tools for the job".

Also, use consistent key hashing where appropriate is another important less
that I should emphasize more.

And "build for 3" is another important lesson. It makes scaling much easier.

------
gwu78
[http://d1snlc0orfrhj.cloudfront.net/presentations/13-jul-
sca...](http://d1snlc0orfrhj.cloudfront.net/presentations/13-jul-
scalingreddit.mp4)

------
endlessvoid94
Thrilled to finally see this on InfoQ -- already an underappreciated resource
for technical talks. What is it about video + slides that appeals so little to
people?

~~~
ics
Large banner and tiny video frame, site rendering completely garbled for 8
seconds until fully loaded, signup required to view the slides (after
realizing that the video doesn't cut to slides at all). The talks are
interesting, and the interview transcription is nice though there are UX
issues there as well– I need to click to view each response, and despite the
entire question being hyperlinked clicking it actually does nothing. Show all
works thankfully (ah, but the frame breaks my scroll).

I'll probably be back to check out more of the videos, but definitely not
because of the site. If the editing is good, YouTube is just fine, otherwise
SlideShare plus an audio file is just perfect.

Edit:

    
    
        [F] First Byte Time
        [C] Keep-alive Enabled
        [C] Compress Transfer
        [A] Compress Images
        [A] Progressive JPEGs
        [F] Cache static content
        [ ] Effective use of CDN
    

Source:
[http://www.webpagetest.org/result/130816_PE_AYH/](http://www.webpagetest.org/result/130816_PE_AYH/)

~~~
masklinn
> SlideShare plus an audio file is just perfect.

Although slideshare is not as bad as infoq by a long shot, it's not very
useable (or fast) either. I much prefer speakerdeck.

~~~
ics
Speakerdeck is actually what I was thinking of, but the gist is slides + audio
= happy. A more polished UI would do some good for SlideShare, but I don't
have any major issues.

------
vegasje
Does anyone know how the different storage systems are utilized, and why each
system is utilized for that purpose? The presenter mentions using memcached,
Cassandra, and PostgreSQL, and mentions the same type of data when discussing
each (votes, for instance). I would definitely benefit from a more in-depth
understanding of how each system is utilized, and why.

~~~
jedberg
Each tool has a different use case. Votes is a great example.

Memcache has no guarantees about durability, but is very fast, so the vote
data is stored there to make rendering of pages as quick as possible.

Cassandra is durable and fast, and gives fast negative lookups because of its
bloom filter, so it was good for storing a durable copy of the votes for when
the data isn't in memcache.

Postgres is rock solid and relational, so it was a good place to store votes
as a backup for Cassandra (we could regenerate all the data in Cassandra from
Postgres if necessary) and also for doing batch processing, which sometimes
needed the relational capabilities.

~~~
vegasje
That makes a lot of sense. Were the majority of your systems using this
"durability chain" so to speak -- memcached -> Cassandra -> Postgres?
Additionally, in retrospect do you find this type of chain to work well, and
would you use it again (perhaps you already are over at Netflix)?

------
heed
Side question: is vote queuing the reason behind the sometimes large drops in
score on highly active and popular submissions on reddit?

~~~
jedberg
No. That's generally because once something gets popular and jumps to the
front page, it gets a huge boost in visibility, especially from people who
weren't looking at the niche subreddit it comes from.

A lot of those people aren't interested in that content, so it will suddenly
get an influx of downvotes.

~~~
heed
Thanks for the reply.

I'm glad there is an explanation based on user behavior for this phenomenon
because admin level vote tampering is such a tired theory.

------
conradfr
I wonder if the demise of Digg three years ago and the (supposedly) inflow of
new users have been problematic at the time.

~~~
jedberg
There wasn't actually a very large jump in traffic when Digg v4 was launched.
Most of those folks were already reddit users. Traffic bumped a little bit,
but not all that much.

It's important to keep in mind that reddit was already doing twice as much
traffic as Digg before they launched v4.

~~~
conradfr
Thanks for your input, didn't know that.

I, for one, went from a /r/php lurker to real user at that time so I thought
there was dozens of us, dozens !

------
grandpoobah
Reddit still uses Pylons, and I imagine moving to Pyramid would be very
painful. He's not clear on this in the video.

~~~
jedberg
I don't think I made any indications one way or another, but yes, it would
probably be difficult to move.

~~~
grandpoobah
Great presentation man. Do you think Pylons will serve Reddit for years and
years to come? Is there any need for you to switch to Pyramid or another
framework? I made the choice to use Pylons for a recent project, and it just
feels kind of odd using an old framework which is now in "maintenance only
mode", but I truly did not like Pyramid... much less Django.

------
perlpimp
For me link works only in safari. Firefox plays only video and chrome just
does not work.

------
diminish
I am curious how much a Go lang rewrite would make a difference in scaling up.

~~~
jlouis
It depends how much time is spent doing computation and how much time is spent
doing lookups from disk. If the latter is clearly dominating, then a switch to
Go will not help much.

~~~
diminish
thanks for the insight. i did not have much to do with Go. But hear a lot of
positive benchmarks here on HN. Reddit is kind of like an example social web
app, and I read a lot on its architectural changes and scaling efforts. So for
social Web 2.0 apps, which are getting older now, I m curious how much Go
would make a difference. Mainly for Google applications, apparently Go brings
a lot of speed and performance on the same server.

~~~
jacques_chester
It doesn't really matter what the _genre_ of an application is. What matters
is the runtime fundamentals. How much time is spent computing vs waiting for
I/O? Whichever one is slower is the current bottleneck and is what you should
fret about. Go becomes something to consider if the bottleneck is computation
time. It's tangential otherwise.

------
OGC
Reddit's not profitable though..

~~~
chroem
If there is one lesson that HN needs to learn, it's that profitable is not the
same thing as important.

~~~
FooBarWidget
So they are important, but need to live off ramen every day (figuratively).
What good does that do to them?

I don't think grandparent should be downvoted, he raises a good point. Tons
and tons of people use Reddit, but Reddit has a hard time making a living.

~~~
cac04
> So they are important, but need to live off ramen every day (figuratively).
> What good does that do to them?

Maybe they enjoy it? Maybe it makes them happy?

Not everything is about money.

