

Redis at Superfeedr - julien
http://blog.superfeedr.com/datastore/memcache/mysql/performance/redis/redis-at-superfeedr/#

======
patio11
The numbers involved in web scale computing at turns excite, amaze, dazzle,
and frighten me. 3,500 queries a second times four servers. Crikey. I know at
least one system that cost a million dollars which doesn't see 3,500 queries a
week.

And then you look over at Facebook or Google and a system munching through
five digits of writes in a second looks like a quaint little toy.

~~~
julien
So it's actually each of the server that sees this traffic :)

And yes, we're nothing (yet!) compared to what the big guys just. However,
most of the tim they used their very own software to achieve that; I'm just
using out-of-the-box redis, which makes this available to _anyone_.

~~~
patio11
_available to _anyone__

I get a real kid-in-a-candy-store feel from OSS these days, because it really,
really is bringing down barriers for making wonderful things. Facebook solves
some of the hardest engineering challenges the software industry has ever
known. I sell bingo cards to sixty year-old women. We both use the exact same
Memcached. How cool is that? (OK, so they might have about four terabytes more
RAM attached to it than I do, give or take, but that's just an implementation
detail. :) )

------
ErrantX
Ok I revise my earlier opinion [1]. This is definitely one of the best posts
I've read for the last few weeks. Engaging, front line, full of content (i.e.
links to their code and other bits) and pleasantly written. More of the same
please :)

EDIT: this is the first major posting I have noticed that pushes redis as a
big big technnology. has anyone see other people using it? I ask because at
the time Antirez first posted it my feeling was redis has the scope to be a
really big, important tech: and this looks to be a step in that direction.

1\. <http://news.ycombinator.com/item?id=1140761>

~~~
julien
Thanks man, I'm honored. I wish I had more time to write on more of our
infrastructure stuff! I think many people use redis, but don't necessarily
advertise it, which is also one of the reasons of our "coming out" :)

------
lsb
Here's a question: if you have a barebones join table like

    
    
      create table feed_items (item_id integer, feed_id integer, created_at datetime);
      create index idx1 on feed_items (feed_id, datetime, item_id);
    

how much slower is that covering index for your query of

    
    
      select item_id from feed_items where feed_id=42 order by datetime desc limit 10
    

than the mysql/memcached/redis trifecta, and why not upgrade your database
machine to a 68GB EC2 instance? Writing's definitely going to be slower, but
you can batch transactions every 10 seconds or so, and 9x the RAM will be a
big help.

~~~
mjrusso
The first bottleneck on the MySQL set-up is the reading, not the writing. Even
with the indices stored entirely in RAM (as the author pointed out in his blog
post), the MySQL box can't keep up. (Consider the workflow of a software
system that needs to parse hundreds of thousands of feeds on a continual
basis. Each feed you download will have dozens of entries, and most of the
time you'll have already processed the entry, so it can be ignored -- but the
only way to know is to do a look-up.)

Regardless, A 68GB EC2 instance costs ~$1700/month. Four 2GB slices on
Slicehost are priced at $520/month, with the cheaper set-up performing
significantly better.

~~~
boundlessdreamz
Can you explain/give pointers as to how the indices can be stored entirely in
ram ?

~~~
julien
I think the only way to is add RAM when your indices grow.. that's what we did
up until now. There is absolutely now way a data store to keep this kind of
performance as soon as you involve a disk, so, yes, the secret of whatever
datastore is to _always_ keep everything in memory.

------
dschobel
_A few weeks ago, Antirez added 2 awesome features that made Redis an
excellent candidate..._

You guys are incredibly brave for deploying and leveraging any feature to your
infrastructure which is a few weeks old.

~~~
julien
That's part of what makes life fun :)

