

How to Build a Fast News Feed in Redis (and Rails) - waxman
http://blog.waxman.me/how-to-build-a-fast-news-feed-in-redis

======
deepu_256
This is from my experience of trying to build a news feed (we are about to
launch one for our site which gets about a million unique visitors a month).

The approach you suggested is the first approach that i tried. And frankly it
worked great when i am testing the feature on my dev macbook with just a
couple of 1000's of users. But when i started testing it on our server loading
entire legacy data redis ended up taking more than 6GB of RAM for abt half a
million users.

i told myself to just plan for 2 times our current traffic and then later on
think of a better solution(best thing wud be to plan for 5-10 times your
current traffic) and told my CEO that we might have to bump up RAM on the
redis machine to 32 or 64 GB in the future(the easiest way but not an elegant
way to solve the problem is throw more resources). Also mind that you need
atleast 2 such machines to provide failover. i just hoped salvatore(a great
guy BTW) would release a cluster solution for redis and will save me.

While testing the feed we came to know we needed to add more such actions from
the user and every time you add a new type of action for every user you are
looking at increasing the memory of Redis process by substancial amount. you
will run out of memory faster than you have planned. i thought VM is the
solution to all of this as user feeds typically use only latest data and the
entire feed doesn't need to be in the memory. But quite frankly VM has it's
own set of problems -- [http://groups.google.com/group/redis-
db/browse_thread/thread...](http://groups.google.com/group/redis-
db/browse_thread/thread/6534a6fb5c3ee804).

Not telling that your solution isn't good. Just wanted to warn you about some
potential problems from experience of trying out your solution. Finally i am
using cassandra for news feed. Working great as of now. At least much better
than my experience with implementing the same with redis. The major problem
with cassandra as of now is implementing pagination and counting. distributed
counters are coming in 0.8 . pagination is still a major problem though. But
for news feed you might not need pagination. providing a load more button as
facebook and twitter does should be good enough.

Hope it helps.

~~~
mrduncan
I'm not sure what version of Redis you were using, but depending on the data
you're storing, 2.2 uses significantly less space than previous versions.

Some data from GitHub: <http://technoweenie-ruby-onales.heroku.com/#34>

~~~
technoweenie
It looks like our approaches are very different. Stratocaster stores lists of
integer IDs, which Redis optimizes for pretty heavily. We're taking the list
of IDs and doing a multi-get from either memcache or mysql.

It looks like this lib uses Sets of the actual event data. It won't be able to
take advantage of the same optimizations.

~~~
pjscott
On the plus side, switching from storing zsets of actual event data to storing
integer ids should be a pretty minor code change.

------
mrduncan
I've been thinking about how to make better news feeds lately - here are some
other interesting talks/papers:

<http://blog.basho.com/2011/03/28/Riak-and-Scala-at-Yammer/>

<http://research.yahoo.com/node/3203>

<http://technoweenie-ruby-onales.heroku.com/#1>

[https://github.com/technoweenie/allofthestars/tree/master/ve...](https://github.com/technoweenie/allofthestars/tree/master/vendor/stratocaster)

~~~
jdp23
great links, thanks. [http://www.quora.com/What-are-best-practices-for-
building-so...](http://www.quora.com/What-are-best-practices-for-building-
something-like-a-News-Feed) has some more ...

------
gokhan
A news feed becomes a very interesting problem once you release it to the live
server. I'm dealing with one for the last year, and enjoy reading about them
whenever I can. Thanks for the article.

One problem for your approach would be handling of deletions. You should visit
all the recipient feeds to delete the item, and it's not that uncommon.

Another one is handling comments, which is the real thing creating the buzz if
you're providing the feed for the network effect. The article assumes a
twitter like publishing if I got it right. In a Facebook like feed, you may
want to include a couple of comments with the original item to display the
action (first two is always displayed with the feed item, for examle)and
async. load the rest. You may want to keep the exact number of comments to
display something like "6 more comments", and this means updating all the
copies with every comment.

There's also the problem of changing permissions, if you have one. In my case,
a parenting social network, user can set feed permission to "family only" or
"friends" or "my partner" etc. anytime, hence adding and deleting again to
other user's feed in your case.

------
nleach
Implemented something similar for a project I was working on a few months ago
using PHP. Just wrote up a blog post about my implementation:

[http://nleach.com/post/4315166514/activity-feed-in-redis-
usi...](http://nleach.com/post/4315166514/activity-feed-in-redis-using-php)

It's pretty similar to the Ruby example, but I figured I'd share it anyway.
Feedback very much welcome!

------
sim0n
The example in this post is the way I built a news feed for zazazu.tv except I
used PHP instead of Rails. We stored each unique feed in MongoDB and then
insert a reference to the unique feed to each user's stream (using Gearmand).
Fast and pretty much as scalable as the site would need it to be.

