Hacker News new | comments | show | ask | jobs | submit login

I was at the presentation - it's a very smart system. They basically maintain a whole bunch of counters for any particular thing that's being tracked. For example, say someone clicks on a t.co link to blog.example.com/foo at 11:41am on 1st Feb. Rainbird would increment counters for:

t.co click: com (all time)

t.co click: com.example (all time)

t.co click: com.example.blog (all time)

t.co click: com.example.blog /foo (all time)

t.co click: com (1st Feb 2011)

t.co click: com.example (1st Feb 2011)

t.co click: com.example.blog (1st Feb 2011)

t.co click: com.example.blog /foo (1st Feb 2011)

t.co click: com (11am-12 on 1st Feb)

t.co click: com.example (11am-12 on 1st Feb)

t.co click: com.example.blog (11am-12 on 1st Feb)

t.co click: com.example.blog /foo (11am-12 on 1st Feb)

t.co click: com (11:41-42 on 1st Feb)

t.co click: com.example (11:41-42 on 1st Feb)

t.co click: com.example.blog (11:41-42 on 1st Feb)

t.co click: com.example.blog /foo (11:41-42 on 1st Feb)

So that's 16 counters to track one link, but it means they can do fast, denormalised queries in realtime to track how that link is performing.

It's not just for t.co links - they can use it for internal server monitoring tools, tweet counts, advertising metrics... pretty much anything that involves counting at scale.

It's possible to build a similar system for much smaller scale applications using atomic counters in Redis - I've been experimenting with something like that for some of my own projects.




We use Redis for this kind of stat tracking at http://www.formspring.me/

At 10,000,000 responses per day, we're not that small.

Here's the source from our collector: https://gist.github.com/794474


if you follow various twitter tech talks you'll notice that they spend a lot of time talking about 'fan out'. they basically prefer writing many things asynchronously vs. having to aggregate later. makes sense... disk is cheap and only getting cheaper. but processing time to return realtime data and information is finite.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: