What a bull-shit statement. So Twitter engineers can't take any outside advice because those people aren't working on Twitter's particular system. No one else in the world could possibly know how to make large scale messaging systems work.
I am not saying any of the arm-chair analysts are right but it is pretty presumptuous to dismiss them outright.
While luigi's statement is a little extreme, I think the gist is right. Yes outsiders can offer advice on a system, but it's really difficult to offer sound advice without specific details of the system. Perhaps outsider advice shouldn't be dismissed outright, but it ought be taken with a grain of salt.
I very much agree that the devil is in the details, and the largest detail here is how Foursquare goes about calculating badges. So without knowing that the article can't fully evaluate alternatives. That has less to do with the author not being an engineer on Foursquare's team rather Foursquare not detailing the calculation. Maybe on Riak map/reduce is fast enough and Foursquare can shard by checkin, which would be much easier to balance.
Maybe they should just implement their own sharding based on user activity. Maybe a few SSD drives on their Postgres server would solve they issue. They through the issue out there, if nothing else works tell us why.
Someone on this thread suggested that Foursquare's performance problem is related to the calculation of badges. I'm not sure where this idea came from. I haven't seen any mention of the issue being related to badges in first or secondhand sources.
That said, Mongo DB's map/reduce operation would not be a reasonable solution at this time. Mongo DB's map/reduce performance is, at present, somewhat lacking because it runs via the javascript engine which is currently single threaded. I know there are plans to improve performance of Mongo DB's javascript engine by switching to V8, but I don't know if V8 is multithreaded.
Often design decisions that look bad in hindsight get baked in early. By the time you realize that an alternate design would yield better performance, there may be too much data to migrate so you just have to live with it.
harryh mentioned it here:
http://news.ycombinator.com/item?id=1769909
Computing the badges online (ie at each checkin time) requires having the whole db in ram for acceptable performance.
I am not sure if badge calculation is the bottleneck. Foursquare did say the calculation is why they have to keep everything in memory though. I just assumed that they sharded by user to keep all of the user's check-ins in the user document and that they are doing calculation with that set of data.
I am not saying any of the arm-chair analysts are right but it is pretty presumptuous to dismiss them outright.