The data input bottleneck is small. If you're going to do something in n*log(n), you should be fine. If you're going something in n^2, you're fucked anyway. Map Reduce is your friend. On that note, I only learned about Hive since coming to facebook. They made and open sourced it. Check it out: http://wiki.apache.org/hadoop/Hive
Exactly -- thus the question of how many machines would be needed to perform a certain amount of processing per tweet, which ultimately comes down to my original question of how many tweets there are.
I have decent access to the track/follow apis. I just queue the tweet right away and have a bunch of workers take care of it. Something like rabbitmq works wonderfully here.