Follow up question for a little clarification. Would you write the tweet into the queue once or append a message to the queue for each follower containing the tweet id?
Seems to me you’d need to do the latter right? That way you ensure you process each follower at least once, but each worker doesn’t need to be aware of others it’s just pulling messages from the queue.
I don't think so. Maybe for the active users, but maybe not even then. Most twitter users are inactive and so you wouldn't want to fan that kind of data out so widely. You'd either have to pay enormous gc costs or enormous storage costs for someone like Donald Trump (0.5 GB per tweet assuming 64 bit tweet IDs). But I think I would just have a sharded in memory cache of recent tweets for the big dogs and if someone loads their feed you would do lookups into the sharded db for each subscription. You'd then cache the feed for the user so you don't have to do that lookup very often.
If you were going to write, you'd write into the cache, rather than persisting anything to disk. That's my guess at least.
Seems to me you’d need to do the latter right? That way you ensure you process each follower at least once, but each worker doesn’t need to be aware of others it’s just pulling messages from the queue.