Hacker News new | past | comments | ask | show | jobs | submit login

Excellent piece of writing.

I haven't done a whole lot of concurrency programming, so before reading this article I didn't realize that the events vs. threads debate is still being had. The example that Ryan Dahl likes to use to illustrate the superiority of the event model whenever he talks about Node is nginx vs. Apache. That example did more in my mind to reinforce the idea of events being superior to threads (in terms of speed and memory consumption) than anything else.

Keep in mind when reading this article that Alex recently left a little company called Twitter. It's safe to say that relatively few companies will ever have to scale the way that Twitter has.




Keep in mind when reading this article that Alex recently left a little company called Twitter. It's safe to say that relatively few companies will ever have to scale the way that Twitter has.

You mean the failwhale way? ;-)

Not meaning to discredit al3x but I really don't consider twitter a success story in terms of scaling. I don't know if it's incompetence or just some truly bad early decisions that they're still suffering from.

But one thing is for sure; other sites of much higher complexity have scaled much more smoothly to similar sizes (facebook, flickr, just two from the top of my head).


For what it's worth, I don't think Twitter currently counts as a scaling success story either. Hence this in my post:

"Twitter is still fighting an uphill battle to scale in the large, because doing so is about much, much more than which technology you choose."

That's part of my point.


Well, as you've been outed as being involved with twitter first-hand, can you shed some light on the problems they are having? Or is that internal stuff, not to be talked about?

I'm curious because I've worked with various messaging systems myself. And albeit never having pushed them to near twitter-scale, the principles of scaling those horizontally seem quite straightforward to me, unless complex routing comes into play. But I don't see complex routing at twitter.

To be more precise: For all I know twitter could (should?) probably just append those tweets to one file per user and would scale beautifully from there. Auxiliary services like search are a different story, of course, but those don't need to trigger failwhales when they break...

What wall do they keep hitting?


The complexities of scaling Twitter have been pretty thoroughly discussed elsewhere, both by Twitter staff and informed third parties.

At this point in time, I don't really want to comment any further on what issues they may or may not be having. I haven't worked there in over a couple months, and I'll bet that big parts of the system have changed in that time. I'm no longer informed about what's under the hood at Twitter, and I also don't want to second-guess my former coworkers, who I'm sure are doing their best. Sorry!


I have a feeling that hiring al3x is one of the few reasons that Twitter didn't collapse into a heap of cetacean corpses.

Twitter's original codebase started as a "my first blog" tutorial in Rails — the polar opposite of a messaging platform. That's a hell of a ship to turn around, especially live on the world stage while the userbase and datastore are growing exponentially and you're rapidly approaching the second half of the chessboard.

Facebook both very intentionally limited their growth (new colleges) and avoided their most expensive features (like photos) for as long as possible. Flickr was originally an open-ended MMO/MUD based on user-contributed content called Game NeverEnding, which was far harder to scale than the photo-sharing feature in their chat client that was extracted to base their final business on.


Twitter's original codebase started as a "my first blog" tutorial in Rails

Where on earth did you get that idea? It's not true:

http://en.wikipedia.org/wiki/Twitter#History

Moreover, the idea had been gestating for years:

http://en.wikipedia.org/wiki/Jack_Dorsey




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: