

Twitter responds to Techcrunch about its scaling issues  - lyime
http://blog.twitter.com/2008/05/its-not-rocket-science-but-its-our-work.html#links

======
ComputerGuru
Good to see the Twitter team reply to that disgusting Blaine-bashing that
TechCrunch had engaged in: _The folks at TechCrunch singled out a former
employee of Twitter by name in their questions but Twitter is a team—we share
responsibility for our victories as well as our mistakes._

------
socksandsandals
Wow, that was a perfect response. Couldn't have been classier. Kudos to the
Twitter team for not only responding well but also for taking the edge off of
Arrington's personal attack on Blaine

~~~
iamdave
I agree, and flawless execution contrasting the Mars landing and Twitter
operation. You couldn't ask for a more appropriate response.

------
nickb
"Our new architecture will move our reliance to a simple, elegant filesystem-
based approach, rather than a collection of database."

Finally! Using a database for Twitter was a huge mistake. Anyone who's
developing an IM should not use a database as a general message storage. And
especially not for a system that's multicasting at such a high rate.

~~~
axod
No one in their right mind ever would. Does IRC use a database for messages?
Yahoo/msn/aol/icq? no.

~~~
ajkirwin
Yes, but to counterpoint that..

they don't store anything, anyway. They're simply protocols, a network and a
login system.

~~~
axod
most of them support offline messages as well.

~~~
ajkirwin
Yes, but that's more a temporary thing.

~~~
axod
Well if you also need permanent storage - logs for example - a DB is clearly
not the right thing to use.

~~~
ajkirwin
Logging, in such a case, is usually done locally, not remotely. You're still
not making much sense here, axod.

------
chrisbolt
> We currently use one database for writes with multiple slaves for read
> queries. As many know, replication of MySQL is no easy task, so we've
> brought in MySQL experts to help us with that immediately.

This really makes me wonder if they know what they're doing. MySQL replication
really isn't that hard at all. They're hitting performance bottlenecks with
only 3 servers and don't appear to be looking into sharding/partitioning to
split up writes?

~~~
swombat
On the contrary, I'd say that they must _really_ know what they're doing. How
many sites with a million daily active users do you know that run on only 3 db
servers?

The twitter team deserves a lot of respect for managing to stretch their
hardware so far.

~~~
gaius
_smirk_ The company I work for serves volumes that dwarf that of Twitter. We
don't drop anything, because we are handling actual money, not tweets. And we
get by with 3 database servers, one of which is for failover.

OK admittedly they are wardrobe-sized Suns running Oracle, but that's not the
point. Yes it takes skill to do a lot with a little, but it takes skill not to
paint yourself into a corner too, and you need _both_ skills to succeed.

~~~
swombat
I'd point out that 3 wardrobe-size Sun servers running Oracle probably cost
about as much as 300 MySQL servers like Tweeter is running, but some might say
I'm the one being a smartass then :-P

------
wumi
does anyone else think TC is handling this situation like a reality TV show
where you always suspect they create the drama on purpose?

~~~
extantproject
Yes, TC is a click generator fueled by drama.

... and this game of "let's play Twitter engineer from outside Twitter" the
entire space is playing is pathetic. Twitter will grow Twitter. Ev has been
through this before and will guide the team and the service to stability. For
the rest of us there are better things to do than trying to scale Twitter from
an armchair.

------
axod
At least we know the number of "stories about Twitter scaling" scales. There
has been a big increase in the volume in these stories, but they just keep
coming.

------
gcv
The original TC article made me feel rather upset. What kind of journalist
takes that kind of tone and makes personal attacks against anyone? Arrington
has done this repeatedly over the last few months.

Startups have to strike a balance between getting the architecture right and
getting a product out the door. The bright and lucky ones come up with a
minimally correct infrastructure which is approximately equal to what they can
hammer out with their existing resources in the least amount of time. Most of
the time, teams aren't experienced or lucky enough to pull that off, in which
case it makes sense to try to fail early --- this means not spending time on
infrastructure. They get the concept out, and see if anyone wants it at all.
There's no point in releasing a product which scales to 100 million concurrent
users if no one wants it. In its early days, Twitter must have done what it
could to get its service up and working, and used MySQL for the same reason
you (yes, you!) use it: the team was familiar with it, and was used to either
stuffing code with SQL strings or using an ORM.

After the service took off, a lot of things could have happened. Has Arrington
bothered to ask Blaine Cook if he had the resources to do The Right Thing? Did
Blaine have the time, or did his company tell him to sit in front a glowing
screen and do CPR while the CEO goes out to raise cash? A journalist should
have at least indicated that he attempted to hear the other side of the story.

<http://en.wikipedia.org/wiki/Yellow_journalism>

~~~
delano
Journalist?

------
codesurgeon
What is it with some people whining about the (supposed) "damage that Twitter
has done to the community". Not really bright, IMHO. Nobody is being forced to
use twitter and so far no one has apparently come up with an alternative that
is viable enough for droves of people to switch. I don't want to play the
uncritical Twitter advocate, but I like the concept, the service and this
answer to TechCrunch in particular.

------
1gor
Why Twitter, being a glorified IM, is not moving to an IM architecture like
<http://www.ejabberd.im>? Telecoms have solved their availability issues with
Erlang. Why reinvent the wheel?

~~~
gcv
I heard that Twitter does use ejabberd for a bunch of things, including its
"fire hose" stream of all messages. Anyone care to confirm?

------
bluelu
Even if they defend their lead architect, he still failed at scaling twitter.

~~~
bluelu
Thanks for downmodding me. But it still holds. If he is the lead architect, it
is his job to design the system that works! If it doesn't work, then it's his
fault and not anyone elses.

