

Killing Twitter Before It Can hurt Us - pchristensen
http://whydoeseverythingsuck.com/2008/05/killing-twitter-before-it-can-hurt-us.html

======
jamess
Does anyone else think that all this head scratching over what is the most
trivial of problems is hilarious? Twitter is just another naming service.
Instead of addressing someone as foo@foo.com, they're now just @foo. All
that's needed to make twitter distributed is a new protocol for name
resolution. It could be as simple as GET <http://distribtwitter.com/whois?foo>
returning
<name>[http://www.foo.com/mytwitterfeed</name>](http://www.foo.com/mytwitterfeed</name>).
You could write it in a day as a wordpress plugin.

Then other people could come along and write recursive caching distribtwitter
name servers so that they can still twitter at people they've never twittered
before when distribtwitter.com is down.

The only conclusion I can draw from this is that web 2.0 rots the mind and
turns trivial problems in to seemingly impossible problems.

~~~
tptacek
Elegant though it is, I don't think that idea will work; people who I don't
"follow" can "message" me in Twitter by including my name in a posting. That
doesn't work if I explicitly have to be polling their server to see anything
they say.

~~~
jamess
No, people who you don't follow can ping you saying they've sent you a
message. Your software can simply drop them on the floor if you don't care to
receive messages from people who you aren't following (AKA people who aren't
on your whitelist.)

No polling is involved.

------
iloveyouocean
Open source distributed Twitter - No f-ing way.

The open source part is fine, its the distributed part that would never work.
I don't think people realize the SMS infrastructure/expenditure needed to pull
off something like Twitter.

Their 5 digit short code costs (at least) $1k/month in the US. Twitter has
these in multiple countries. Each received message costs approx. $.03 and each
sent message costs approx. $.05. Added up over 30mil. messages per month (avg.
of $.04 per msg.) = $1.2mil. Twitter is almost certainly getting a better
rate, but their monthly messaging expenditure is certainly multiples of $100k.

For those of you saying, 'Oh, but what about sending TXT msgs via email. I do
that on my homebrew project, and its great!' Sending 30mil msgs (and growing)
WILL NOT work via email, the carriers will NEVER let you do that.

I could go on about the technical reasons Twitter cant be distributed and
FREE, but in any substantial mobile play you have to deal with the carriers,
their infrastructure, and their fees. It is a totally different scenario than
a website that has per page serving costs approaching 0.

~~~
acgourley
If you get enough messages on your shortcode, the carriers start to pay you
(since they make so much per a message). This doesn't mean an OOS solution
will be able to pull this off, but I just wanted to note that twitter need not
be stuck with that bill.

~~~
iloveyouocean
No, the carriers don't start to pay you, ever. The only scenario that leads to
the carriers paying you are if you have a Premium (pay per message) billing
agreement. Twitter does not.

Twitter has taken a significant amount of funding. They have not really
expanded their team much, they havent added many new features. They havent
been advertising.

I guarantee all of the money is going towards messaging costs and
infrastructure. Two things that a distributed, FREE Twitter would have a very
hard time accounting for. [But hey, Im all for it. I have many years in the
text messaging space and would totally be into helping however i could. I just
think that the solution would be far trickier than people might at first
glance suspect.]

------
tptacek
I do not understand how someone can look at a primitive 160-character web SMS
system and suggest that it was open APIs that made them vulnerable to
competitors.

~~~
ericb
The network effect is Twitter's barrier to entry. Everyone goes where everyone
else is. If you can easily move the network elsewhere via the api, the barrier
is gone.

~~~
tptacek
I'm not sure what this has to do with the API. People are "going to" Twitter,
not the Twitter API.

~~~
ivankirigin
The point is that you can add an opensource and distributed intermediary
because of the API.

~~~
tptacek
Yes, if you manage to convert a critical mass of Twitter API users into
Twitter-shadow API users. A similar "technique" converts all AIM users into
AIM-shadow users.

------
mtts
Fairly obvious, I guess. It's always surprised me as well why no one has made
Twitter into a protocol of sorts with a web app that you can host yourself
instead of a centralized service.

(the reason I haven't done that myself is that I have absolutely zero interest
in Twittering or Tweets or whatever that stuff is called)

------
henning
If Twitter went down permanently, all my tweets about Michael Arrington being
a jerk and what I had for lunch would be gone. Nooooo! :(

------
brooksbp
"This means the open source community will almost certainly be more clever
about the solution that one tiny closed source company because there will be
lots of heads focused on an easy to understand problem."

To be honest, I have to completely disagree. Lots of heads is not a solution
to a pure computer science problem. Most companies are formed around a
specific solution to a computer science problem. They then make everything
proprietary and profit.

That's like saying, only smart computer scientists exist in the open source
world. Not true. In fact, you simply cannot claim that there is a strong trend
in one or the other. Many smart brains in open source... Many smart ones in
industry as well...

~~~
jrockway
Twitter isn't a "computer science problem". It's a nice GUI for subscribing to
RSS feeds; RSS feeds that can only contain 140-character posts. Oh, and it can
email new entries to your phone.

~~~
hank777
actually the scaling is absolutely an interesting problem. Its the reason they
keep going down. because messages are not point to point it is a much harder
problem than instant messaging. I have spent a fair amount of time considering
the issue in detail because a friend of mine developed a caching system
targeted at microblogging. It is non-trivial, and cannot be done with RSS
feeds.

~~~
jamess
If you think it's a hard problem, you aren't thinking about it correctly.

When you say "@foo Hi, foo!" You're sending a message to some person named
foo. That's a push. If you start thinking of it as a pull, then you get in to
the sort of trouble you're mired in.

In a distributed system the process goes like this:

1) Sender queries name server: WHOIS foo? 2) Nameserver responds foo is
<http://www.foo.com/mytwitterfeed>. 3) Sender pings recipient
http//www.foo.com/mytwitterfeed?ping=messageid&origin=bar 4) Recipient queries
name server: WHOIS bar? 5) Name server responds <http://bar.com/mytwitterfeed>
6) Recipient requests message text,
<http://bar.com/mytwitterfeed?id=messageid> and verifies it actually is
addressed to him.

From then on, you have to worry about spam but that's a solved problem
(killfiles, Bayesian filtering, etc.) No caching is involved. Recipients store
messages addressed to them, just like any other messaging system. This isn't
at all a hard problem.

~~~
tptacek
This appears to be a system that allows strangers to coerce my client into
contacting their server.

~~~
jamess
What's your point? You might as well say HTTP is a method that allows
strangers to get files from my server! While technically true, it doesn't say
much.

Yes, people can send you false pings, which is why you need to go back through
the name server and do a reverse lookup, and once you fetch the message you
need to parse it and make sure it within bounds (less than 160 characters or
whatever) and addressed to you (has @foo in the message text.) and is not spam
(the name isn't in your killfile, is in your whitelist, akismet says it isn't
spam, etc.) Rate limiting helps too, real people aren't going to send you 100
messages a second.

~~~
Retric
IMO: They are dealing with a tiny amount of bandwidth say there are 30million
1kb text messages a day that's 900 gb / month which is well easy for a single
core CPU to handle.

Personally I would go with:

 _User Adam sends message to Bob:_ 1) Adam sends message to Twitter service.
"Bob>Hi." 2) Twitter check to see if Bob knows Adam. 3) Twitter sends
"Adam>Hi." to Bob.

(With optimal storage so Bob and Adam can see their old conversations)

Your system let's random spammers people find out people's address which IMO
is bad.

PS: Or Adam could send a message to "All>I like this soup." and Twitter then
sends the message to everyone that cares about Adam including Bob.

Edit: It looks like Twitter is sending around 2million Tweets a day.

~~~
brooksbp
You're thinking too qualitatively; imagine the spikes they get... Also most
apps are I/O restricted, not CPU...

Also, all these proposed messaging architectures are somewhat flawed. Twitter
isn't really "sending" messages to other people. It's more like a person's
message history is bound to their account, and appropriate privileges are
applied. Then, when people try to "read," privileges are obeyed and
information is produced...

Am I the only one who believes the solution to Twitter does not involve a
massive distributed system? Twitter is inherently centralized... people don't
see that.

~~~
Retric
Ok I can't edit the old post but they are basically just reinventing mailing
lists. Anyway, poling is fine if it's initiated by a user action or it's a
vary limited number of systems but if you have a few million users you can't
uses it per user. As to reading old log's that's a trivially parallizable
problem see database replication for some hints. So you separate the active
messaging that automatically adds new Tweets to a user’s cell phone from the
way users read old messages with the webpage and your home free.

PS: Text is cheep EX: Slashdot is running off of ~4 computers. And bandwidth
is not a scaling issue until you need a single system to handle more than ~1GB
of bandwidth per second. (Saturating an OC-3 line costs a lot of money but you
don't need to change your architecture to do so.)

------
whydowork
twitter is only cared about in an extremely small community of uber nerds.

The ideas in this post are overkill, although social portability cough
opensocial cough is already on track to assist.

~~~
estherschindler
I respectfully disagree.

While I confess that I am an ubernerd (in addition to other fine qualities
such as being a well-known chocoholic), I see plenty of people on twitter who
are not techie at all.

I follow a few quilters, for example; one of them has a blog with neat how-to
videos. And for some reason a lot of real-estate people from Phoenix are on
twitter (they follow me, I'm not too interested in them).

I realize that they can be invisible to techies. Like LinkedIn: you only see
the people on twitter who you follow personally, so there's a sense that
everyone is like yourself. But that's not the case.

------
yan
I like this post. I wouldn't mind being on the open source twitter effort.

~~~
diego
Let's do it. I just registered the twitshadow domain and I'm willing to
dedicate some of my company's resources to the project. I'm thinking about
what would be the best language to do it in. My first choice would be Python.

~~~
m0nty
"I'm thinking about what would be the best language to do it in."

"When you gotta shoot, shoot. Don't talk." Or, to put it another way: the best
way to start is to cut some code, don't get bogged down in discussion.

~~~
diego
Good point. Off to the whiteboard I go, see you in a while :)

edit: I'm aiming to put something up on the wiki here by the end of today, we
can go from there:

<http://code.google.com/p/twitshadow/>

------
hoffmabc
This was actually a really interesting posit. I think that it's sort of like
instant messaging. Someone is going to create the Jabber of Twitter technology
but I think there will still be room for the Twitters and Jaikus to run the
game.

