

Decentralized Twitter? You just don't get it. - volaski
http://volaski.tumblr.com/post/29720297216/open-twitter-nope-you-dont-get-it

======
csense
Why does a decentralized service entail high latency?

The article's author takes the single example of email, and, extrapolating
from that, assumes that it's impossible to build low-latency decentralized
publishing protocols.

And his statement, "Protocols are for when there’s no good centralized way of
getting things done," is so ridiculous, I don't even know where to begin.

~~~
volaski
Please begin though, since I will never know why you disagree with me if you
don't begin.

"Protocols are for when there's no good centralized way of getting things
done" is actually not as crazy statement as you think, since it also means "If
there's a good decentralized of way of doing something via protocols, you
don't need a centralized approach".

What I meant by "getting things done" was efficiency. Decentralized and
centralized systems each have their pros and cons, and efficiency is
definitely not one of the pros of decentralized systems.

I do apologize for the incoherency in the post but I still don't think I
didn't make any incorrect points. I think it would be if you could provide a
counter example to my statements.

~~~
erichocean
_efficiency is definitely not one of the pros of decentralized systems_

The economy is a simple example of a system where decentralized == efficiency
and centralized == inefficiency.

~~~
volaski
Great point! I agree, once a system becomes really huge it becomes hard to
maintain and centralized system which used to be efficient becomes
inefficient. However when the system CAN be maintained they are generally more
efficient IMO. For example many developing countries go through a stage where
the government makes a centralized plans. Even the U.S. went through this
until the point where the scale itself brought forward inefficiency. However
when it comes to an internet service like Twitter which CAN be maintained in a
centralized manner--at least so far--I don't think decentralized approach has
any benefit in terms of efficiency :)

------
mbell
I get the impression the author doesn't have a clue what 'realtime' means. I
appreciate he is trying to say 'within seconds' but that is not realtime, true
realtime is a completely different game.

~~~
protomyth
Yep, twitter is not the classical realtime taught in school. He probably would
have done better by referring to it as "everyone in the same living room
commenting on the TV". Its timeliness is directly linked to how many people
are at / watching the same event. The tweets about a NASCAR race or the plane
coming down in the Hudson really would be a pain / confusing if received hours
later. For a lot of stuff, this doesn't matter so much (like e-mail).

"Pause and Consider" is pretty much a lost concept in the online news world.
Look at the Apple screw fiasco for a point when the repeaters didn't. Maybe
having a distributed network of sorta timely servers wouldn't be such a bad
thing. I seem to remember debates happening fairly rapidly on USENET.

~~~
thinkingisfun
There ought to be a way to bring IRC into all this..

~~~
ChiperSoft
Exactly, IRC networks have been realtime AND decentralized for decades, they
even solved the redundant data problem.

------
rosser
The only thing using a centralized service buys you is a single point of
failure.

(FTR, this is hyperbole, but not by much.)

~~~
volaski
That's exactly my point. Single point of failure means single point of
responsibility, which means that single point of failure must do a damn good
job maintaining it.

That's why a "non-profit" approach will never work.

~~~
smlacy
So, by that measure, "The Web" as a nonprofit (w3c.org?) will never work and
should be a huge failure, right?

~~~
volaski
Did you get to read my post by any chance? That's not what I talk about in the
post. My point was, sites like Wikipedia are successful as non-profit, but the
type of value Twitter provides is very different from the value w3c or
wikimedia provides and should be considered with a different frame.

~~~
smlacy
The only value twitter provides is in the value of it's users. There's
virtually no value in the platform itself.

------
rbanffy
I have been playing with the concept of a distributed Twitter like thing built
on Google's App Engine. You download it, edit one file (or run a setup
program) and upload to your own account. Unless you become insanely popular,
you should be well under the free tier limits.

Google would probably hate me.

~~~
im_dario
I already did it <http://code.google.com/p/qantiqa/> It was my CS degree final
project.

It is based in an obscure P2P framework from my university (based in another
one) and it uses GAE as an overlay's entry point. The network is fully
functional and independent of the entry point, but it is needed to do
authentication.

~~~
rbanffy
I imagined relying on Google accounts for the App Engine application and on
https for internode communication. Pushes to subscribers could require a
confirmation callback but https would ensure you'd be conforming with the
right sender.

~~~
im_dario
Wouldn't be more easy to solve sender identity with PGP keys? Just a random
idea.

~~~
rbanffy
XMPP is cooked into the platform. I wonder how hard would it be to use it with
other implementations.

PGP is probably too complicated for the average user. I wouldn't like to
expose people to that (and failing to do so while relying on it would open a
lot of attack vectors)

------
mikeryan
Honestly of my tweets showed up at the speed of email instead of the speed of
an IM my twitter experience wouldn't suffer one iota.

------
dmacedo
So AT&T isn't decentralized? I'm not an AT&T customer, nor do I live in the
states, but I assume AT&T is a a decentralized _network_ which uses a
communications model or let's call it a "protocol" to talk to other networks.

If you tell me AT&T is centralized because it's covering a country, or that
you can call all your family and never leave AT&T's network, that doesn't mean
it's centralized per-se, it might be centralized for it's own services, which
makes sense for any company.

But it's still part of a decentralized network of communication providers,
which use a protocol... A live one actually, kind of like what twitter could
become a king node of (after all, the celebrities are there, right?).

Oh... I see what you meant, I might have misread that whole post!

------
TYPE_FASTER
Twitter is just hosted, proprietary pub/sub. You could do the same thing with
a XMPP server and client for a long time before Twitter appeared. I think the
only reason people are asking for decentralized Twitter is they're not aware
pub/sub protocols that do the same thing exist.

~~~
volaski
In fact, "Twitter is just hosted, proprietary pub/sub" is exactly the type of
thinking that I was criticizing in the post. Sure the technology is not
difficult to implement, but the point I was trying to make was Twitter's value
is not in the technology but the realtime delivery network it created.

~~~
smacktoward
But XMPP does the exact same thing, and it works in a distributed fashion. I
can have an instant-message conversation in my IM client over XMPP with
messages being exchanged effectively instantaneously. There's no appreciable
delay between me hitting Enter and the other party seeing my message, barring
problems on the network between me and them; and those problems would degrade
my perception of Twitter's performance too, if they were between me and
Twitter. While centralized instant-message services exist (AIM, for instance),
in 15 years of using instant messaging more or less daily I've never ever had
the experience of feeling like a conversation I was having over AIM was
happening "faster" than one I was having over XMPP. Never. It has literally
_never happened._

XMPP is performant enough for Google to use it as the IM component of Google
Talk. Google Talk is instant messaging. Not "eventual messaging." Instant.
Just like the centralized alternatives.

Centralized services have some pros over distributed ones -- ease of
discovering other users is a big one. But you're just making an assertion that
only a centralized system can pass messages back and forth fast enough to be
perceived as effectively instant without providing any evidence to back that
assertion up.

~~~
volaski
Let's say you hosted your Diaspora or Status.net or whatever on your heroku
server. You post something and it gets huge traffic and the server goes down.
And this doesn't happen just for you but for lots of people on average. Who
will maintain this? In this type of environment, you cannot make any
assumptions on realtime delivery. That's what I mean by it's not a technology
problem. You can use XMPP in Google Talk since most of the times you're doing
1:1 chat. Twitter does 1:N (where N goes up to as large as several millions),
and this becomes a big deal. Does this make sense?

------
jerrya
Imagine tweeting what you ate for lunch, only to have it show up at
dinnertime.

I believe this is what the author means when he says the value of twitter is
in realtime delivery.

------
neilk
We already have an existence proof of near-real-time 140 character messages
crossing networks. It's called SMS.

------
vannevar
Are there any statistics on the average delivery time for email? Meaning
delivery to the mailserver, not final retrieval by the client. While email
tends to be _used_ more asynchronously (ie, most people don't constantly check
it), is it really much more _inherently_ latent than Twitter?

~~~
volaski
The important thing is this has nothing to do with technology. It's more about
psychology--what type of assumptions people make when using these
technologies.

------
thinkingisfun
_by definition decentralizedness means no one is responsible_

That is flat out wrong.

 _In this sense, I would go as far as to say “Protocols are for when there’s
no good centralized way of getting things done.”_

That isn't even logically connected to the previous sentence at all. I don't
understand this bit, either:

 _Realtime protocols like PubSubHubBub and RSSCloud are there because that’s
the only way to make the Web work in realtime. It doesn’t work the other way
around._

The only way I can parse this is as refuting the strawman that "the web works
in real-time because there are realtime protocols". But of course it does,
even Twitter has internal protocols that work in real-time.

Would GMail be faster if they had invented their own protocol? No, it would
just be dumb.

So... in summary: wait, what?

~~~
volaski
I agree my post wasn't coherent. Forgive me I just wrote down what came to my
mind and posted it.

That said, I don't understand what you are trying to say either.

It would have been nice if you discussed why my argument "by definition
decentralized means no one is responsible" is "flat out wrong".

Also regarding my statement "Realtime protocols like PubSubHubBub and RSScloud
are there because that's the only way to make the Web work in realtime", maybe
I didn't state it clear enough, but it's not as obvious as it sounds in my
opinion. Before RSSCloud and PubSubHubBub, if you wanted to run an aggregator
you had to actively poll sites, which was extremely inefficient. With these
protocols now aggregators can deliver content in near realtime because they
just have to implement those protocols. The point I was trying to make was,
even these efforts will never be completely reliable because they are built on
top of protocols and standards. Protocols and standards work only when "good
enough" is good enough.

For example, building a decentralized version of a stock ticker system sounds
cool but in practice its performance will never be as good as the original
centralized version.

~~~
andreasvc
In a decentralized system like email or jabber there is responsibility,
@example.com is responsible for delivering messages to its users, and for the
necessary communication with other servers.

And of course you'll want to use push messages for communication between
servers. It really doesn't have to add all that much latency, a few roundtrips
is perfectly doable. That's a very small price to pay for the benefit of being
independent of the whims (or unreliability) of a single entity.

That said, it's probably not going to happen soon, but not because of
technical issues. I think it's just a matter of inertia among users, just like
people are still using MSN messenger or skype instead of jabber.

~~~
volaski
I think the when we talk about Twitter, the definition of "reliability" is
entirely different. When we talk about emails, we can say it's reliable as
long as my email gets delivered one way or another. For example, if I send a
mass email to 20 friends, I don't assume anything about when each of them will
receive my message. I just think that it will get delivered in a reasonable
amount of time.

However what Twitter brings to the table is this: It is safe to assume when I
tweet, all my followers will be able to see it immediately. This is a huge
difference not because of the delivery time, but because of the type of
behavior it enables. Being able to assume that my message is delivered in
realtime is very different from not being able to assume. And this is not a
technical issue, so even if you build a "good enough" realtime system using
decentralized architecture, it wouldn't be half as good as Twitter.

~~~
andreasvc
It think the difference between 0.1s and 1s, as I imagine it would be, is
quite irrelevant. The immediacy comes from the push, adding another hop for a
federated server does not have to add a significant latency.

