
Twitter Could Have Become a Protocol - gardnersmitha
http://blog.austingardnersmith.me/how-twitter-could-have-been-a-protocol/
======
macca321
They built it.

They had an experimental project called 'annotations' where you could attach
1k of json to each tweet, like a DIY microformat.

I got onto the beta and created a prototype twitter client which you could
attach mini 'apps' to tweets based on the payload type, e.g. you could tweet
out a poll, or an inviation to play a game or a job advert or whatever, and
you could attach your own app as a listener.

I think it could have been pretty amazing, basically A Message Bus For
Everyone. If you build this yourself, you will run into the chicken and egg
position, twitter could have pulled it off as they had the eyeballs and the
developers.

Unfortunately they pulled the plug on it, around the time they started to
close down the ecosystem.

~~~
throwawayReply
It's not surprising really, is being a centralized message bus really
attractive when it just means that you'll be responsible for policing it?

If you run a central message bus off which people can produce their own
applications, you'll find yourself having to deal with everything that ISPs
and mail providers have to deal with. That means being asked to keep logs for
years, dealing with DMCA takedowns, filtering the worst of the internet, etc.

And for what? As soon as anyone relying on it achieves scale they'll re-
implement it themselves.

Besides, what you've described is the opposite of a protocol: an inner
platform.

~~~
macca321
I disagree that anyone else could re-implement it. Once you have a critical
mass of listeners, it would become the first port of call for any message you
wanted to distribute.

As for policing, I think that would probably be where the money would be made
- reputation/quality of the messages being posted.

~~~
loceng
Why would people pay additionally when they don't need to pay now (relative)
to distribute/post messages?

~~~
macca321
Suppose twitter had become a giant json firehose, and you could listen to it
and, for example, build a jobs listings website from the jobadvert
microformatted tweets.

It would end up full of fake job spam, and the listeners might want quality
filters. I can imagine that you might pay to validate your identity as not-a-
spammer.

~~~
loceng
How would "Twitter" capture that value and not some third-party platform who
and offers those filter services? The value and revenue isn't with Twitter
then, so then how does Twitter pay bills? Or am I seeing this in a different
light than you do?

~~~
dasil003
As far as I can tell a simple freemium model would have worked. You charge for
the firehose and anything high volume, but you give a generous free allowance
to keep the grassroots ecosystem healthy. They could have played all the same
pricing games that Facebook is now playing, except across a much wider range
of channels.

I'm not saying it was a slam dunk, but a lot of talk about this back in
2007/2008 and there was a palpable excitement about Twitter apps. Within a few
years the media-company mindset had taken grasp, the valuations and
expectations exploded, and now we have boring old Twitter which is still a
damn cool thing in its own right, but which is seen as a failure because of a
combination of pedestrian vision and mismanaged expectations.

------
ronjouch
A similar complaint came up here recently on a thread called _Next steps for
Gmane_ [1], starting with _" I really miss the newsgroups that focused on just
the messages, and could be consumed by any NNTP client, stored offline,
searched, etc."_

Quoting a portion of a great answer [2] from HN user niftich, that feels very
appropriate here:

> _" Not enough people make new running-on-TCP or running-on-UDP protocols
> because new protocols are hard to design, they don't work with the one
> application where everyone spends 70+% of their time (the web browser), and
> they probably get blocked on a middlebox except if you use port 80 or 443
> and fake being HTTP anyway. For all but very specialized use-cases, vomiting
> blobs of JSON (or if you want to feel extra good, some custom binary
> serialization format like protobuf or Thrift or Cap'nProto or MessagePack)
> across HTTP endpoints is pretty okay."_

[1]
[https://news.ycombinator.com/item?id=12440230](https://news.ycombinator.com/item?id=12440230)

[2]
[https://news.ycombinator.com/item?id=12440783](https://news.ycombinator.com/item?id=12440783)

~~~
batbomb
Running over 80 and 443 using an arbitrary protocol often doesn't even work
with next gen firewalls because they don't recognize the protocol as HTTP.

~~~
nostrademons
Curious how that would work on 443, given that the whole point of SSL is that
you can't recognize the payload as...well, anything.

~~~
laumars
Payload aside, SSL is identifiable as SSL. The host name is transmitted in
clear text (SNI) as well as the protocol version and cipher negotiation.

Just because the traffic is encrypted, it doesn't mean the connection type
isn't identifiable.

~~~
nostrademons
Right, but presumably any arbitrary protocol that plans to tunnel over port
443 would use a standard SSL negotiation, standard HTTPS connection, and then
send the application-level protocol as part of the encrypted payload.

My very first job (back in 2000) actually did exactly this - we tunneled
arbitrary TCP/IP traffic over HTTPS. The startup died over various management
& investor relations fuck-ups, but the product was working - you could do
things like make a SourceSafe connection across a firewall without exposing
the server to the Internet. There was some concern about whether firewalls
would kill the connection if it was open for too long, but it turned out that
none of them actually did this (which unfortunately we didn't figure out until
we were almost out of money and had wasted several months handling this case;
see again re: management fuck-ups).

~~~
laumars
That's a different point though as you're now effectively using HTTP as your
transport layer rather than using your proprietary protocol "naked" on port
443. So whether you send your protocol in a proprietary binary format over
HTTP or via JSON API's, it's still just HTTP over TLS. You're still sending
HTTP headers, just with our own proprietary content as the body. The real
problem is if you want persistent types of connections. As websockets have
taught us: a great many equipment that supports HTTP don't support websockets
and thus why even websites are still often stuck with RESTful APIs even when
that doesn't always fit their problem quite right (that situation is improving
however).

If one is using HTTP to transport your protocol (and the aforementioned
websockets point isn't an issue) then your protocol should work over port 80
as well as 443. Barring any corporate / nationwide web filtering; but as
others have already pointed out, that could also be an issue with 443 even
with advantage of TLS.

By the way, was it Socks2HTTP you worked on? I remember that project back
around the time you described and was quite fascinated by it.

~~~
sly010
> That's a different point though as you're now effectively using HTTP as your
> transport layer rather than using your proprietary protocol "naked" on port
> 443.

HTTPS is built on top of TLS, not the other way around. You can't (passively)
tell if a 443 TLS connection is HTTPS or a proprietary stream. You can take a
guess based on statistics (which is what the Chinese firewall does to detect
tunneling) but that's about it.

~~~
laumars
I'm aware HTTP is built on top of TLS in HTTPS. I think what you're discussing
now is a little different from the HTTPS as a transport vs bespoke protocols
running "naked" over TCP/IP (unless I've badly misunderstood any of the
previous posts?).

Warning: nonsensical brain dump follows:

However to address your point, you might be able to use SNI (which is sent
from the client before the TLS connection is encrypted) to make some
assumptions about the content. Granted this would be more in the realm of web
filtering where you'd blacklist suspect domains or - in extreme cases - banned
terms within hostnames. I wouldn't be surprised if SNI is one of the
"statistics" the Chinese firewall uses (I'm not familiar with the
implementation details of the Great Firewall of China")

------
wvenable
> Twitter had a chance to become a sort of de-facto API for lots of other
> applications.

The problem I see is that there doesn't seem to be any way to make money from
this, for Twitter.

And really if everyone started using it this way, the privacy concerns would
be even greater than the concerns people have about Facebook.

~~~
dkarapetyan
Yes there is. The same way there is money to be made in providing premium
content. You could pay for extra capacity, bandwidth, fault tolerance,
indexing, archiving, and many other desirable properties you'd need out of
your messaging layer.

Instead the convoluted strategy by the higher-ups destroyed the entire thing.
They thought they could make money with ads and "promoted content" but they
somehow managed to fumble the execution of that one as well.

~~~
phmagic
I think this is the Medium strategy. I think there is a mid size business in
it, but one that's hard to defend and will see margins erode with time.

There is absolutely no "moat" in Twitter being a protocol.

~~~
dkarapetyan
You'd be surprised what happens when you become a hub and start seeing the
effects of 3rd party integrations. That aspect can be turned into a moat if
managed properly.

~~~
phmagic
I think you are right but there's a time and place for it. I don't think it
can save Twitter at this point but could have created a healthy ecosystem if
timed well.

But it's very hard to time this well. Even Facebook and their apps ecosystem
didn't create enough of a moat.

~~~
enraged_camel
Well, Facebook's "app ecosystem" was dominated by Zynga. For a long time,
Farmville was the top app and it was the only app that Facebook's highest
engagement users interacted with. As such, when Zynga went downhill, it took
the entire ecosystem with it.

------
kbar13
[http://webcache.googleusercontent.com/search?q=cache:vkzvQD4...](http://webcache.googleusercontent.com/search?q=cache:vkzvQD4TMkkJ:blog.austingardnersmith.me/how-
twitter-could-have-been-a-protocol/&num=1&hl=en&gl=us&strip=1&vwsrc=0)

------
Spooky23
Twitter was too busy navel gazing and pushing PR about how they had the
ability to incite revolutions.

They were afraid to grow beyond tweets. In life you need to grow or die. Ten
years from now, they'll be fondly remembered as the AOL Instant Messenger of
the 2010s.

~~~
enraged_camel
>>In life you need to grow or die.

Says who? Plenty of businesses reach a certain size and maintain healthy
levels of profitability over the long term.

"Grow or die" only holds true if you receive venture capital.

~~~
Spooky23
Not just grow in terms of dollars, grow in function and purpose.

If someone knocked you on the head in 2011 and you woke up today and logged
into Twitter, you'd find that little has changed.

~~~
nbb
So what?

------
wlesieutre
Didn't App.net try to be more of a protocol than twitter? How's that going for
them?

~~~
idlewords
It worked out great for Dalton Caldwell, who failed upwards to become a YC
partner.

------
dkarapetyan
Yup, and they bungled that one pretty well when they shut down the entire
ecosystem. They never recovered after that. They lost all the developer
goodwill and any chances of becoming a platform was vaporized at that exact
instant.

------
ilaksh
See reddit.com/r/rad_decentralization

All of these technopolies will eventually be superseded by protocols. It just
doesn't make sense to continue to rely on monopoly companies to provide core
services. That's not to say there can't be variations or companies that build
on top of core protocols etc. Just more than one, and not for the most common
aspects.

------
phmagic
I think the author has some good ideas about the things Twitter can leverage
once it becomes the de facto method of broadcasting in the world.

However become a protocol would not have helped Twitter get there.

I think the better path is to build and then protect a captive audience.
Instead Twitter saw its audience based erode away not once, but multiple times
with Facebook News Feed (Twitter for news) then Instagram Video (Vine).

Twitter once had a massive captive audience and unique data. Now their
competitive advantages have all disappeared. Opening up more won't help them
gain an audience.

------
rm_dash_rf
This is where I think people are still missing the boat on twitter.

Twitter is not an aggregator of data to be passed along. Twitter is a massive
un-walled global community. Though people may want access to the data as a bi
product of that. The future is in cultivating community, not protocols.

~~~
mxuribe
I respectfully disagree as to your note about twitter being "un-walled". The
easiest test would be: if twitter would shut down right now, and its
infrastructure would go dark as a consequence, could you/our/any community on
twitter continue? I believe the answer is "no", because its centralized (and
proprietary). I do agree with you that cultivating communities are
important...though we do need some universally accepted, open, and available
protocol to allow for cultivating of communities...sort of like what legacy
protocols have done - e.g. email, irc, etc.).

~~~
rm_dash_rf
I see the power of twitter in its brevity not its depth. Its a content
gateway.

I think legacy protocols such as email, IRC, SOAP...have proven time after
time that there is no need for an information specific protocol.

Twitter could be more open, but I think they realized that would be giving
away the keys to the castle.

------
gamache
I think one of the more insidious problems with many of these unicorn startups
is that there is no longer the fiduciary freedom to make "modest" plays like
becoming a protocol.

A company with $1B in venture funding isn't allowed to aspire to being a $500M
business. It must grow as big as the sun, or die trying.

~~~
0xcde4c3db
I think there's an even more general disdain for commodity offerings, at least
in American culture. There's probably a guy who's in the .01% of his ZIP code
because the regional store brand paper towels needed some of those cardboard
tubes. But "success" looks like Zuckerberg or Jobs, not like that guy.

------
herbst
V1 api was awesome and part because i started to love programming, v2 was a
pain already" simply took away the simplicity and now i can not even create
unlimited accounts anymore, even less use the API with those.

Twitter lost big in that regard imo

~~~
dingaling
Indeed, my weather station used to sit and tweet happily all day and all night
to all _four_ followers.

Then they introduced rate-limiting ( 60 per hour, when the station generated a
Tweet every 48 seconds ), application validation, phone-number verification
for accounts... I gave up after that and just resorted to RSS.

------
soufron
Wasn't there already Jabber or IRC? But they don't make money. Sad world where
we prefer badly coded shit that makes money, to awesome tech that don't.

~~~
pmlnr
badly coded _fancy/shiny_ shit with a web interface ;)

Apart from the sarcasm, I'm serious on the web part, which is terrible: all
around the world people are "porting" protocols to JSON+HTTP ( example:
JMAP[1] ). IRC is awesome[2], but it's not GET & POST and the new kids on the
block run away if it's a real protocol instead of a HTTP hack.

[1]: [http://jmap.io/](http://jmap.io/)

[2]: [https://aaronparecki.com/2015/08/29/8/why-i-live-in-
irc](https://aaronparecki.com/2015/08/29/8/why-i-live-in-irc)

~~~
tormeh
It's because of firewalls.

~~~
pmlnr
Sorry, but this is bs.

You could run any application on any port: IRC on port 80 with SSL would still
run just fine.

~~~
oarsinsync
It'll run fine, but I won't be able to connect to it from the inside of a
corporate firewall, which doesn't actually grant me access to the internet,
but redirects me transparently to an HTTP proxy that only permits HTTP
requests to tcp/80 and tcp/443, with SSL interception on the proxy.

You can run whatever you want however you want, but that doesn't change the
reality of the enterprise environment.

~~~
wtbob
The thing is, you really shouldn't be using your corporate network for non-
work-related things. Use your own network service for that kind of thing!

Do you _want_ your boss to see your reddit history?

~~~
notacoward
Some of us don't have Reddit histories we'd be ashamed of others seeing.

------
athenot
I've been thinking along the same lines. I really wished there was an open
protocol that provides all of twitter's functionality (and no more, to avoid
scope creep).

But as much as I hate to admit it, Twitter's value is Protocol + Moderation.
I'm not savvy with their operations but having worked with other cloud
platforms, I know that any platform that has even a modicum of visibility is
immediateley abused in many ways that are hard to forsee. Malicious attacks
against the platform itsef are also an issue.

Email is a good example of open and federated platform, which unfortunately
carries more abuse & noise than actual signal.

I still think building a protocol that incorporates a form of moderation is
possible, but I'm not sure how to solve this.

~~~
Tenhundfeld
> Email ... carries more abuse & noise than actual signal.

Really? That is not my experience at all. There was a span of years where spam
got bad, early-2000's IIRC, but my email is now mostly signal and has been
since since at least 2010. I think this is partially due to better spam
filters, e.g., gmail was better than average when it went public in 2007ish,
but also due to the rising popularity of DKIM and SPF.

I see the abuse and noise out there on twitter and in comments sections, etc.
I don't see it with email. I know people with big public personalities do get
hate email and whatnot, but I thought for the average person, email was kind
of a solved problem.

~~~
athenot
That is indeed due to spam filters. Look at what a typical MTA gets before the
filter kicks in.

But to your point, maybe the same mitigation technique can be applied.

------
ihsw
I was under the impression that the intention was that Twitter would be the
largest message bus in the world, but management decided it would become a
media platform instead. Clearly as a media platform it has failed and now it's
neither.

------
jonahrd
Twitter has restrictions that make perfect sense given its use, but would be
really strange decisions for a protocol: ie. 140 character limit.

~~~
LeoPanthera
The 140 character limit is so that Tweets can fit in a 160 character SMS, with
space left over for a username. SMS was originally a popular way to use
Twitter. (And I think you still can?)

~~~
lucaspiller
SMS is actually 140 octets, but the GSM 7-bit encoding that's typically used
means you can get 160 characters. If you use characters outside that set, its
switches to 16-bit UCS-2 so you are limited to 70 characters.

In reality you can send messages longer than this and they will be split into
multiple messages over the wire - however in the US where you had (have?) to
pay to receive messages it meant you would be charged for each individual
message.

TL;DR; These limits may have made sense for the MVP, but as soon as most
people moved to IP clients they were obsolete.

------
sulam
Keep in mind that operating Twitter, even if you eliminated all the stuff
that's there to support advertisers, requires (bare minimum) roughly 2K
servers just to handle the runtime request path. Monitoring those minimally
requires another ~500 servers to ingest and aggregate all the log data and
metrics. This wouldn't support any analytics platform, wouldn't allow for spam
identification (because you wouldn't be able to build any spam models), so if
you care about that stuff you need to add another 5K servers or so for the
Hadoop ecosystem.

Essentially you can't run an advertising-less Twitter under roughly 10K
servers (order of magnitude accurate -- people will want to quibble over these
numbers but they won't be able to push it below 5K). I may well be forgetting
something important that pushes it higher!

End result: there's no way to monetize Twitter at the scale it operates at
without decentralizing it. Many of you will go "yeah, of course -- it
shouldn't be a centrally controlled system in the first place." That's a fine
sentiment, but many problems you can solve in a straightforward (not easy, but
doable) way in a centralized system immediately become much, much harder, and
now you are asking ISPs and individual users to operate these resources for
free.

Take spam as one example. The main "solution" to spam we use these days is to
all use a system that see enough email to power machine models that identify
and filter spam out before we have to deal with it. IOW, we use centralized
systems. This gets paid for with advertising!

You can fractally reinvent the system or you can just have Twitter.

------
KirinDave
Blaine Cook used to talk this way. Then he "left" twitter, rumors saying it
was (among other reasons less interesting but dreadfully predictable) because
the management didn't want to try and make twitter a hybrid of a human-to-
human and machine-to-machine network.

Blaine even gave a really interesting presentation I remember watching on the
subject, about how very challenging such a backplane is compared to a more
simple human-human messaging network.

------
bryanrasmussen
I remember a past boss of mine said one time "it's hard to monetize
protocols".

It's really the only thing he said I thought was smart, so it stuck with me.

------
hossbeast
So where is the open source Twitter alternative?

~~~
chridal
There are a lot of them.

i.e.
[https://www.gnu.org/software/social/](https://www.gnu.org/software/social/)

~~~
cyphar
The biggest problem with creating a free software social service is that you
can't just apply the old method that GNU has used historically: Just mirror
the interface and you're done.

That method works fine for many different tools. But with social apps you need
to be able to federate, which means you need to have a proprietary vendor who
wants to help, which isn't going to happen.

I'm hoping Matrix will provide the bridging required to be able to send
notifications from Diaspora to Twitter (for example). I will always be a hack
though. :/

~~~
chridal
I think there's also a whole lot of other issues. The technical part is one
thing, but people's feelings are a lot harder to deal with.

We, hackers, think that everything is so sexy with P2P, federation and
decentralization in general, but I don't think 'normal' people share that
sentiment.

People love brands, they're surrounded by them, and they feel loyalty towards
them. If there's no company behind something, it just won't feel right. You
won't feel that push towards a thing. The reason why people use Snapchat isn't
just because it's a good tool, it's also because it's cool.

I just don't see how something like GNU Social will ever become cool, if there
isn't some timely and powerful brand to push it.

~~~
mxuribe
Good points...tech is no longer enough...we have to have a brand as well as an
attraction point; that is, some "place" where people think "the party" is
happening (alternatively not a party but for example, where stuff or news
happens). I think I've come to the point where - right now - we don;t need
_everyone_ to adopt decentralized platforms like gnu social...But, i do feel
if there is at least a non-trivial percentage of users who can choose to not
be limited to interaction by way of silos, then i think that's a pretty good
start. And who knows, maybe in the future, when there is enough to attract the
casual users (as well as the tech is easier to setup and use), more folks will
slip over to something decentralized, though still be able to interact with
the/their communities.

~~~
chridal
Yes, I totally agree with this. We need to improve the tech, to the point of
it being accessible and as good as the mainstream offerings.

The thing is also that we don't have any clear incentives to use P2P-services
right now, because the silos aren't posing any clear threats to us as
individuals. If we see something like a major data breach, or something like a
"Snowden for social networks" that change the way we relate to these
behemoths, we might just see users getting ready to give them up.

------
pjc50
One of the things currently regarded as a serious problem for Twitter is
abuse. Federation doesn't make dealing with this easier, it makes it harder -
look at what happened to USENET.

------
revelation
IRC is a protocol. Slack is where the money is at.

(Don't become an software architecture astronaut. IRC is also a terrible
protocol. It was still successful)

~~~
fluxsauce
If you feel that it could improve, contribute to the IRCv3 working group!
[http://ircv3.net/](http://ircv3.net/)

~~~
rspeer
Why is IRCv3 still relegating chat history to "IRC bouncers" instead of making
it a feature of the protocol?

Chat history makes chat more usable, and it makes it usable at all on
intermittent connections.

~~~
herbst
why would they include that? (Seriously curious)

As far as i can tell it would add no benefit but only overhead to the
protocol.

~~~
rspeer
Are you okay with missing messages, or do you only ever communicate with
people over an always-on wired connection?

~~~
herbst
IMO thats the magic of IRC. Channels i care about i bounce, channels i do not
care i dont bounce in. If messages would wait for me that would mean so much
overhead for the protocol.

I like IRC because its simple. I've build a IRC out of boredom, and a bouncer
because the one i used missed a feature i wanted. Please nobody take that
simplicity away :/

------
CaliforniaKarl
I'm sorry, I was only able to read about a third of the way through. Around
that point, my screen got taken over by something suggesting I subscribe to
something. The experience was so jarring that I just stopped.

Please, don't do that! Besides, I'm not going to make a subscribe decision
until I've read the whole thing.

------
bascule
There was OStatus:

[https://en.wikipedia.org/wiki/OStatus](https://en.wikipedia.org/wiki/OStatus)
[https://www.w3.org/community/ostatus/](https://www.w3.org/community/ostatus/)

Never went anywhere though, AFAICT

~~~
riffic
Was? GNU Social is a viable and actively developed and maintained OStatus
implementation:

[https://git.gnu.io/gnu/gnu-
social/commits/nightly](https://git.gnu.io/gnu/gnu-social/commits/nightly)

------
jnpatel
We've even seen botnets in the wild, using Twitter for their command &
control.

[http://www.welivesecurity.com/2016/08/24/first-twitter-
contr...](http://www.welivesecurity.com/2016/08/24/first-twitter-controlled-
android-botnet-discovered/)

------
shp0ngle
Whenever I go to twitter, I think "I wish there were MORE robots and MORE
automatically sent spam".

Also tweets from my fridge.

------
aikah
Twitter is an infrastructure, not a protocol. Anybody can copy twitter but
it's a bit more expensive to actually run it as scale. It took VC money, and a
lot of it, which expects a lot of returns. There is already a "Twitter
protocol", it's called RSS.

------
digi_owl
The basic problem for Twitter was that they opened up their API for third
parties before they had any clue about how to monetize the service. This then
shifted the focus from SMS to apps and web virtually over night.

------
ainot
It was, they called it IRC.

~~~
Torgo
As far as I can tell, the biggest users of IRC in this fashion have been
malware and bitcoin (I think bitcoin daemon finally removed this method of
peer discovery.)

------
draw_down
Well, venture capitalists typically do not invest in the hopes of building a
free, open protocol for all to use. That's... not really what they're trying
to accomplish.

------
nullc
Thank the gods for small blessings.

