
Twitter and the Internet War - dsr12
https://www.vanityfair.com/news/2018/02/how-twitter-lost-the-internet-war
======
continuations
> Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-
> application framework that made it nearly impossible to find a technical
> solution to the harassment problem.

This is probably a new low in journalism.

~~~
kenning
Can you elaborate? I don't know much about ror and would like to know more.

~~~
ericd
The web framework (Ruby on Rails) has nothing to do with the ability to build
anti-harassment tech.

~~~
liquidgecka
Its not described well at all in the article, but the instance on sticking
with a RoR framework well beyond what it could rightfully scale up to is what
kept us from building a great, great many features, least of which was
harassment tooling. It wasn't RoR's fault beyond being the framework selected
early on, and dogmatically stuck too long after it stopped helping us move
forward quickly.

------
firasd
My general feeling when it comes to questions about harassment on Twitter is
that people look at it too much as a policy issue ("you're not banning and
deleting enough accounts according to my ideological stance"), which has its
place, but there's a lot of product aspects that make it such a shouty place.

A good example is the way the 'quote tweet' feature is often used to start a
pile-on. It's not a bad feature in itself, but there is a significant portion
of usage that lends itself to starting food fights. Could things be improved
by, for example, a setting that limits quote-tweets to people who follow you?
There's a lot of dynamics like that which can be explored.

~~~
philwelch
Jeff Vogel:

> Twitter was designed, from Day 1, to enable any random person to send
> messages directly to any public figure. In other words, from Day 1, it was
> designed to be an abuse and harassment engine. It's not a bug. It's a
> feature. All that abuse and controversy is how it gets clicks and money.

[http://jeff-vogel.blogspot.com/2016/04/how-i-deal-with-haras...](http://jeff-
vogel.blogspot.com/2016/04/how-i-deal-with-harassment-abuse-and.html)

Twitter's feature set is perfectly optimized for harassment and abuse.
Changing it to prevent harassment and abuse would kill the product.

------
JumpCrisscross
“I think what religion and politics have in common is that they become part of
people's identity, and people can never have a fruitful argument about
something that's part of their identity. By definition they're partisan.

Which topics engage people's identity depends on the people, not the topic.
For example, a discussion about a battle that included citizens of one or more
of the countries involved would probably degenerate into a political argument.
But a discussion today about a battle that took place in the Bronze Age
probably wouldn't. No one would know what side to be on. So it's not politics
that's the source of the trouble, but identity. When people say a discussion
has degenerated into a religious war, what they really mean is that it has
started to be driven mostly by people's identities.

...

More generally, you can have a fruitful discussion about a topic only if it
doesn't engage the identities of any of the participants. What makes politics
and religion such minefields is that they engage so many people's identities.
But you could in principle have a useful conversation about them with some
people. And there are other topics that might seem harmless, like the relative
merits of Ford and Chevy pickup trucks, that you couldn't safely talk about
with others.“

[http://paulgraham.com/identity.html?viewfullsite=1](http://paulgraham.com/identity.html?viewfullsite=1)

------
danso
This is the excerpt I've seen being tweeted about this piece:

 _At the same time, her defenders say, Harvey has been forced to clean up a
mess that Twitter should have fixed years ago. Twitter’s backend was initially
built on Ruby on Rails, a rudimentary web-application framework that made it
nearly impossible to find a technical solution to the harassment problem. If
Twitter’s co-founders had known what it would become, a third former executive
told me, “you never would have built it on a Fisher-Price infrastructure.”
Instead of building a product that could scale alongside the platform, former
employees say, Twitter papered over its problems by hiring more moderators.
“Because this is just an ass-backward tech company, let’s throw non-scalable,
low-tech solutions on top of this low-tech, non-scalable problem.”_

Setting aside the whole "Ruby/Rails is slow" discussion, I would've loved to
see more elaboration by these sources about how a web framework makes
moderation and policing "nearly impossible". Compared to what? The PHP scripts
that became Facebook?

~~~
liquidgecka
Hi, early Twitter employee here. The reliability team called the
infrastructure Fisher-Price internally so this wasn't just one random
executive coming up with a term on his/her own.

The problem wasn't ruby. The problem was the way that Twitter used Ruby. We
had one big mono repo with every single function and every form of business
logic baked into a single place. That logic relied on monkey patching and all
sorts of crazy horrible glue to keep it working together. Every time we had to
scale up we would glue infra in place to keep things working while we came up
with a real solution (which never really materialized).

In my time there we had memcache instances which held timelines. Populating
them took hours/days and while they were unpopulated the site was offline.
Rebooting/restarting the caches was simply not an option. We had a data
sharding strategy that was temporal. We would spin up a new database cluster
every few weeks to handle all of the incoming tweets and failing to spin up a
new cluster in time meant we would have a global site outage. Don't even get
me started on the "load bearing mac mini".

In reality the only problem rails really contributed on its own was that it
could only process a single request per process at a time. Each machine would
spin up 16 or 32 processes to handle requests in parallel but each process
needed its own connection to the database, to memcache, etc. At one point we
had something like 100k processes all trying to talk to a single mysql master.
Much of this could have been mitigated by better design of course, but rails
encourages models that don't scale up to crazy dimensions.

In reality moderation was virtually impossible because we were in a 24/7 fight
with ourselves about how to keep the system alive for the next couple of days.
Constant infighting, managerial changes (I had 9 different managers in 3
years), focus changes (we didn't finish the last major site redesign before
starting the next one) and a general unwillingness to pause features long
enough to stabilize the system meant we were always on the losing end of a
infra battle.

~~~
stryk
I don't know nearly enough about large scale projects or the startup world
maybe this is a silly question but when it became apparent that it was blowing
up far beyond what you were expecting, was it not an option to have another
team (hire or split existing) in parallel, to start writing a re-implmentation
in another language or system that would work better for your new quickly
expanding needs? Was that just not feasible?, no doubt it was a crazy hectic
time for you folks back then, I can't even really imagine what that was like.

~~~
liquidgecka
Thats what kept happening and it was an abysmal failure.

Every so often a person would have a brilliant idea on how to solve our
scaling issues. They would then disappear into a corner to invent yet-another-
bird-themed-datastore. After a few weeks/months they would appear with a
magical new thing that would fix all our problems and would make everybody
happy. Every single time it would fail.

Having a team that is not the main team design something means that they
likely didn't understand the state of the thing that they were replacing. The
thing they were replacing was a bucket of edge cases non of which they knew
about. The scale never looked like what they expected because in the meantime
the load had changed. This was compounded by the constant desire to hire
somebody external that could solve the problem for us. They would come in with
ego and a feeling that they had a mandate to replace it all. Eventually they
would learn just how fragile and complicated the system was, only to then be
considered old guard enough to be replaced by the next wave of experts. =/

But the number one killer was that every single thing was baked into the mono
repo so it wasn't like they could have just easily shimmed in something to
replace the old thing. All the while that they are building in a change to the
data store another dev has added 15 new features that they now have to port
over. In the time it took to port those over another 20 had been added.. etc.

Just getting the okay to pause feature development was like pulling teeth and
it only bought you a few weeks at best.

~~~
Abderian
Maybe I'm being dumb here, but twitter doesn't look like a product from the
outside that has many features. Are these focused on advertisers, analytics or
what?

~~~
liquidgecka
At the time Twitter had a ton of features under the hood that kept being
supported and maintained, all of which just added complexity to the system.

We had an API service, a web interface, the legacy web interface that was
still used for select devices because the new UI didn't quite work right on
them, the even older legacy interface that was necessary because a bunch of
badly behaved early day clients still relied on the functionality and they
were popular enough that turning them off would cause outrage, the "zero"
interface used in countries with low bandwidth capabilities, the mobile
interface.

Each interface had to implement all the different variations on functionality.
Timelines with inline tweet rendering (automatic expansion of images, etc),
list (alternate view time lines), the whole following graph (duplicated for
lists as well), verified users and all the infra around that, search,
public/private designations, direct messages, notifications via email, text
message, and mobile app, favorites, retweets, replies, plus a slew of
statistics and information tracking data integrated directly into the site..
Thats only the user visible stuff. There are a TON of experiments and projects
that run behind that interface in a way the user will never completely see.

We heard over and over that twitter was so simple that it could run on a
laptop and every time it reminded me just how clueless most developers are
when it comes to seeing the body of work needed to make something like twitter
work, even more so at the scale we are talking about.

------
mark242
Part of the problem is that Twitter encouraged automated signups in the early
days, blazing the trail for gigantic bot-farms that we all talk about today.
They pushed popular rss-to-tweet gateways, wordpress plugins for auto-tweeting
blog posts, etc. There should have been gigantic red flags waving when the
hypergrowth of Twitter really started, because you knew these weren't all
people signing up for one account.

I'm not sure if Twitter can ever put pandora back in the box, but at the very
least, requiring a mobile number is a start. It's hard(-ish) to generate
thousands of bots if you have to have a unique phone number with multifactor
for each signup.

------
icelancer
Lost? How is Twitter losing the Internet War? Their platform is more
influential than ever and helped to seriously impact a presidential election
(the results of which not too many people in media are happy with, fair
enough).

Yes, the platform is a failure in so many ways - failure to protect the
identity and safety of the people on it. (Though as someone who has served a
subpoena to Twitter, let me tell you something - the legal team isn't exactly
handing over data easily. It was a huge pain in the ass with tons of
individual privacy concerns the whole way.)

And the platform is failing revenue-wise, yes.

But if it's about the Internet War, so to speak? Twitter is at the top of it
all. No one has to like the externalities - I sure don't - but their influence
is undeniable.

~~~
oceanghost
You're going to think I'm trolling, but I'm not--

What value does Twitter provide to ordinary people? I have friends who are
celebrities in their industries or trying to market themselves-- but other
than that it seems like the most hostile place imaginable. Why would I want to
participate in that?

~~~
craftyguy
In the US, most ordinary people pay attention to twitter's top customer:
donald trump

~~~
olivermarks
I suspect the vast majority of people read what Trump said on Twitter on their
favorite news site, rather than using Twitter itself.

------
anonytrary
This smells like boring FUD. Twitter is an incredible tool for real-time
information, and everyone already uses it. Twitters content problems are great
problems to have, similar to the problems Facebook faces. I think Twitter has
a very bright future.

Using Facebook, I have always felt that the website tried to "force" me into
where it thought I belonged in the social graph. Twitter has a much lower
barrier to entry. I prefer the lurker-first philosophy. Facebook tries way too
hard to engineer interactions (so does Twitter, but I think it's not as bad).

------
pg_bot
Well at least we now know the solution to all of society's problems is making
Ruby on Rails fast. Perhaps one day we can let out a collective sigh of relief
once we remove the global interpreter lock.

------
iisbum
What a cop out.

If they really wanted to solve the problem all it takes it consuming their own
firehose API and writing back to delete flagged content.

You really don't need to insert the moderation into the "backend", you just
have to want to solve the problem, instead of accepting the problem because it
fuels your growth.

~~~
itronitron
This article is an interesting variant on the 'we're trying but it's a really
hard problem' puff PR piece.

------
flashman
> There are two main components to Harvey’s job, this person told me: to
> formulate a clear set of rules for what constitutes abusive speech, and to
> be consistent in enforcing them.

The odds are against Harvey being the first person in human history to solve
this problem without false positives and negatives.

~~~
otterley
The big question in my eyes is not how a human can reduce the error rate, but
rather, which category of errors one should bias towards.

I know where I stand on this one: be biased against the mean-spirited.

~~~
proofbygazing
Wow you're really great

------
ohiovr
I lurved the hate they had on ruby. I've never used ruby but strikes me as
childish to blame all your ills on software that they developed to run their
whole company on. I mean they could have used PHP, a real professional
language! Just like what Facebook used :D

------
petraeus
Twitter chased growth at the expense of quality, quality moderation, and
quality infrastructure. Thats all it boils down to.

------
Aurelia_Cotta
I am not a coder by any means, but I am a heavy social media user and know a
lot about politics, sociology, psych, organizing, health, comms---and I admit,
until I read this thread, I liked this article because it gave me a better
reason for all the screw ups, beyond, "The Executives are dithering and have
no business skills and social skills." Or the theory "The Executives do
nothing because they don't care if women and vulnerable people die." Which is
far more disturbing....it can't be true, even if it _feels_ true. I would give
anything if they'd listen to users who have been around awhile. So few people
worked there and also used it at the same time. (And users longingly miss the
Fail Whale logo...) As a user (250,000+ tweets) under a pseudonym, twitter has
incredible uses, things Facebook and other platforms didn't do, because they
were so closed and hard to search. They created echo Chambers, because you
could only see people you already knew or were slightly connected too. Or
worse, your mom or mother-in-law could find you. Twitter was so open--if I
want to discuss philosophy or Japanese food or an MRI result, 24/7 I just
search and people who love that are there. And pseudonyms had to exist,
because thousands of people in real life have the same name. (just like the
rest of the Internet back then and yes, now too) They thought people would
just post status updates like, hey eating lunch, but we did way way more. They
made it 140 chars with 20 chars reserved for names because many people had
expensive tiny data plans, but could update by text message; allowing a much
wider demographic to use it, and across multiple countries, even low tech ones
with limited access. The most critical piece tho is that users invented
everything good about twitter (Sorry creators--but we did). TW allowed some
html symbols, so a user put an @ in front of a user name to reply, and it
worked! Users also invented hashtags, the first one was for organizing BarCamp
--which was kind of a conference not about alcohol and the original manual RT,
and MT for modified tweet, and commenting on top of someone's tweet, or at the
end of it and we figured out how to shrink long urls to post links. bit.ly had
no purpose til then, and many users became Developers and got full access to
the API and we crowdsourced hundreds of changes and ideas, from pictures to
videos to emoji to gifs to analytics, to accessible apps for people with
vision, hearing, speech issues. All while making lots and lots of jokes. They
didn't start out with a heavy respect for pseudonyms, and privacy, and free
speech--but TW learned it fast after the Green Iran Revolution, and after many
patient users wanted to keep privacy because they had rare diseases, and
mental health issues, and parents of kids with autism and speech issues took
to it and found each other and felt less alone. It was less complicated than
blogging and writing long stories on laptops after events happened. I could do
everything from live tweeting a doctor's appt to an ER visit and get reactions
from friends who could tell me what to ask. We still tweet everything from
recipes to exact instructions on how to ride a bike, how to organize groups
like #occupy to crowd sourcing Flu symptoms and rashes, to who is watching
what TV show and how cool it is to watch live sports "together" even for
people who can't leave the house that night and meet. Weekly hour long Chats
take place under special hashtags like #hcsm for Health Care Social Media or
#meded for medical education. (doctors and science fans and academics found
twitter and they debate articles, techniques, crowdsource diagnoses) Police
and emergency responders and good Samaritans have used it to (swear to god,
it's true) befriend people who sound troubled or suicidal and validate their
pain and sadness and find them help. It is to this day, the only platform I
know that allows people from many different areas to find like-minded people
and to bring together people across multiple subject areas. Nothing else does
it quite the same way. Especially with the ridiculous algorithms other
platforms use.

