
The Infrastructure Behind Twitter: Scale - kungfudoi
https://blog.twitter.com/2017/the-infrastructure-behind-twitter-scale
======
burgreblast
100,000's of servers for 100,000,000 of messages/day ?

I understand that half the servers aren't even doing messages, but, isn't
WhatsApp doing _2 orders_ of magnitude more messages with _3 orders_ of
magnitude (?) fewer servers?

Is that right? I'm curious how one would justify 10,000X worse?

So for each message, 10,000X more equipment is needed?

~~~
sametmax
Also:

\- whatsapp doesn't have to allow browsing the entire history of their
billions of messages;

\- whatsapp doesn't have tags. A message can go not only to 1000000 users, but
also to so many apps requesting updates for one tag.

\- Twitter allows advanced search, where you can browse, in real time (or down
to the entire history), a complex combination of people, tags and free text.
With settings such as choosing the lang or the date.

\- Whatsapp has a list of messages. But Twitter has a graph : message can be
RT again and again, answered to and liked.

\- all those features have some impact or the other on the way the tweets are
displayed to the user.

\- Twitter's API is much more used than Whatsapp's.

~~~
jraedisch
WhatsApp also does not need ad related analytics.

~~~
monocasa
That's a big leap. They don't have to actually show ads for that analytics
data to be extremely valuable.

------
niftich
The day before, Discord did high-quality a write-up on why they chose
Cassandra [1], and now this post hits explaining how one of the world's most
popular and trafficked service has engineered their infrastructure; it's like
a dream.

I'll echo the praise I wrote earlier, that insights like this aren't only some
of the best content to hit HN, but become some of the most valuable resources
for designers who have yet to face a scaling issue, but know they will soon.

Since you have developed custom layers on top of open-source software to fit
your particular usecase and load profile, and host all this in-house, have you
considered monetizing your infrastructure for outsiders who may have similar
needs?

Today, one has limited, unpleasant choices: either pay out the nose for
something like AWS or Google Cloud to get elastic scaling and the captive
storage systems that can be made to handle these kinds of workloads and still
have to write a fair bit of custom glue to get all pieces to play nice, or you
can build out the servers yourself, but have to employ dedicated talent with
the requisite expertise. Either way, the barriers are fairly steep; you could
tap into an under-served market should you choose to sell IaaS (edit: or, more
accurately, PaaS). Has this conversation come up in the past?

[1]
[https://news.ycombinator.com/item?id=13439725](https://news.ycombinator.com/item?id=13439725)

------
atcole
This is a fairly technical analysis, and the terminology used in many cases is
above what I know about networking. But the one quote that will stick is this.

"There is no such a thing as a temporary change or workaround: In most cases,
workarounds are tech debt."

~~~
kristianc
Tech debt is a choice. Sometimes you'll want to embrace some level of
technical debt in order to bring something into production quickly, with the
understanding that you will fix later. It's part of a triad with Speed and
Quality.

That may not be the choice Twitter has made in this instance, but it's a
viable choice nonetheless. Defaulting to tech debt = evil wrong imo.

~~~
dingaling
The problem with technical debt versus financial debt is that the latter has a
monthly mandatory cost ( interest payments ) that can't be ignored and which
is visible all the way up to the C-level, whereas middle-management can keep
obscuring the presence of technical debt and pushing its repayment out to the
right.

Essentially it's a 'free' internal debt, regardless of how often architects
and developers complain of its cost.

Thus in a contest between doing something right, but expensively, versus good-
enough-for-now but technically-constrained the latter will usually win.

~~~
tomblomfield
It's not "free", it's just much harder to measure.

The cost is reduced development velocity, and perhaps reduced systems
stability.

~~~
lostcolony
Which developers get blamed for, even though it was a managerial decision to
take on the debt.

Financial debt has clear cost; technical debt doesn't. So it seems 'free' to
the non-technical.

------
rollulus
It strikes me that so much of the components they use (e.g. under "Storage")
are in-house built (several dbs, blob store, caches, etc). Is that because at
that time equivalent solutions didn't exist? Is that because Twitter suffers
from NIH?

~~~
fizx
I built custom storage for Twitter back in the 2010-12 period. There wasn't
much off the shelf in those days that worked out-of-the-box at scale besides
Cassandra, and Twitter had a well-documented attempt at using Cassandra for
primary workloads that failed due to the amount of variance in IO and latency
for high-volume workloads.

Most of the time, we were building custom distribution layers on top of open
source storage (e.g. memcached/mysql/redis/etc). I think blobstore was the
first thing twitter put in production that was mostly custom, followed a year
or two later by manhattan. I'm not sure if there's even now good open-source
competitors for those projects, largely because any reasonable smaller company
uses s3 or dynamo.

There's plenty of open-source things twitter created, or nurtured out of the
existing ecosystem, from mesos to memcached to some of the
hadoop/scalding/parquet stuff.

~~~
seanp2k2
The flip side of that imo is that if you don't have Twitter-scale needs for
the specific things they've optimized their infrastructure for, you probably
don't need their solutions :)

------
tabeth
I know nothing about storage, so I'm a bit confused about why Twitter needed:

1\. Hadoop

2\. Graph

3\. Redis/Memcache

4\. Blobstore

5\. SQL variants

(and a few others).

I do see that the post has a short snippet briefly describing what they're
storing, but I'd be curious to know why (speed, cost, latency, space
tradeoffs/constraints).

Also, if any more experienced folks want to chime in: Elixir/Erlang is "built
for concurrency" as they say. I'd love to hear people's opinions on their one
sentence simplifications of what kind of situation Hadoop/SQL/Redis/etc should
be used for (similar to how Erlang is best used for situations where
concurrency and fault tolerance is desired). In particular, is there a "Code
Complete" type book for storage?

~~~
marcinzm
That seems a pretty standard stack, I don't know about twitter but I can say
why I'd use them in my experience:

5\. SQL variants: This is where you keep your source of truth data needed to
serve request. It's flexible, reliable but moderately fast. The backbone of
your infrastructure and you can consider everything else as enhancements to it
to overcome specific limitations.

4\. Blobstore: Your SQL database is not going to be good at storing large
binary blobs such as files or images.

3\. Redis/Memcache: Your SQL database may be too slow for some requests and/or
it's data model may require too much work in the API layer to align with the
final presentation model. So you want a blazing fast store that you can use
for that final presentation view when requests don't differ from each other.

2\. Graph: SQL databases aren't the best for graph queries or data (I mean you
can hack it but it's going to be slower and uglier code wise) so if you need
to look at data as a graph something dedicated makes sense.

1\. Hadoop: You're getting lot's of log data from users, what they do, when
they do it and so on. You want to analyze this data, build models (who to
recommend for people to follow for example) and use it to probably serve ads
(gotta pay the bills somehow). You could put this in SQL but it's a lot of
data and you'll be doing a lot of read requests which will end up being slow
and messy in SQL. Hadoop, on the other hand, is well suited for this type of
data and workload. Large slow read heavy queries on data that is appended to
but not modified in place.

~~~
dajohnson89
FTFA: >Graph: Our legacy Gizzard/MySQL based sharded cluster for storing our
graphs. Flock, our social graph, can handle peaks over 9 million QPS,
averaging our MySQL servers to 30k - 45k QPS.

Isn't storing a GraphDB in MySql a Bad Idea? Graph queries are extremely
poorly suited for relational databases.

~~~
thinkingfish
I don't work on GraphDB, but basically the situation is this: 1) yes, it is
not a great use case for MySQL, but that's how it started, partly because
Twitter needed join on its graphs; 2) legacy systems die hard, especially at
scale- Twitter is working on a better solution, but for now Flock is still
what's running in production.

~~~
dajohnson89
Thanks for explaining. I had a similar situation -- storing a graph db in
postgres. The performance sucked, but we didn't have much of a choice.

------
jzl
Lots of CIO-type buzzwords and acronyms in this, including many coined by
Twitter themselves. More worthy of a skim than a rigorous reading. But
interesting for a bird's eye view of how complex Twitter is behind the scenes.

------
mnutt
They mention that they used to base their geo routing on the location of the
client's DNS server but have moved to BGP Anycast. I've heard that there can
potentially be routing issues for long-running connections using anycast to
end users, is anybody else doing something like this and do these issues
happen in practice?

~~~
pyvpx
TCP Anycast - Don’t believe the FUD

Matt Levine (CacheNetworks), Barrett Lyon (BitGravity), Todd Underwood
(Renesys)

Operational experience with TCP and Anycast

[https://www.nanog.org/meetings/nanog37/presentations/matt.le...](https://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf)

------
seanmccann
> Fast forward a few years and we were running a network with POPs on five
> continents and data centers with hundreds of thousands of servers.

That seems high given Twitter's size and the hardware distribution pie chart
they showed. Does anybody have an idea how this compares?

~~~
smueller1234
I'm working for a very successful Internet company with 9 figure user count.
We have 7 or 8 POPs on 4 continents and run a low 5 digit number of servers.

This being said: we do not have to deal with anything like the network effect
and lifestyle type usage that twitter has. Once you pass a certain size and
complexity of infrastructure it happens really easily that economy of scale
doesn't help and the complexity of everything having to be distributed works
against your efficiency big time. So I'm not shocked by their numbers.

------
ashayh
| We have over 100 committers per month, over 500 modules, and over 1,000
roles.

| we were able to reduce our average Puppet runtimes on our Mesos clusters
from well over 30 minutes to under 5 minutes.

This isn't just tech debt .. it's poorly designed, poorly thought, poorly
architect-ed and poorly managed in the first place.

Is it because Twitter cannot find good talent because of its falling stock?

------
marknadal
I've talked/pitched (full disclosure: got a "no") some of the bigger names
behind the product at Twitter. I was a little disappointed because they seem
to be proud of their at least $15M+/month server costs which is partly driving
their company into the ground (the user facing product hasn't improved despite
"re-engineering" of their backend, at non-substantial price differences, and
lack of innovation for consumers have made them lose everyone to Snapchat or
anti-censorship sites).

Of tweets alone (about 200bytes per tweet), over the last decade, they
probably have about 3 petabytes. Unknown to them (because of the
aforementioned pride) they have 1.5 petabytes a month of free storage/caching
they aren't even touching. If they switched to a P2P model like IPFS or (full
disclosure: I work here) [http://gunDB.io/](http://gunDB.io/) , but Twitter
seems determined to stay as a centralized monolith. Which is too bad, because
that has now become their own death - the regime change is happening and
decentralized services will win instead.

Edit: Compare against 100M+ messages (1000bytes each) for $10 a day. 2 min
screencast here:
[https://www.youtube.com/watch?v=x_WqBuEA7s8](https://www.youtube.com/watch?v=x_WqBuEA7s8)
. Even if you multiplied the feature set by 10K times, you would still be
saving $12M+ a month. At this rate the Discord guys (pushing 120M+ messages a
day) are doing way better - there post is on top of HN right now too, see my
comment there as well. And they only have 4 backend engineers.

~~~
jhoechtl
Why the downvote? The post contains valuable architectural links worth a
discussion

~~~
rapsey
1\. Because it's a .js database.

2\. Because the website does a poor job explaining what problem it solves.

~~~
eduren
Agreed, their website definitely turned me off.

Quick tip for them: Having a section for showing that a bunch of people are
using and "Trusting" the product is good, but your's seems to mostly (4/7 from
what I can see) be the investors behind your product.

Edit: Looked at the names more, and it seems 7/7 are individual investors or
VCs

[https://angel.co/gun](https://angel.co/gun)

Which is all well and good, but maybe the wording should be "Investors and
Backers" rather than "Trusted By".

~~~
marknadal
Sorry the website turned you off :( what recommendation do you make for
balancing both enterprise and developers? Currently we have two website, the
other for developers: [http://gun.js.org/](http://gun.js.org/)

Better links:

(again)
[https://www.youtube.com/watch?v=x_WqBuEA7s8](https://www.youtube.com/watch?v=x_WqBuEA7s8)
100M+ msg a day for $10.

Doing distributed machine learning:
[http://myrighttocode.org/blog/artificial%20intelligence/part...](http://myrighttocode.org/blog/artificial%20intelligence/particle%20swarm/genetic%20algorithm/collective%20knowledge/machine%20learning/gun-
db-artificial-knowledge-sharing)

30M+ ops/sec performance testing:
[https://github.com/amark/gun/wiki/100000-ops-sec-in-
IE6-on-2...](https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-
Atom-CPU)

1 minute demo of seamless multi-server failover during a live chat app:
[https://youtu.be/-FN_J3etdvY](https://youtu.be/-FN_J3etdvY)

1 minute demo of recovering from a complete server meltdown:
[https://youtu.be/-i-11T5ZI9o](https://youtu.be/-i-11T5ZI9o)

You are right - trust should be proven by demonstration, not the blind faith
of others.

We're also deploying out to customers with 1.5M users in production and
customers with a product being shipped to 1.5K homes. And we're nearing our
production-ready v1.0 stable release.

Feedback like yours is helpful, so please hit me up with any other critiques
or ideas on how we can explain ourselves better. Thanks for jumping in on the
convo :)

~~~
eduren
Thanks for being receptive. My original recommendation is all I have right
now.

>maybe the wording should be "Investors and Backers" rather than "Trusted By"

~~~
marknadal
Done! Thanks :)

------
hueving
They switched to a BGP anycast model for twitter.com, which implies TCP. I'm
curious how they deal with situations where route preferences change in
intermediary ISPs mid-TCP stream. Does the new server reject the TCP
connection or are they synchronizing TCP sessions across clusters?

------
therealmarv
No Ansible? In an older (2014) Ansible video they claim Twitter is using
Ansible but I only see Puppet mentioned.

~~~
rco8786
Only ever used puppet during my time there, fwiw

------
turbohz
If Twitter can manage to operate at profit, then I might be interested in
Twitter's infrastructure.

