

Handling five billion sessions a day in real time - knes
https://blog.twitter.com/2015/handling-five-billion-sessions-a-day-in-real-time?hn

======
buster
What i never see is how much servers/hardware/resources they are spending.
It's _always_ like "we process a quadrillion messages per second" but it never
mentions that you'll need a quadrillion expensive servers to do so. Same for
many many database benchmarks, sadly...

~~~
frostmatthew
It's a couple years old but _What it takes to run Stack Overflow_ [1] lists
the number of DB servers, Web servers, etc

[1] [http://nickcraver.com/blog/2013/11/22/what-it-takes-to-
run-s...](http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-
overflow/)

~~~
jackweirdy
This update is more recent:

[http://highscalability.com/blog/2014/7/21/stackoverflow-
upda...](http://highscalability.com/blog/2014/7/21/stackoverflow-
update-560m-pageviews-a-month-25-servers-and-i.html)

------
MrBuddyCasino
I had a hunch I'd see the data structures du jour in there: Bloom Filters and
HyperLogLog. Since they are pretty accurate by themselves already but run much
faster than the brute force approach, why keep the latter? The error margin,
if used properly, should be small enough so people shouldn't really care.

~~~
ropiku
They are not for accuracy. One usage for instance is to recompute stats if
there are bugs in the stats processing code. Or if you want to add more
metrics. (I work at Twitter but not on Answers).

------
meesterdude
> What happens if our speed (real-time) processing layer goes down? Our on-
> call engineers will get paged and address the problem.

Innovative!

But for real, that's some architecture, and very insightful to see what
they've got. Totally nothing I'd have guessed.

------
toblender
First thing I noticed was Amazon in there Stack. Amazon really owns the
internet...

~~~
philippnagel
In what way does Amazon own the Internet?

Yes, they own some of the computers that make up the Internet, but by far not
all of them.

~~~
metaphorm
they are the dominant vendor of hardware/hosting to the largest and most
complex web applications on the internet. AWS is the pillar that some of the
most high volume sites on the internet (like Twitter and Netflix) rely on. Its
safe to say that without AWS (or some other similar vendor) the internet as we
know it wouldn't exist.

~~~
latch
That's ridiculous. It's like you're saying: without Honda, the car industry,
as we know it, wouldn't exist.

The hosting industry is huge. There's a ton options, many of them better,
larger and older than AWS. There are _so_ many sites that don't use AWS. For
example, I believe that, pre-Google, youtube used ServerBeach.

~~~
sanderjd
> There's a ton options, many of them better, larger and older than AWS.

Some examples would be more interesting than a general "you're wrong".

~~~
latch
For every site that's reliant on AWS, there's probably more than 10 000 that
aren't. The giants, for example: Google. Facebook. Wikipedia. Microsoft.

It's hard to get actual numbers, but Equinix might be the largest. I _believe_
their Singapore data center is actually what AWS, Digital Ocean and Azure use.
(there's a lot more of these, such as Internap and Level 3 (which provided
(and might still) services for Netflix and iTunes.))

AT&T is quite large. World of Warcraft used to be hosted on AT&T's
infrastructure (might still be).

The above tend to be for large setups. Going smaller, you have the Rackspaces
and Softlayers (now IBM). And, to a lesser extent, providers like OVH and
Hetzner.

And, you can't forget the likes of Digital Ocean and Linode.

It's hard to come up with examples because more things are examples than
aren't. AWS' market share is growing, but that's to be expected, in a lot of
ways, despite EC2 being ~10 years old , it's still a new player.

AFAIC, what's more interesting about AWS is:

1 - How dominating it is over Microsoft and Google

2 - How popular the less invasive offerings are. A lot of people are sticking
to their colocated, dedicated or VPSs, while using some part of AWS for DNS,
backups or email. I hope they keep this up.

~~~
sanderjd
Thanks for the reply—exactly what I was hoping for!

------
domoarevil
twitter: @copypastaa and then appending ?hn is at least consistent :)

------
joosters
"How we handle spying on your mobile phone activity on a global scale"

~~~
bpg_92
hahaha but it is a good architecture you have to admint :D

